Dear all,

I was just informed by Ulf Brueggmann that the emergency power off test was successfully performed in FXE.

The  reality is now reflecting the planned logic, exeption being the power plugs in the experiment hutch. They do not go off. This is due to some missing cables. These will be installed in the next weeks and a new test will be planned for next shutdown.

The tests will continue tomorrow in SPB/SFX.

Summary of Karabo 2.2.3 deployment dates:

• Karabo 2.2.3 deployed in SASE3 tunnels last friday ==> no major issues. DAQ can be deployed
• Karabo in SASE2 Tunnels to be deployed today at 14:00
• Karabo in SASE1 Tunnels to be deployed tomorrow at 08:00
• Karabo in SPB/SFX experiment to be deployed today and tomorrow
• Karabo in FXE experiment to be deployed on Wednesday

SPB

• By tomorrow the mirror installation will be done.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.

ITDM

• Infiniband port fixing ongoing
• GPFS bug to be fixed with a patch that should be issued on 24.04. To be checked whether it is still OK to install the patch or it is too late.

General

• PSPO will ask HSinn for a detailed schedule concerning closing of the tunnels to fine tune the activities going on this week in the SASE2 and 3 tunnels

SPB

• Karabo has been deployed  without major  problems! 
• Now PLCs update follows. Sample chamber's PLC will be updated next week due to ongoing tests until monday.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.
• Updating PLCs for MPS in the tunnels 

ITDM

• Network stability problems in SASE3 are understood and a workaroung is being implemented.

General

Closing time of the tunnels on monday still not clear. To be clarified by PSPO.

SPB

• Timing tool for optical laser installed
• Sample chamber's PLC will be updated next week due to ongoing tests until monday. Pending.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Pending. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Pending.

AE

• SASE2 EPS Loop running
• Updating PLCs for MPS in the tunnels. The update was stopped last friday due to a problem with the on/off funcionality of some ion pumps. The problem was found and the fixing requires an update of the VAC loops whoch will be done in the next hours/days. In case the loop update is not possible due to specific requirements the problem can be bypassed in Karabo. 

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

SPB

• Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
• Smaract motors for PAM are needed and Nerea is on it.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions  have to be checked next week when HSinn is back. 
• During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it. 
• Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
• Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

• Access  of XTD1 today @14:00 to fix some EPS switches
• SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

AE

• SCS exp: first loop running. 2 creates missing but loop is up.
• MPS connection tunnels SASE2 done

ITDM

• Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
• MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

INFO

As reported by the PRC (J. Gruenert) during the operation meeting this morning, due to the first lasing attempt in SASE2 no lasing of SASE3 is to be expected until the middle of next week.

CAS (C. Youngman)

Following the Technical commissioning phase for the SASE2 tunnel components, the need of a technical coordination activity is strongly pointed out. This is in particular important to make sure that the proper final tests are performed by the components owner  before e.g. first lasing attempts.

SQS (T. Baumann)

Lopp 1 being brought up. First tests to be started soon. The CAS and AE contact persons should be in involved in the tests as early as possible.

SPB

• After re-compiling the relative karabo device, the configuration procedure of the AGIPD is now smoothand reliable.
• PLCs update ongoing. AGIPD is in a safe state to allow the PLC update.

SQS

• Meeting today at 11:00 with AE, EET, CAS to define the next steps of the end to end tests.

AE

• SASE2 tunnels - Only VAC conditions are transmitted from the EPS loop to the MPS. All the other conditions were supposed to be not necessary for first lasing and were not implemented. Nevertheless this was not clear for the people working in the BKR (especcially for diagnostics) during the last days. It has to be discussed with the PRC (H. Sinn this week) whether these have to be implemented on a short time. In pronciple AE can do it if requested. Otherwise the definitions will be implemented in the next shutdown.

ITDM

• Failure of one IBM server for online storage. Thanks to the implemented redundancy this will not affect the operation. The performances have to be checked.   IBM has been contacted.
• On monday evening ITDM OCD received a call from BKR because of some problems with the DAQ in the tunnels. It seems that this was due to the fact that the system was not properly tested by the "users" before. Seems to be solved now.

SPB

• Loops 5,7,11, update and Karabo deployed.
• Loops 12,13,8 update ongoing.

SQS

• Loop1 cabled up. Today the formware will be uploaded.
• Loop 2 will probably be ready on Wednesday

SCS

• Out ov 50 valves tested 5 showed problems. Investigation ongoing to understand where is the problem (PLC, Cables, Valves)
• In Loop 2 a problem with a crate containing Elmo controlers was found. AE on it. Next loop to be tested will be Loop 4.
• 1 crate does not have enough power to power up all the terminals. CR already prepared. Will be solved soon.

​AE

• By the middle of the week the valves will be included in the interlock systems (SASE1 issue)
• The EPS interlock definitions for the tunnels in SASE2 are being finalized by AE, XPO, XPD

ITDM

• Failure of one IBM server for online storage. Fixing ongoing.

DET

• Online preview and charge injection issues are being solved restarting the DAQ and re-deployng Karabo.

SPB

• Test on the mirrors peltier showed some issues due to the fact that the test boxes were different from the actual stuff. R.Bean will clarify it with FMB.
• Loops 12,13 update to be done.
• Magnetic turbos control: the serial communication has some issues. Being checked with AE and CAS

SQS

• Loop1 cabled up. Some names haad to be corrected in the firmware. As soon as the export is delivered by EET to AE the formware can be uploaded, Probably today.
• Upstream turbo pumps running using controlers in the rackrooms but still no PLC control.

SCS

• Some issue with the movement of the motors found. AE is on it.
• 1 crate does not have enough power to power up all the terminals. Will be solved today.

​AE

• The valves will be included in the interlock systems (SASE1 issue) - Loop ready to be updated - The MPS has to be disabled before updating the loop to avoid tripping of the beam.
• FXE loops update started

DET

• Hotfix for AGIPD ready to be deployed. Can start today. Details will be sorted out with SPB.

CAS

• SASE3 Soft Mono is now technically commissioned. PBLMs next in the line.
• XTD1 CRLs will be integrated iin the control system in KW 22
• XTD6 priority on the MID beamline. 

General

It is not clear if and when the experiment in SASE1 could get beam during this week. Has to be checked with the PRC.

SQS

• Loop1 still being worked by AE. Not yet up and running. 
• Loop 3 is needed to be up and running in 2 weeks since the mirror motors ar eincluded in this loop and they are needed for the final check with FMB.

SCS

• Bernard has been added to the SCS operation logbook and he is getting al the updates concerning commissioning.
• Testing of the Loop 1 motors ongoing., 2 out of 20 motors have problems. Being investigated.
• Some connectors names are reported not to be consistent with the EPLANS. Martin Teichmann will check that with EET.

​​AE

• SCS:
  • loop 2 running. Bernard has recommended that you start with commissioning of KBS_VDM, _HFM, _BFL, VFM motors because they are using MC2MOTOR soft device class (controller is Elmo).
  • Crate BKF00157 can be removed and refresh terminal will be added without interrupting the PLC - only equipment connected to this crate will not be available during this period of change.

• SASE3:
  • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail

• SASE2:
  • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail

• SPB:
  • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people.

• FXE:
  • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

DET

• Working of the optimization of the python piece of code allowing the display of online images. Right now 30 images per second rate is possible.

ITDM

• GPFS server fixed.

CAS

• XTD1 CRLs will be integrated iin the control system in KW 22
• 20.07 is the goal date for beam in MID and HED optics Hutch.  

​​AE

• SASE3:
  • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail ==> AE updated the loop. Nerea is making sure that the MPS signals are enabled in the BKR 
  • Stepper motor for the mirror bender was found to be miswired. Has been already fixed. Similar issue (inverse wiring) was found in SASE1. Cannot be fixed now. It will be done in the next shutdown. For the time being one has to drive the morot the opposite way.

• SASE2:
  • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail ==> DONE. Not clear if some motor parameters for XRO devices were saved before. D. La Civita and CAS are checking that

• SPB:
  • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people. ==> DONE. 

• FXE:
  • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

ITDM

• DAQ has to be upgraded to thelast verison of Karabo (just hotfix). Will be checked with Gero what is the best way to go.

The MPS comm for SASE 2 and FXE were tested and the only remaining part is to test the SPB to MPS changes. Tomorrow morning a  timeslot will be agreed between AE and SPB.

General

• Please remember to send the list of the tasks you planned for the next shutdown to Monica Areste by the end of the week.
• Please remember to send the list of the alignment works needed until July to Gerd Wellenreuther.

AE

SQS loop 3 is almost ready. 1 crate is currently in the AE lab but should be installed soon. The loop 3 is needed by SQS for the motor tests with FMB Oxford next week.

SQS

• Due to a pump that stopped working (Caverion) this morning there was water on the floor of the laser hutch and part of the experiment hutch.  Some cables (not powered and packed) saw water. These are being checked. Also lasermet boxes on the wall saw water. Lasermet is checking that.

• For the testing SQS requested the test box to AE 

SPB

• Final DAQ test ongoing. Slow data acquisition OK. This is needed for the users program this week. Fast data acquisition (inc. AGIPD) will be tested next week.

ITDM

• Patch released for Slow data acquisition FXE/SPB. LPD acquisition OK. AGIPD issues being investigated.

General

• For the upcoming emergency off test in SASE1 SPB and FXE should propose  a possible date in the next shutdown to be then discussed with AE.
• The machine is down due to the failure of a cryocompressor. Beam will be probably coming back tomorrow morning. Tomorrow morning will be also confirmed whether SPB and FXE cold use the beamtime for the user run.

SPB

• In the week end there was a failure of the PP laser shutter. It was possible to find a very temporary solution to circumvent the problem.
• The beam has been unstable for most of the time. No drift, mostly gitter. Reasons are unclear. The instability was also measured with the XGM in XTD2. SPB is analysing the data collected to be able to quantify the instability. 
• DAQ tests for AGIPD were planned for tomorrow but the request is to shift them to today to take advantage of the missing beam.

FXE

Some issues were reported: LPD readout of the data, Fast digitizers mainly. The details have to be clarified with the DET and ITDM groups.  

AE

SQS loop 3 is up and running. 

SCS

• Tests continue

SQS

• FMB Oxford will be working on the mirror tank starting form tomorrow until next week. Acces to the hutch is limited.
• There will be a meeting with AE to discuss the set up of the motors for testing (especially for the mirrors). SCS will also participate. 

ITDM

• SASE3 projects updated. Performance increased.
• Possible to move the AGIPD test for SPB to today

CAS

• Camera device had to be reinstanciated 4 times during the week end. Investigation ongoing.

AE

• CRLs in XTD1 being implemented
• GATT and EPS loop in SASE3 being updated

SCS

• Tests continue - No issues so far

SQS

• ELMO controllers have to be configured for the mirror motors test. Slit motors have lower priority. Cables can be tested w/o being physically connected to the motors, but to configure the ELMO controllers the cables have to be connected to the real motors. Tomorrow AE is available to support these activities.
• Basler cameras set up and working.

​ITDM

• SPB AGIPD test today or tomorrow

CAS

• Hot fix for SPB released yesterday. This will address the issue of crushings when instantiating a new device. Has to be tested with SPB. 
• XTD 6 will have no electricity for the racks today since Wille is completing the cabling. MTCA being shut down accordingly.

SCS

• List with the problems encountered so far during the end to end test was sent to AE. Bernard will get the info and eventually inform EET if problems with cables are found.

SQS

• Elmo motors for KB system were ocnfigured and are being moved right now. If problems arise AE will be contacted.
• It was not possible to control the turbo pump controllers via PLC. Bernard will have a look. In general also tirbo pump cables can and will be checked with the test box.
• Loop 2 is being prepared including the MC2 motors. Should be ready by tomorrow.
• SQS would like to start setting up the digitizers environment in Karabo. uTCA crates can be used on the HW side. AE acan support that.

AE

• XTD6 VAC loop is being patched. Small problems with the cabling of the CRL occured but solved. 

XRD

• XTD6 components being cabled. 2 Imagers still need to get the EPS signals. in 1/2 weeks ready for local tests.

​ITDM

• One IBM storage server failed. Reasons not clear. No impact on the performance of the system.

CAS

• This Week VAC loop in XTD 6 will work.
• For the MOV 2 loop still  EPLAN exports are needed.
• MOV1/MOV2 interfaces have to be defined.

SPB

• Ready for the Users run. No big issues.
• The first 2 weeks of the shutdown will be used for cleaning and rearrangement of the hutch by SPB, in preparation of the mirrors instalaltion. PLCs update and Technical interventions by TS or external companies have to be carried out in this period. 

AE

• XTD6 VAC loop running.
• CRL XTD1 working
• Loop 2 SQS experiment still not running due to small issues being solved.
• Loop2 SCS was restarted yesterday after some fix

ITDM

• Gpfs should be fixed by now.

SPB

• Beam delivery issues.
• After several days of usage the Karabo GUI slows down. Servers are reported to have crashed a couple of times in the last shift.
• Same situation with data aggregators. Once per shift they have to be restarted. This in known to ITDM.
• Interlock system seems to work fine now, but the impression is that the people in BKR cannot correctly interpret the interloch signals coming from the experiments. The issue relates to the Interlock panels to be developed in BKR (Issue list, Friday's Operation meeting)

FXE

• Reading the logbook it seems that some problems with the optical laser's shutter forced to stop the laser operation at about 3 AM. 

SCS

• Tests  continue

SQS

• Loop 1 one more ion pump will be implemented today by AE
• Loop 3 - Configuration of the rest of the  motors expected today
• Loop 2 should be up by tomorrow according to AE
• Loop 4 and 5 are supposed to go up in the next 3 weeks 

AE

• EPS in SASE2 should be ready today

ITDM

• FXE reported missing images from Gotthardt detector in the H5 files. The data does not reach the DAQ. Tests have to be made, but cannot be done while SPB has beamitime. Tests foreseen for tomorrow.
•  

CAS

• Some issues with scanning tool reported - working on it
• Some issues concerning data acquisiion from imagers were reported from SPB  - Valerii on it.

SCS

• Tests  continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

• Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
• Loop 3 - Motors test continue
• Loop 2 is up and running.
• Loop 4  probably will run tomorrow.
• In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

AE

• EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

• Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

• Karabo version 2.2.4. being tested with PLCs. 

EET

• Firmwares fo all the SCS loops have been generated.
• Firmwares up to Loop 5 for SQS have been generated.