SPB/SFX

• The vacuum section of the burthrough monitor was vented for some intervention  and is now being pumped again.
• CSS component has issues with limit switches implementation. Will be discussed with AE and EET.
• Jungfrau loop can be brough up once AE gets the fuse boxes formt he company.

FXE

​SQS

SCS

• Waiting for loops updates
• DSSC can be switched on since no more works on the power lines are foreseen.

MID

• ATT XTD6 being re commisisoned.
•  Mono in XTD6 being commissioned.
• Optics hutch commissioning ongoing.

HED

• Loop 2 needed for commissioning. AE checks the status.
• Modules for the attenuator also needed. AE checks the status.

​OPT Lasers

​XRO

• Encoders for the mirrors in XTD6 are being checked.
• Commissioning of benders in SASE3 planned for 8,9,10. Feb. SQS hutch will be interlocked and shutter opened.

​XPD

​VAC

• Tunnel access schedule was updated: https://ttfinfo.desy.de/elog/XMLlist?file=/XFELelog/data/doc/Schedule/2018-12-12T08:13:57-02.xml&xsl=/elogbook/xsl/elog.xsl&picture=true

AE

• The timing software needs to be updated. Today at 14:00. 

ITDM

DAQ in the tunnels has been updated.

A mail will be circulated with the new proposals for users.

SA1 will be switched to the new storage. Corrdinated with CAS. This will allow the users to access data on the online cluster without first migrating the data to the Maxwell cluster. System will be tested on Wed afternoon and Thu. In this time frame interruption of the storage systems can happen.

EET

​CAS

AoB

SPB/SFX

• Trigger for Jungfrau detector is needed. 
• Commissioning of the instrument should start next week
• Loop priorities for the donwstream components have been set with AE.

FXE

​SQS

• VAC interlock test will be done during next days after update of the loop.
• Loop 4 has to be updated following some HW changes. Should run by next week since is needed for the preparation of the users run.
• Digitizers crate is being tested by AE and CAS and will be installed/usable by the middle of next week.
• Burst trigger - status and requirements will be discussed with BFernandes after this meeting.
• MPOT test CY/TB Today at 13:30

SCS

MID

HED

• Preparation of Hi res mono has been completed last week
• OPT hutch CRL commissioning ongoing. ATT will follow (needed missing crate will be installed next week)
• Chamber motion for the monochromator has to be checked 

​OPT Lasers

​XRO

• Loops update in thetunnels have been completed. the MOV1 in SASE3 included also some PBLMs ==> XRD should check whether everything is fine afte the update.

​XPD

​VAC

• Tunnels are ready for beam.

AE

• EXAPOD in SPB can be tested.
• HED need PPU in XTD6. To be re-checked.
• The timing software needs to be updated. BFernandes circulated a mail with proposed date: Next monday at 14:00. In case of different suggestion, please contact Bruno. CAS suggests to keep the proposed date.

ITDM

ET

​CAS

• Data logger schould be working for every component now. ==> Meeting later with VAC and Laser group to define the use case and start a new optimization project.
• VAC of HED Opt tests are ongoing (CY) before pumping down.
• For the debug of gauges control is possible that a ZZ access will be requested. (CY)

AoB

• Tunnel access schedule will be updated by HSinn after the Operation meeting with the latest information.

SPB/SFX

FXE

• Fixing session of the sample stage. No controls modification involved. Done by today.
• Shutter controls from the hutch. Some works were supposed to be done by MPS. What is the status?

​SQS

• New interlock definitions have to be tested and implemented. Some tests were performed and bugs fixed. New tests are needed and planned for Wednesday. ==> Master loop will be updated until tomorrow. Tests still planned for Wednesday. ==> Test successful. Now SQS has to decide whether to use the interlocks during next runs.
• VAC interlock test will be done during next days after update of the loop.

SCS

• Updated interlock definition will be given to AE by SCS==> Done.
• Electronics and cabling for the laser hutch is coming next. MTeichmann will talk to EET to agree on priority concerning the cables production and pulling. It may be necessary for SCS to pull a few cables as soon as possible (by themselves?). Anyway, cables route will be needed. ==> EET is waiting.

​MID

• Loop commissioning priorities being discussed with AE. (as well as checking whether all the important PLCs are there)
• Next Tuesday the cabling shoud be completely done. 

HED

​OPT Lasers

• in SASE1 the seeder was swapped. This affects the timing. Will be discussed with AE.
• PP laser should also get the train id. Change request to be prepared soon.

​XRO

​XPD

​VAC

• Leak test for HED mono has to be performed
• AE reports they got feedback that also SASE1 vacuum  needs to be tested. MD checks. 

AE

• AE was asked to update MOV1 MOV2 and EPS loops for SASE3 tunnels. This is due to the implementation of a new B4C to be mounted and controlled. ==> MOV 1 will be updated. Also CAS will be asked to save the motor parameters. NO EPS and MOV 2 loops update necessary for now. ==> SASE3 MOV 1 will be updated today.

ITDM

• DAQ installations for SASE2 and 3 will be physically migrated in the balconies to separate them from SASE1. Will be completed in the next days. Will be finished before DAQ has to be operative.
• DAQ SASE1 running for detectors test.
• MID will also get DAQ working soon for AGIPD test.

DET

​CAS

• Data logger schould be working for every component now. ==> More info to come.

AoB

• LIEN fixed the sliding doors in SASE3 experiments. Drag chains have been installed. Only 1 cable has to be ordered and exchanged. Will be done during the next shutdown.

SPB/SFX

FXE

​SQS

• AQS chamber has been removed. The reaction microscope is now being installed. (Currently sitting next to the beamline)
• New interlock definitions have to be tested and implemented. Some tests were performed and bugs fixed. New tests are needed and planned for Wednesday. ==> Master loop will be updated until tomorrow. Tests still planned for Wednesday.
• Mpots control and integration is still being debugged by CYoungman. Additional tests planned for tomorrow.

SCS

• All the PLC hardware is now fixed, with the exception of some high voltage interlock controls, which is waiting for the E-plan to be finalized. 
• Firnware update needed and planned for the next days.
• Updated interlock definition will be given to AE by SCS
• Electronics and cabling for the laser hutch is coming next. MTeichmann will talk to EET to agree on priority concerning the cables production and pulling. It may be necessary for SCS to pull a few cables as soon as possible (by themselves?). Anyway, cables route will be needed.
• This morning the PLCs in the SCS raclroom went off. The information from TS is that the emergency power off button was pushed in the SCS experiment hutch. 

MID

• Has to be clarified when the chiller is going to be back
• Homing test motors in XTD1. Small issues found. Update needed.
• MC2 Motors are working properly
• Workin out commissioning priorities with AE and CAS based on the cables availability

HED

​OPT Lasers

​XRO

• Tomorrow the M2 in XTD will be ready for commissioning
• On Wednesday the MOV2 loop in SASE2 tunnels will be updated. CAS is saving the motor parameters right now.

​​XPD

• Screens have been exchanged
• XTD2 photonic science camera is back and working
• XTD10 SQS2 pop in monitor back and working
• All tasks foreseen for the Winter shutdown have been completed.

​VAC

AE

• AE was asked to update MOV1 MOV2 and EPS loops for SASE3 tunnels. This is due to the implementation of a new B4C to be mounted and controlled. ==> MOV 1 will be updated. Also CAS will be asked to save the motor parameters. NO EPS and MOV 2 loops update necessary for now.

ITDM

DET

​CAS

• Data logger schould be working for every component right now.
• On Wednesday the MOV2 loop in SASE2 tunnels will be updated. CAS is saving the motor parameters right now.

AoB

SPB/SFX

• The interlock definitions in the tunnel were updated. They have to be tested.

FXE

​SQS

• Power cut of the primary power line in the rackroom will occur to allow fixing of the emergency power off system (as agreed between TS ans MM last week). Happening today. After completion CAS and AE support is needed to switch the systems back on. ==> Completed. No issues.
• AQS chamber has been removed. The reaction microscope is now being installed.
• New interlock definitions have to be tested and implemented. Some tests were performed and bugs fixed. New tests are needed and planned for Wednesday. 

SCS

MID

HED

​OPT Lasers

​XRO

​​XPD

​VAC

• Mono loop in SASE 3 has to bee updated (22nd). Agreed with involved parties. 
• MOV2 Loop SASE2 also needs to be updated. Agreed with involved parties.

AE

• AE was asked to update MOV1 MOV2 and EPS loops for SASE3 tunnels. This is due to the implementation of a new B4C to be mounted and controlled. This task was not included in the shutdown list and therefore has not been planned. The issue needs to be reported in the operation meeting since a reshuffle of priorities and relative decisions are eventually needed.
• CRLs for SPB will be controllable at the end of March/ beg. April as agreed with the instrument people. 

EET

ITDM

DET

​CAS

• It is pointed out that the CAS policy is not to implement changes in the software on Friday in order to guarantee the smooth operation of components during the week end. This should be also reported in the Operation meeting.

AoB

• 22/23 January Fa. LIEN will fix the sliding doors in the SASE3 hutches.

SPB/SFX

FXE

​SQS

• Power cut of the primary power line in the rackroom will occur to allow fixing of the emergency power off system (as agreed between TS ans MM last week). Happening today. After completion CAS and AE support is needed to switch the systems back on.
• AQS chamber is being removed
• New interlock definitions have to be tested and implemented asap. AE will discuss a date to SQS, although Friday might me a good time.
• SQS is eager to get the new trigger system from desy and AE and help testing it. It was noted that the trigger is linked to a specific SASE.

SCS

MID

HED

​OPT Lasers

​XRO

​​XPD

​VAC

• M2 mirror has been installed, the chamber closed. MCP is being mounted back. 

AE

• VAC loops updates in the tunnels are ongoing and being coordinated with affected groups
• Motion 2 in SASE2 loop needs to be updated. Being coordinated with AKoch.
• SASE3+2 implementation of changes and commissioning ongoing.

EET

ITDM

DET

• During some AGIPD test with cold detector Karabo lost connection with the PLCs. This was due to the installation of new CPUs for the SFX components. For 10 minutes was not clear whether the vacuum was OK and for safety purposes the detector has been warmed up. These works have to be properly communicated next time.

​CAS

AoB

• 22/23 January Fa. LIEN will fix the sliding doors in the SASE3 hutches.

SPB/SFX

• D.23 Cabling ongoing. Minor problems concerning Chicane filling. 10 cables need to be routed through another chicane.  Being sorted out with EET.
• Silicon oil pipe has been fixed and is undergoing pressure tests. So far successful.
• Issue concerning tripping power plugs in D.09. Needs to be investigated (SPB/SFX <=> TS)

​FXE

​SQS

• This week the AQS chamber will be removed. 
• Power cut of the primary power line in the rackroom will occur to allow fixing of the emergency power off system (as agreed between TS ans MM last week). TBaumann will agree the time with TS. Note from MTeichmann: Even though 1 hour was foreseen, for  SCS it took 5 hours. Be careful and plan accordingly.

SCS

MID

• Commissioning ongoing. The homing function of motors in the optics hutch will be tested this week.
• VAC system in the optics hutch operational. Some issues with the control of the ion pumps need support from AE (1 hour). ==> Tomorrow, 14, Marjan.

​HED

• Mono in XTD6 testing ongoing. Piezo and encorders need some adapter. 
• Hi res mono vessel is being opened and will be closed again at the end of the week
• Si- mono vessel does not need to be opened. 
• Commissioning of the OPT hutch starting tomorrow (Loop 1 and 2)

OPT Lasers

​XRO

​​XPD

• All FEL imagers have been replaced (now with Al coating reducing reflections).
• XTD2 XGM gauge has been replaced.

​​VAC

AE

• VAC loops in the tunnels are being coordinated with affected groups
• Motion 2 in SASE2 loop needs to be updated. Being coordinated with AKoch.
• Test with the 3 triggers coming from the machine for the 3 SASEs are ongoing.

EET

ITDM

DET

​CAS

AoB

• Fridays OP/Readiness meeting is not taking place anymore.
• A new Operation meeting is being set up.
• TT will propose a time slot to put a replacement for the Op/Readiness meeting where technical issues and information spreading after the dispatch meetings can follow.
• Issues concerning the handling of the maintenance day on monday has been raised by the groups. Has to be seen how to cope with that.

SPB/SFX

• New Karabo release has been installed.

​FXE

​SQS

SCS

• EET and AE should get in touch to discuss how to speed up the crates fixing. Some details need to be sorted out.

MID

​HED

• Mono in XTD6 need some work. Meeting today to discuss details with AE
• HI res mono needs to be opened. Work being organized for next week.

OPT Lasers

• The power glitches l2 nights ago caused a power loss between 1am and 7 am. The seeder in SASE 1 went off. It is being switched on again but this reduces considerably its lifetime. No problems in SASE3. Tha's probably because different protection voltage drops are used (20V/40V) for the fuses. TS will look into that to understand why and whether the situation can be improved.

​XRO

• M2 in SASE3 goes back this week and will be pumped during the week end.
• Vac group busy ventin sections to replace imagers,

​​XPD

​​VAC

AE

EET

ITDM

DET

​CAS

AoB

• Power cut in the SQS/SCS rackroom needed to fix the emergency power off system. 1 hour needed for each rackroom. The redundant power line will not be affected. The following dates have been agreed:
  • SCS - MTeichmann will bring the PLCs ina safe state and give confirmation to TSchoen within 1 hour today
  • SQS - Power will be cut next Wednesday morning during the chamber switch. Exact time will be agreed next week.
  • Optical lasers are not involved in the power cut.

SPB/SFX

• CRLs will be brought in the SPB lab in XHQ. The vacuum group is installing the replacement pipe.
• SFX cabling starting today.

​FXE

​SQS

SCS

MID

​HED

OPT Lasers

​XRO

• M2 in SASE3 is being replaced and the mechanics need to be re calibrated.

​​XPD

• FEL imager screen in XTD2 has been replaced.
• FEL imager screen in XTD10 will be replaced today or tomorrow.

​​VAC

AE

• SASE2 VAC loop has been updated. 
• The VAC group requested an update of all the motion loops in the tunnel due to some problems with the encoder readout. In principle this could be done next week, but AE will meet with VAC and OPT to discuss the details before.
• MID/HED monos being commissioned.

EET

ITDM

• There was a network update last week
• This Wed/Thu the Maxwell clusters will be updated. No offline computing possible.
• Most of the activities planned for the WS have been done.
• Installation of new storage for SASE1 will be done in thenext days.
• OS in the PCs in the control rooms need to be updated. This will be coordinated with the intruments.

​DET

​CAS

• CY is now back working on the Mpots (SASE2 and 3). Changes have been implemented. SQS to perform tests. Documentation has also been updated and will be circulated probably next week.
• Karabo training has been organized. The agenda will be circulated soon.
• SPB Karabo upgrade has to be agreed with SPB/SFX.

AoB

• Fridays OP/Readiness meetying moved to 9.30 from now on.

SPB/SFX

• SFX part1 cabling preparation ongoing (WCBs mounting and labeling). Good progress.
• CRLs will be brought out of XTD9 tomorrow.
• Scaffolding below D.23 has been removed. An official communication that now the area is completely accessible is still lacking.

​FXE

​SQS

• For the loops 9-13 the HW is ready. CAS projects have to be prepared.
• Some issues with middle layer devices have been notified to CAS (Cyril)

SCS

MID

• Karabo 2.3.2 has been deployed apart from the AGIPD control (waiting for tests by DET group before update).
• CY is now back working on the Mpots (SASE2 and 3)

HED

OPT Lasers

​XRO

​​XPD

​​VAC

AE

• SASE1 power was back yesterday. SPB and FXE have been brought up again yesterday afternoon. Today morning because of unplanned and badly communicated MPS works the power was cut again ==> additional, unplanned efforts needed.
• Hazem will be OCD during Christmas, only for PLCs. 
• No uTCA OCD during Christmas.

EET

ITDM

​DET

​CAS

• 2.3.2 Deployed in SASE2
• SPB update postponed to January
• SASE1 planned for today
• No OCD until 09.01.

AoB

• Next Disptach meeting on 07.01.2019 at 9:30

SPB/SFX

• SFX part1 cabling preparation ongoing (WCBs mounting and labeling)
• waiting for the power to come back

FXE

​SQS

• Some interlocks need to be activated. Discussion with AE ongoing. Will be probably done early next year.
• Loops updates are being discussed/executed plus CRs status are being checked.
• pnCCD CRD is being prepared. Information like what electronics can go in the Rackroom and what in the huctch due to cable length restrictions will be collected and spread to the relevant people (e.g. safety)

SCS

• Successful beam time with users - many thanks to all the support groups
• Fixing of crates and additional cabling is being prepared/ongoing.
• DET group asked for a coordination meeting concering users requirements for the fast ccd in the next users run.

MID

• Cabling ongoing
• Support from AE to modify the Racks that will host the electronics for the MID and HED monochromators has been requested. PGessler will check whether support can be granted this week (other activities ongoing). The goal is to commission the two monochromators in the tunnel in the shutdown.

HED

OPT Lasers

​XRO

• MCP in XTD 10 has been removed. Also the M2 has been removed. The reference positions will be lost when the components are back (14.01.).

​XPD

• Screens in SASE3 are being installed. Venting of the line is being coordinated with the vacuum group. No reference positions lost is expected.

​VAC

• CRL in XTD 9 will be removed and stored in a labeled box, waiting to be installed in the SPB hutch. This will happen after the winter shutdown. Procedure to safely handling the berillium lenses is being finalized with Safety.

AE

• Once the power in SASE1 XHEXP1 is back AE needs 1 day to bring the electronics back.

EET

ITDM

​DET

​CAS

• Hotfix is being prepared. This will solve issues in the config tables related to units.
• Deployment schedule is being updated (inc. deployment of 2.3.1. in SASE2)

AoB

• General interest precutions concerning components should be circulated, if any, by the responsible groups in view of the holyday period.

SPB/SFX

• Radiation damage tests were successful. 
• Instrument is set for the power cut.

FXE

• According to the information available to AE the instrument should be set up. 

SQS

• Successful users run in the last 2 weeks
• Some interlocks need to be activated. Discussion with AE ongoing
• Loops updates to be discussed/executed plus CR status will be checked
• Triggers requests are being sicussed with BF

SCS

MID

• Cabling ongoing

HED

• Loop 1 will be operative probably today. CAS support for creation of scenes has been asked.
• Cabling ongoing.
• Additional Racks in A27 have been installed

OPT Lasers

• As soon as the system is ready for the power cut in SASE1 a mail will be sent  to PSPO.

XRO

​XPD

​VAC

AE

• XTD6 crate 219 has to be rebooted.

EET

ITDM

• Maintenance and updates (e.g. online clusters ) has started  as indicated in the shutdown activities planning. Should go on until the end of the week.

DET

​CAS

• 2.3.1. will be deployed in SASE3 tunnels today
• Schedule for 2019 has been circulated by HS to the readiness mailing list

AoB

SPB

FXE

• Timing tool has been tested during the week end. 
• PP laser inch modes have been tested with the Gotthard detector.
• Intra/inter train gitter test results being analyzed.
• APG mode of the digitizers had issues. Experts from AE will be contacted.

SQS

SCS

MID

• AGIPD saw light. Support groups, in particular AE and DET were thanked.
• Beam stop saw also light. 
• Small issues have been / are being fixed: leak of the silicon oil pipe, timing issues.

HED

XRO

​XPD

• CAS will run some test in SASE3 for the diagnostics after the latest time server development.

​VAC

AE

• PP laser motors issue reported onm friday has been investigated. It seems that the motor was emchanically stuck and it was possible to implement a workaround (more power fromt he controls) to make it work. It needs to be further investigated in the shutdown.

  EET

ITDM

DET

​CAS

• 2.3.0. has been released. Hot fix for minor issues (e.g. back compatibility of the GUI) is being prepared. This does not affect the tests made by AE, DET and ITDM
• FOr sase1 PLC data has been recorded in preparation to the power cut.

AoB

Power cut in SASE1  

• The plan is to cut the power in SASE1 in the XHEXP1 on Wednesday 12.12. around noon.  This will only happen after PSPO will receive feedback from AE, ITDM, FXE, SPB and Optical Lasers that everything is in a safe state.
• The power could come back on Thursday (13.12.) morning if no problems occur.
• The normal light could be affected. In any case the emergency lights and Radiation lights will be working. Working in SASE1 hutches and Rackrooms in this light condition is not safe w/o additional light and the suggestion is to postpone the activities when the light is back.
• If the time plan needs to be changed due to the radiation damages tests being delayed, the relevant parties will be informed by PSPO.

SPB

• pulse arrival time measurements to be done this week.
• second Gotthard detector small issues being fixed (online data viewer fixes - ongoing)

​FXE

SQS

SCS

MID

• Loop 10 partially needed. Smaract stages are working withing loop10 since yesterday

HED

XRO

​XPD

​VAC

AE

• AE agreed to power down the PLCs in SASE1 experiments and optical laser before the power interruption. Dates are being rediscussed since radiation damages tests are foreseen for Monday and Tuesday! (10-11.12). TBC.
• The synch of the Gotthard detectors with 1 trigger does not work for long trains (synch is lost). Multiple triggers are needed and also possible (AGaller is already using them). AE can give support also to the other users.

  EET

ITDM

DET

• SCS fast CCD temperature drift issue being addressed.

CAS

• Tests of the 2.3.0. Karabo version are ongoing.  Karabo update for SASE1 will be deployed after the  PLCs are back again (ca. 2 days after the power is back). Proposed date by CAS is 19.12. but needs to be discussed again once the power is back.

AoB

SPB

• pulse arrival time measurements to be done next week.
• second Gotthard detector small issues being fixed

​FXE

SQS

• Successful experiments during the WE

SCS

• Z-motion issue for the fast CCD. Has to be checked again with AE today

MID

• Loop 7 is running
• This week light is expected on the AGIPD
• Mono commissioning issue: the pico motors position is not reproducible. ==> motor is now stuck, needs to be replaced.
• Loop 10 partially needed. To be discussed with AE

HED

XRO

​XPD

​VAC

AE

• The synch of the Gotthard detectors with 1 trigger does not work for long trains (synch is lost). Multiple triggers are needed. AE is looking into that involving A. Galler since he is an advanced user of the Gotthard. The solution will then be deployed to all the detectors.

  EET

ITDM

CAS

• Tests with AE/ITDM ongoing. In the new version of Karabo/DAQ the filtering of slow data for device will be included.

AoB

SPB

•  commissioning ongoing
• Radiation test was not successful since the beam diameter was smaller than expected. 

​FXE

SQS

SCS

MID

• Loop 7 will run by today evening
• Mono commissioning issue: the pico motors position is not reproducible. ==> the issue solved by itself. Tests ongoing.
• AGIPD commissioning is ongoing. 

HED

XRO

​XPD

​VAC

AE

• In sase3 the y axis of the nano motion system (sample environment) does not work properly uinder vacuum. JTolkien and C Dieter are on the issue.

EET

ITDM

CAS

• Release tests ongoing. Release candidate being tested with AE.
• OCD for SASE3 users experiment has been set up

AoB

SPB

• KB system commissioning ongoing
• Agipd commissioning stopped
• Radiation tests preparation ongoing.

​FXE

SQS

SCS

MID

• TueV Test successful.
• Mono commissioning issue: the pico motors position is not reproducible. This is a show stopper for the Monochromator commissionig. Not clear whether also FXE has the same kind of problem. AE has been informed and will try to support taking into account also the requests by SASE3 for this week (commissionig for the users run)
• AGIPD commissioning is ongoing. Chiller has to be replaced due to a leak.

HED

• TueV Test for the optics hutch successful
• Cabling ongoing. 
• Fw export and crates/loops preparation ongoing
• Screen is needed for first beam.

XRO

​XPD

​VAC

AE

• OCD: intervention request during the WE to fix a train id issue. Should be ok now but the problem still needs to be reproduced and undertstood.

ITDM

• Network intervention needs to be planned for the beginning of January. KW is clarifying the details. Will be included in the shutdown planning.

CAS

• Release tests ongoing. Small issue with the DAQ has been found. Fix already in place. Final tst of the fix is being performed. 

AoB

SPB

• KB system and AGIPD commissioning ongoing

​FXE

SQS

SCS

MID

• TueV Test ongoing.
• Mono commissioning to be done this week.
• Loop 7 is needed for the middle of next week. As of now no fw export and cables for this loop appear to be there. AE/EET will have to check whether this makes sense.
• AGIPD commissioning:
  • It seems that some cables needed for the loop 11 were not connected and some other are not yet installed. ILobato and K. Sukharnikov are now dealing with the issue. This delays the foreseen plan. The mitigation strategy for the missing cables issue is to have a list with these specific cables and ask Fa. Schulz to put these hight priority. It already worked with other cables/connectors. Discussion MID/EET/AE needed.
  • uTCA crate is now operational.
• It is pointed out by CAS (C.Y.) that component having priority concerning cables and loops operation are AGIPD, DES, PDS, Mirror.  It has to be checked whether these are aligned with the actual priority list for loops and cables discussed with MID. 

HED

XRO

• in XTD10 horizontal slits were moved to better scan the beam.

XPD

​VAC

• in XTD9 the B4C aperture in front of the monochromator grating was changed to a larger one. 
• in XTD10 the current amplifiers for the XGM were successfully changed.

AE

• OCD: intervention request during the WE to fix a train id issue. Should be ok now but the problem still needs to be reproduced and undertstood.

ITDM

CAS

• Release tests with ITDM ongoing.

AoB

SPB

• Gotthard needs to be tested.
• Pulse picker will be used and commissioning still has to be done.
• Sample chamber encoder drift issue. Has to be investigated further. 

​FXE

SQS

• Beam during last WE: no major issues
• Interlocks will be updated tomorrow
• Some issues concerning direction of motors need to be fixed with AE.
• AVC300 valve. Firmware will be available this week.
• Digitizers and photonics camera are needed. Work ongoing

SCS

• Motors for the Fast CCD cameras are not working properly and interlock about pressure for the detector is needed ==> Loop 7 needs to be updated, AE will take care of that.
• When the CCD will inject data into the DAQ ITDM will be informed.
• There was a request to CAS to provide values for the Monochromator at a rate faster that 10 Hz. This requires evaluation of the feasibility by the experts (CAS/AE). For the time being the middle layer will not be changed. 

​MID

• AGIPD commissioning:
  • Loop 11 needed. Today tests by AE. Until Tue. evening all the intelock tests should be done. From Wed. on operation.
  • uTCA crate needs to be operational too. According to AE can be done.
  • Thu. on: DQA tests.

HED

XRO

• Temporary solution concerning Karabo topics management to alllow the reading of the Monochromator values by SCS. The definitive fix will come in the future.

XPD

• XTD10 XGM repaired
• XTD1 ph. science camera was replaced with the one previously installed in SASE1. Repair ongoing.
• the company went bankrupt and was bought by a new company. For info contact A. Koch.

​​VAC

AE

• OCD: intervention request during the WE to fix a train id issue. Should be ok now but the problem still needs to be reproduced and undertstood.

ITDM

CAS

• Release tests with ITDM ongoing.

AoB

SPB

• Lopp 5 update - will be agreed between SPB and AE
• Smaract limit switches issues - AE will have a look. In the meanwhile also the company of the controlles has  been asked by SPB
• KB motor steps have different size with respect to encoders. This issue is known and a workaround / solution has to be discussed with AE offline
• Pulse picker will be used and commissioning finalized.
• Sample chamber encoder drift issue. Hasx to be investighated further. AE can support.

​FXE

SQS

SCS

​MID

HED

XPD

​​VAC

AE

• MID Loop 11 - Fw received. Being tested.
• PLCs for optical lasers in SASE3 and 2 control being set up
• SASE3 support for commissioning ongoing 

ITDM

CAS

AoB

SPB

• More interlock updated will take place this morning.

​FXE

• Supporting the diagnostic and optics group for the testing of the beam feedback mechanism. 
• A.Galler was informed by CAS (Vajid) that a Karabo update was requested to CAS last friday late afternoon. The update was stopped by FXE, but to avoid this in the future communication between the cvarious groups has to be improved.

SQS

• Beam during the WE. NO issues.
• Imaging processors are being used for cameras. There are issues with the averaging. CAS is informed.
• Various loops updates needed. Will be agreed offline with AE.

SCS

​MID

• Loop 11 could be ready by next week. Today the FW export by EET is going to be available to AE.

​HED

XPD

​​VAC

AE

TDM

CAS

• Karabo 2.3 is now a release candidate. Testing starts today (2.5 wks internal tests). Info about new features of the 2.3. release are available on the white board in front of the mockup control room.

AoB

SPB

• Karabo was updated yesterday. Test is ongoing. So far no issues.
• The macros for cooling down and warming up the detector do not work properly. If the button is pushed twice the setpoint changes continuing e.g. cooling down too much. Marcin will bring the issue up with CAS (HS).

​FXE

SQS

SCS

• Issue was found with Elmo controllers crates: due to a problem in the EPLAN macro the ground is not connected to the motor cables. The mitigation strategy will be discussed by PG, LW, MF and MT in a meeting.

​MID

• AGIPD will move in the hutch today.
• PCs are being installed. Karabo need to be deployed.

​HED

XPD

​​VAC

AE

TDM

CAS

• Karabo 2.3 is now a release candidate. Testing ongoing.

AoB

SPB

​FXE

SQS

• Crates for Loop12 have been installed. Loop will probably run by today or tomorrow.
• Loop 2 and 3 need to be updated because of interlock definitions.
• Issue with  valves behavior in Loop 3 and 5. AE will check.
• Encoder w/o 5 V. BB checks.
• Pfeiffer Motor valve issue will be fixed by next week.

​SCS

​MID

• AGIPD will move in the hutch this week. Preparation ongoing.
• Loop11 will be ready early next week

​HED

XPD

• Several tests planned for the end of the week, including the feedback device by CAS. PRCs will be made aware of the activities in the tunnels.

​VAC

AE

• PPL update in SASE1 today.
• Update of the Loop 5 (AGIPD) tomorrow

ITDM

CAS

• Issues in the calibration pipeline will be coordinated with the relevant groups (DET, AE) before and during the next chapter's meeting.

AoB

SPB

• A smaract stage sensor had issues. The behavior changes whether the chamber is pumped down or not. The Issue is being addressed by AE.

​FXE

• Users are acquiring data with the Jungfrau detector. It is a success so far.
• Successful users shifts.

SQS

• Configuration of some motors is incorrect. BB looking into that.
• It is required to change the sampling rate of the digitizers. CAS is on it.
• MPOT issue: control is lost as a power supply channel trips. CAS: a fix exists, needs to be deployed.
• Note from NC(AE): one has to check that the parameters and information written in the CRDs are consistent with what is being used in reality. This would make the time of the OCDs more efficiently used.

​SCS

• Interlocks tripping too easily: CAS will show SCS how to change the thresholds.
• KB coordinated motion: bug has been reproduced by cas. Fix ongoing.
• Full control of the scroll pumps still not possible. AE is working on it.
• Loop 13 will be fixed (miswiring problems)

MID

​HED

XPD

​VAC

AE

ITDM

• Issue with the office netword socket in D.09. Fixing ongoing. In general all the operation critical devices have to go to the control network sockets to avoid such issues.
• Request from HS to add the generic XFEL unix user to the special group of doocs to change the bunch pattern. This has to be first discussed internally to CAS. AS will come back to KW and the others involved players as soon as this has been clarified in CAS.

CAS

• CAS need  a time window to deploy and test the fix for the Lima Basler cameras in XTD6. No feedback since last friday. A.Koch tries to find out a possible time window and come back to ASIlenzi.

AoB

Groups are reminded to send GW (PSPO) the activities planned for the winter shutdown.

SPB

​FXE

• During the last users run there was a loss of optical laser. This was due to a mistake in CAS. Has been clarified already. It is pointed out by FXE that one should avoid  working on e.g. the control system during the users period. This is however, not always possible and a mitigation strategy is needed and requires discussions.
• Successful users shifts.

SQS

• Issues with motors have been fixed.
• Loop 3 will be probably updated today.
• In general several loops need to be updated, will be coordinated with SQS.
• SQS has the night shift today. Eventual tests have to be done today during the day.
• Motor valve can be implemented in the controls next week.
• Would be nice to have the GATT pressure in our DAQ system. CAS is looking into that.

​SCS

• KB coordinated motion has been improved

MID

​HED

XPD

​VAC

AE

ITDM

AoB

Another issue related to the non independency of interlocks between the SASEs: FXE waiting for pulse on demand. In the meanwhile in SASE3 the pressure in the GATT increases and the apertures go in to keep the pressure high. As designed, the interlock stopps the beam for 1 or 2 seconds, but also in SASE1. FXE loses the pulse on demand. 

It is clear that this is a complex environment and finding a solution requires time and the experts both on the desy side and XFEL side (e.g.AE, CAS) on the table.

SPB

• Issues with the detectors pipeline occured (also for FXE). In the future a test button to check the status of the pipeline wil be implemented.

FXE

• Pulse on demand pump and probe experiment successfully performed.
• The support groups got  the Jungfrau detector working for the users run. 
• VAC sample environment  working.

SQS

• MPS conditions update: Loops will be restarted tomorrow. Tests to then be done in the afternoon and on Wednesday.
• Loop 4 and 5 motors need to be updated . To be discussed with AE when is the best time.
• Pulse on demand capability needed. SQS will contact the CAS contact person.
• Ph. Science camera can be connected again.

​SCS

• First shifts succesfull. Support from the support groups is appreciated.
• DAQ groups that have been overlooked are now needed. ITDM asks SCS to send a ticked with the request and the required excel sheet.
•  ITDM also points out, that only DAQ groups that have been already properly commissioned should be used during night shifts.
• Minor problems with gauges. Will be discussed with AE.
• Problems with motors (sometimes they work, sometimes not) has to be discussed in details with AE.

​MID

​HED

XPD

​VAC

AE

ITDM

AoB

SPB

• KB motors issue: the motors do not go to the proper position. This has been communicated to AE. Analysis of the problem and fixing ongoing.

• The leak of the AGIPD colling pipe has been fixed but the connectors were not cleaned since this requires time. It was decided to operate the detector and postpone the cleaning to the next shutdown.

• Software update for the AGIPD should now avoid problems with the configuration of the detector.

FXE

• Pulse on demand pump and probe experiment successfully performed.
• The users would request the Jungfrau detector working for friday. Tomorrow there will be a meeting with FXE, DET,AE and the users to manage the expectations and discuss what can be delivered in terms of detector performances.
• Von Hamos spectrometer motors sometimes go out of service. The origin of the problem is not clear. AE will investigate.

SQS

• MPS conditions update: Loops will be restarted next tuesday
• Issues concerning the motors of the newport stages have been fixed.

​SCS

• Coordinated motion does not work properly. The bug identification and fixing will be done offline in the lab. TF is preparing the test environment with 2 stages and Elmo controls. AE/CAS involved.

​MID

​HED

XPD

​VAC

AE

ITDM

AoB

• Yesterday a HZDR electrician had an electircal accident working in A23. He only got a burn, but he is fine. The accident report is being filed. After A23 has been checked it was opened again for the laser people. 

SPB

• Issue with the detector cooling pipes during the week end required intervention. Now fixed.

• Commissioning ongoing.

• Interlock definitions to be deployed probably tomorrow.

FXE

• Pulse on demand capability successfully tested
• Issue with the DAQ: crush of data aggregators. ITDM OCD has been notified and the issue is being investigated.
• The users would request the Jungfrau detector working for friday. FXE asked for feedback about the status of the detectors integration to DET, CAS, ITDM. The feedback is needed to inform the users.

SQS

​SCS

​MID

• Loop 4 is waiting for ion pumps cables to be commisisoned. as soon as the cables are there AV will inform CY and MID.

HED

XPD

​VAC

AE

• SQS/SPB interlocks will be propbably deployed tomorrow.
• SASE3 master loop will be updated tomorrow.

ITDM

• ITDM needs to know until when the photon commissioning in SASE2 tunnels is ongoing to update the DAQ servers. The information should come from HSinn or PRCs, who are not at the meeting. JS will try to contact them directly.  ==> HW was updated but still with the old configuration. Moving of data services to new aggregators needs discussion.

SPB

• Problems with the GUI not being accessible after update solved with the CAS OCDs 

• Loop 5 and Master loop will be updated today. New MPS conditions included.

FXE

• No issues

SQS

​SCS

• KB coordinated motion control has issues. CAS will ttry to reproduce the bug to make sure it's not Karabo related and eventually pass the issue to AE

MID

• Loop 1 tech. commissioning done. Loop 4 is waiting for Ion pump cables to be delivered by Schulz.
• Loop 2 and 5: beckhoff crates installed. As soon as it is needed the firmware can be uploaded.

HED

XPD

​VAC

AE

• SASE3 EPS loop being updated to comply with the new D3 directives. EPS being updated. Then final tests needed.

ITDM

• ITDM needs to know until when the photon commissioning in SASE2 tunnels is ongoing to update the DAQ servers. The information should come from HSinn or PRCs, who are not at the meeting. JS will try to contact them directly.  

CAS

• To preventy the copy of entire projects with all the relative subprojects and additional problems, the GUI servers in SASE2 have been updated. This is being done also for the other beamlines.
• The milestones of the SASE2 Technical commissionig Task Force headed by CY have been achieved. Starting from today CY is no longer taking care of the SASE2 tunnel activities planning and coordination. It has to be discussed in the Friday meeting how to go on, since the planning and coordination of the shutdown activities is needed.
• Concerning the SASE3 radiation protection issue in the tunnel, it is stated that CAS are implementin the requested software solution, although Karabo was not designed to manage personal safety functions. To this aim, it is needed to understand whether the concept is going in the right direction. The large scale issue is being managed by Safety (ZA). A report about the situation will be given on Friday by THaas.

SPB

• Ready for users.

• DAQ tests to be done on the SPB side.

• MPS definitions being defined.

FXE

SQS

• Loop 7-10-14 are running. 13 and 12 to come.
• Crates for the loops 1/4/5 have been/are being modified.
• Interlocks definition update. NJ, BB and SQS are clarifying the requirements.
• Interlocks not working for the loops 3 and 5. Info sent to AE. Being worked on.
• Still no beam available

​SCS

MID

• No issues.

HED

• Beam in XTD6
• Some CRLs arms are deactivated to avoid accidentally focusing on the shutter.

XPD

​VAC

AE

• MPS update for SPB is ready and has to be done. This includes the interlock definition for the detector's z motion.==> being reviewed by SPB.\
• SASE3 EPS loop being updated to comply with the new D3 directives. Today a test with Z.Ansari will be performed.

ITDM

CAS

• To preventy the copy of entire projects with all the relative subprojects and additional problems, the GUI servers in SASE2 will be updated. Only 1 server is running at the moment.
• The milestones of the SASE2 Technical commissionig Task Force headed by CY have been achieved. Starting from today CY is no longer taking care of the SASE2 tunnel activities planning and coordination. It has to be discussed in the Friday meeting how to go on, since the planning and coordination of the shutdown activities is needed.

SPB

• Maintenance of the AGIPD ongoing.

FXE

• No issues to report.

SQS

​SCS

MID

• Loop4 is ready but after the electrical shutdown form last week the PLCs racks aha to be re initialized (not powered right now). This is done by AE and is ongoing.

HED

• HR mono commissioning ongoing. Being pumped down.
• Pulse picker tested. Logic has to be slightly changed. CR will come soon. 

XPD

• FXE diamond screen for beam position measurements  has been installed in XTD9. Section being pumped this week.

VAC

• 3rd mirror chamber  in SASE2 being pumped now.
• XTD6 all vacuum gaps are closed.
• FXE MONO will be pumped probably on Wednesday

AE

• SASE3 Master loop has to be updated. 
• MPS update for SQS being worked out. Conditions have been clarified with SQS.
• MPS update for SPB is ready and has to be done. This includes the interlock definition for the detector's z motion.

ITDM

• Commissioning SQS - CAS support needed to configure the DAQ for the cameras.
• DAQ links for Basler cameras in MID are needed. This will be done asap.

​CAS

• 2.2.6. hot fix will be released probably today. It will include fixes for the DAQ middle layer device.

AoB

Status of the redundant power for the racks in the SASEs according to Technical Services:

• All the works have been successfully done: the foreseen racks get now 2 working redundant power lines.

• The power is back and no further interruptions are planned/needed.

• This week a visual inspection and relative "Abnhame" will take place, without any actual electrical works being involved.

• All eventual / additional interventions will be scheduled for the winter shutdown.

SPB

• Maintenance of the AGIPD ongoing.

FXE

• Beamtime was successful
• FXE imagers cannot be seen by the people in the BKR and that's a problem to efficiently communicating with them about e.g. beam stability issues. According to CAS a Karabo client is available for that purpose. The issue should be addressed by the PRC with the help of CAS. 

SQS

• Interlock definitions - clarifications with AE ongoing.
• Loop 7-10-14 being deployed
• Some crates for the first loops were taken off for modifications. They will be back this week.
• Trigger signal has to be provided by AE. TB will ask BF today.

​SCS

• Some cables for XGM had to be pulled last minute. In the XGM foreseen crate there was no space for additional terminals and another BKF module was chosen. Now some cables are too short. MTeichmann will adress the issue to the EETf
• Fast CCD interlock definitions: MT will check whether SCS is on schedule. Info needed by the DET group.
• Digitizers expected this week. 
• DAQ tests will be done.

MID

• Loop4 is working now. VAC section being pumped.
• Request to CAS for GUI training. Training will be announced for the 04.10.

HED

• HR mono commissioning ongoing. Some switches will be recabled now by KS. Has to be done asap. Motors direction has been checked and is correct.
• Pulse picker will be ready tomorrow. ==> Since all the tunnels are open at the same time, there is a conflict of resources pointed out by CAS.

AE

• SASE3 VAC loop will be updated again today, then SASE2 VAC
• MPS update for SCS and SQS being worked out. Today updates will be ready.
• FXE MONO the damaged crates will be changed and loop will be update today. Commossioning starting as soon as teh motors are replaced (today or tomorrow)
• Scroll pump feedback issue: the feedback signals coming from the controllers were not correctly read. AE foud a solution and already presented a CR fo fox the crates.==> the time slot has to be agreed with SQS and SCS.
• New framework should be deployed forst in the SA2 tunnels. Will be discussed offline with CY.

ITDM

​CAS

SPB

• Last monday shift was cancelled since there were prooblems with the cooling of the AGIPD. Not possible to reliably cool it down. Maintenance of the cooling units plus investigation of the controllers ongoing.

FXE

SQS

• Interlock definitions - clarifications with AE ongoing.
• Loop assignment changed. Next loop to be brought up is Loop 10 (probably by next week by Hazem)
• Network storage for the control PCs in the control room. Discussion ongoing (data policy, accesibility...). Is being done offline and will be reported to a larger audience when clarified.

​SCS

• Interlock definitions being finalized. Update of the loops will be done soon.

MID

• Loop 1 deployed and technical commissioning of the OPT hutch is ongoing. No major issues.
• Loop4 needed for next week.
• Request to CAS for GUI training. CY and AS have to clarify who organizes that in CAS. CY understanding is that Hugo Santos is going to do it. Will be discussed offline.

HED

• Rack 2.19 shows messy cabling. KS will contact UBrueggmann to understand who is responsible for the devices in the racks. 

AE

• This week all the modified crates for SCS and SQS will be reinstalled and loops updated by BB

ITDM

• The upgrade of the DAQ in FXE was postponed to the next week on request from FXE.

​CAS

• Karabo 2.2.5. patch being released today

SPB

• Bug in the AGIPD cooling macro (setpoint resets itself to -59 C)  -  Jola knows about the problem. AS: Macro will be fixed.

• AGIPD dark calibration: when new darks were injected with calling Steffen's script "xfel-calibrate",  error message "Disk quota exceeded" popped up caused by a GPFS-related issue -> issue was solved. In general the script was not tested (scratch space was used!). One should not release script without ITDM knowing and haveing thested it before. 

FXE

• Commissioning going on in the last days. Not able to get 14 keV beam for next users experiments. They will stick with 9.3 keV.
• Karabo stability issue (GUI) reported in the commissionig logbook.

SQS

• Problem with the digitizers bandwidth has now been solved.
• Loop 6 running w/o smaract stages.
• Flow switches issues solved . Rewiring of PLC needed
• Pfeiifer flow valves ready to be stested this week.
• Loop assignment changed. Change request handed in. As soon as it is approved it will be clear what loop to bring up next.

​SCS

• Loop 7 is running. Needs to be connected to Karabo.
• Loop 8 running too.
• Readiness of loop 1 and 4 ==> Interlock conditions need to be implemented. 
• It is unclear whether the homing capability has been implemented already and can be used. Discussion ongoing. The use case has to be clarified.
• Network storage for the control PCs in the control room is requested. Scope definition and discussions with CAS and ITDM will be done offline.

MID

• Loop 1 deployed and technical commissioning of the OPT hutch is ongoing. This week Slit and ATT will be commissioned. No big issues so far.
• Loop 4 being prepared by AE. Will be ready next week.

HED

• Work in tunnel ongoing
• HRMono- ready to start grouting. Mono will be here next week.

DET

• FXE/SPB GPUs crash issue  - to be checked with ITDM - ITDM was not contacted.

ITDM

• DAQ was upgraded to the user friendly version in SASE3. After having discussed with FXE in the meeting, it was decided to install it also in FXE. 
• DAQ training requested by SQS

CAS

• Karabo 2.2.5. patch being prepared. Will solve minor bugs and the crash due to empty data files.
• 28 Mpot update requests were received by CY. Need to be planned and coordinated with H.Santos.

The GUI crashes reported by FXE will be fixed with the hotfix release coming ~tomorrow.

It is related to very high or low values in plotting widgets scratching the maximum floating point number (1e308 ~ infinity).

-- Wajid & Dennis

SPB

• Testing of the 4 images preview mode for AGIPD will be done today.
• Interlock definitions are being updated. Loop update will be done after the users experiments.

FXE

• ​No issues to report

SQS

• Problem with the digitizers bandwidth has been addressed by AE. ITDM has to reconfigure the DAQ. ==> Some issues still present. It seems because of lack of communication.  AE/ITDM/CAS/SQS meeting tomorrow to clarify.
• Loop 6 being prepared. A couple of issues with Smaract and Technosoft controllers have to be addressed with the involvement of AE and CAS. ==> Still ongoing.

​SCS

• Loop 7 is running since this morning. More loops needed.
• Missing crates for Loop 1 and 4 are finishing to be tested and will be installed back soon (Loop 1 crate already today). 
• It is unclear whether the homing capability has been implemented already and can be used. AE to check. 

MID

• Loop 1 still not deployed and needed. Plan is to commission the MID Opt hutch until the end of next week.

HED

• Work in tunnel ongoing
• CRL to be tested today
• HRMono- ready to start grouting.

DET

• FXE/SPB GPUs crash issue  - to be checked with ITDM

XRD

• MCP in SASE2 have problems and the section has to be vented. The detector will be replaced on 23rd of Sep.

ITDM

• Office network switches will be upgraded. Intervention cancelled! Will be probably done in the Winter shutdown.
• DAQ to be upgrade to the user friendly version also in SASE3. ITDM to ask FXE whether they also want it.

SPB

• DAQ was updated and no big issues were found.
• Some issues are being discussed with CAS and ITDM. Among them also the request to add more channels to the DAQ.
• Interlock definitions are being updated. Discussions with AE ongoing.

FXE

• ​The beam position at 14 keV  was significantly changed (horizontal offset by ca. 600 um) with respect to the position used before at 9.3 keV. The realignment of the beam took time. Test with LPD will be done today if no other issues with the beam are found.

SQS

• Loop assignments will be re-checked with EET and NCoppola. After that the decision about what loop to bring up after Loop 6 will be made.
• Problem with the digitizers bandwidth has been addressed by AE. ITDM has to reconfigure the DAQ. 
• 4 uTCA boards (For the reaction microscope from the University of Frankfurt) are found to be wrong: the firmware seems to be incopatible with the intergration in Karabo. ==> AE confirmed that a different firmware can be uploaded and this will solve the compatibility issue with Karabo.
• Loop 6 being prepared. A couple of issues with Smaract and Technosoft controllers have to be addressed with the involvement of AE and CAS.
• New interlock definitions list hase been sent ot AE.

SCS

• Interlock definition were sent to AE by M. Izquierdo. M. Teichmann is discussing the new interlock definitions with Nerea Jardon. 
• FAst CCD interlock definitions being discussed with the detector group.

MID

• Loop 1 can be probably deployed today

XRD

• As already reported on Friday, In sase 3 tunnels the new interlock definitions do not affect the imagers, that remain unprotected from high intensity beam. This will be discussed by XPD with HS this week.
• Reflection on the imagers issue: a solution involving a new coating has been found. If you have this problems, please contact AKoch for the implementation of the solution.

​CAS

• MPOT power supplies: CY got feedback from DET, SPB, SQS, SCS. Next monday a mail with the scheduling of the activities will be sent.
• Eventual requirements for features to be implemented in the next Karabo release should be sent and discussed with CAS.

ITDM

• Office network switches will be upgraded. Please refer to the previous entry in this logbook for the details.

HED not represented.

The office switched reported below have to be upgraded. Some of them belong to accelerator and some to us. Below you find the detailed list of the switches and the time schedule (assuming everything runs fine) for the update. ITDM needs feedback the latest today evening.  

if you see problems with the upgrade, please reply to this e-mail. If we do not receive complaints until this evening, we assume that everything is fine.

Monday, 24.09.2018, 06:00 - 08:00 am:

sw-hh-al-324.desy.de    XFEL_XHEXP1_SASE2_A21.UH3.03

sw-hh-al-325.desy.de    XFEL_XHEXP1_SASE2_A22.UH7.02

sw-hh-al-323.desy.de    XFEL_XHEXP1_SASE3_E21.UH2.03

sw-hh-al-322.desy.de    XFEL_XHEXP1_SASE3_E22.UH3.01

sw-hh-al-274.desy.de    XFEL_XHQ.1.174

sw-hh-al-298.desy.de    XFEL_XSDU1_UG01.001_R01_3175m

sw-hh-al-295.desy.de    XFEL_XTD1_SA2.R002_R26_2238m

sw-hh-al-296.desy.de    XFEL_XTD1_SA2.R005_R16_2384m

sw-hh-al-297.desy.de    XFEL_XTD1_T3.R007_R33_2500m

sw-hh-al-299.desy.de    XFEL_XTD1_T3.R010_2.1_2581m

Tuesday, 25.09.2018, 06:00 - 08:00 am:

sw-hh-al-287.desy.de    XFEL_XTD10_3.13_3297m

sw-hh-al-285.desy.de    XFEL_XTD10_3.4_3141m

sw-hh-al-286.desy.de    XFEL_XTD10_3.7_3209m

sw-hh-al-290.desy.de    XFEL_XTD2_T4.R009_R14_2590m

sw-hh-al-293.desy.de    XFEL_XTD3_T5.R005_R01_2846m

sw-hh-al-292.desy.de    XFEL_XTD3_UN1.R002_R16_2740m

sw-hh-al-288.desy.de    XFEL_XTD4_SA3.R001_R27_2838m

sw-hh-al-289.desy.de    XFEL_XTD4_SA3.R002_R48_2888m

sw-hh-al-291.desy.de    XFEL_XTD4_T4D.R006_R19_3057m

sw-hh-al-294.desy.de    XFEL_XTD5_UN2.R001_R04_2972m

SPB

• DAQ will be updated today by ITDM to make it more user friendly.
• Smaract stages issue. Only 2 out of 10 are not working. For the planned experiments these are more than enough. The last 2 will be finxed at the end of the month.

SQS

• While Cyril is not here the CAS contact person for SQS is Juri.
• SQS would like to meet with ITDM to discuss how to make the DAQ interface more users friendly. KW reminds that on one side, the update being deployed in SPB helps, on the other side people should get proper training, and for SQS this did not happen so far. SQS will send a list of people who need training to ITDM.
• Problem with the digitizers bandwidth is being addressed by AE. but was not known by CAS. Juri has to be involved in the discussion.
• 4 uTCA boards (For the reaction microscope from the University of Frankfurt) are found to be wrong: the firmware seems to be incopatible with the intergration in Karabo. AE will check it.

SCS

• Working together with BB to fix the issues with Loops.
• Loop 4 being brought to life, as well as Smaract stages.
• Interlock definition were sent to AE by M. Izquierdo. M. Teichmann did not know that and pointed out that Manuel is no longer an SCS member and thanks to this meeting he got the information. Has to be clarified.

MID

• Today at 14:00 in XTD 6 the technical commissioning of imagers will be carried out . ILobato will attend.
• First 2 loops for the MID Optics hutch being prepared.

​AE

• A request to improve the Mpot power supplies has been sent to CAS by many groups. CYoungman will send an e-mail to the relevand people and organize a meeting. The experiment groups have to also make sure that all the interested people are informed.

ITDM

• Hibef has to decide the network structure to have. In the past they wanted to have a private network behing a gateway. Now the concept is being rediscussed. Discussion ongoing. CAS and AE will also be involved 

FXE, HED not represented.

SPB / SFX:

• Experiments are ongoing, users collected data on Sat. and Sun.
• There have been issues with the detector/DAQ due to operator error
• There have been issues with the sample delivery
• There was an issue with the X-ray / Laser interlock systems on Saturday, which was fixed the same day.
• Beam time was partially lost due to these issues...

FXE, SCS, SQS, MID, HED:

• Did not attend
• Support points pointed out it will be much more difficult if the instruments do not attend this meeting to provide support in-time
• PSPO announced the Dispatch meeting will be discontinued if the instruments will not attend (it is not considered useful with just one instrument attending on a regular basis)

Detectors:

• Server for calibration data died, ITDM and DESY are working to revive it, S. Hauf will bring it up again.

CAS:

• CY will have the usual meeting regarding SASE2

AE:

• PBLMs in SASE3 being integrated.
• SmarAct stages for SQS are being brought up
• Vacuum group + AE working on SASE3 interlock

SPB

• New users run starting on Thursday. All set.
• Discussions on image processors and scan tool with CAS were held. Ongoing.
• Smaract stages issue. After the installation of a wall connection box by SPB only 2 axes out of 10 work. That's not an issue for the next users run but for the upcoming commissioning time it is. SPB will write an e-mail with details to AE support. 
• Ion pump issue: pump not sending feedback to the PLC.  SPB will write an e-mail with details to AE support. 

SCS

• Most of the crates needed for the forst run have been fixed by AE. Last crate will be ready today. Then Lopps 1, 2 and 3 can be brought up.
• Loop 4 can be brought up after 2 crates have been modified. CR submitted already by AE. in progress.
• With the ITDM support all the Basler cameras are now connected properly.

SQS

• Coordinated motion has been set up by AE. Tests by SQS are being runned. Feedback expected.
• For Loop 6 everything is prepared. If the cables have been installed AE can install the crates and switch on the loop. T. Baumann will check.
• Pfeiffer flow regulator status unclear. AE will check.
• Many small issues related to creates have to be fixed. AE and SQS to check offline.
• Digitizers issue: when a couple of channels are used, the transfer rate goes down to 80 Mb/s. AE and ITDm to address the issue offline with SQS.

AE

• Implementing new set of machine safety interlocks for SASE3 operation. No saftey PLC used. 3 signals are now provided to the MPS interface:
  • GATT pressure
  • Absorber open signal
  • Position of mirrors

The interlock conditions cannot be disabled/modified from Karabo. Karabo device already deployed.All ready for the test with D3 scheduled for tomorrow.

This announcement is to notify you that the intervention planned to configure the interlocks as read-only will be carried out at 11:30.

A following message will notify of the end of the intervention

Alessandro

Intervention completed.

SA3_SYS_EPS2MPS/DCTRL/BEAM_GATT_OK has now read-only interlock properties

SASE3 general

• Deployment of the new interlock definitions will be done in the next 2 days. Final test foreseen for Thursday.

SPB

• Users collecting data. Problems of clogging of the sample injection, this could result in empty image files. Wil be checked by the users..

AE

• FXE requeste a 2nd digitizer channel. Will be finalized with CAS.
• SQS digitizers issies reported on Friday solved.

DET

• FXE - LPD module is showing flickering. To be investigated after the users run.
• AGIPD pipeline issues (crashing). Are probably caused by misconfiguration of the detector by the users. 

 CAS

• XTD6 technical commissioning will be focussed of the MID CRL this week. No feedback from LBatchelor and MID received relative to the bellows. If no feedback is received, the bellows will be aligned in the central position and the motors disconnected.
• Log for GATT almost ready (related to the new interlock definitions for the SASe3 operation)

SASE3 general

• Interlock definitions for the tunnels: today a meeting with AE/HSInn/CAS will take place at 14:00 to continue the discussion.PGessler pointed out that one has to make sure to implement a concept which is also OK with D3 to avoid wasting time. The common understanding is that HSinn is the interface to D3 at this moment.

SQS

• Deployment of new interlock definitions needed. Definitions being prepared by SQS. Ongoing.
• Loop 6 will be needed in the next days. A couple of cables for these 2 crates are missing. Should be there in a couple of days. SQS wants the loop running anyway.

SCS

• Due to absence of beam, SCS is now focussing on the end to end test of components considered priority 2 so far.
• Elmo motors of the FFT chamber need to be rewired. AE on it.

AE

• Loop 3 SQS ready to be updated.

XPD

• Exit slit imager for SQS repaired yesterday.
• Procedure to correctly operate the XGM in XTD2 has been posted by J. Gruenert in the Photon commissioning team logbook.

CAS

• XTD6 technical commissioning will be focussed of the MID CRL this week. Some tests were already done, others have to be done when L.Batchelor is back. AE support (N.Jardon) is requested.
• For CAS top priorities are: SASE1 and SASE3 operation/readiness

SASE3

• If the bender of the M2 mirror is used the beam can be focused on the shutter and caouse damaged. This is a safety relevant issue that prevents SASE3 from getting beam. Interlock definitions related to the gas attenuator are being discussed to overcome the problem.

SQS

• Digitizers being set up with ITDM and AE
• Deployment of new interlock definitions needed. Definitions being prepared by SQS
• Loop 6 will be needed in the next days. AE reports that it's almost ready and in the next days will be deployed.

SCS

• Cabling almost 100% done.

AE

• Problems with ELMO motors reported on 10.08. for the KB optics has been solved.

XPD

• 2 screens were damagend in SASE3. One has a hole, the other one has surface damages.The imagers should be put in the interlock when multibunch mode is used. For the single bunch mode a warening signal is now appearing in the Karabo scene.

General

• The next dispatch meeting will take place as usual this Wednesday at 9:30.

SPB

• Middle layer for coordinated motion is being reifned.
• Request to have the pulse picker ready asap as it is required for user period. AE working on that,
• Scan tool has to be stested. CAS required.

FXE

• Problem with the shutter in FXE hutch: when openning the shutter with more that 2 pulses the electron beam is dumped. Bypassing the MPS signal FXE is able to get into the hutch beam in 2 pulses mode. The problem seems to be at the interface with the MPS. FXE and AE will check offline.

XPD

• due to the current beam parameters gas pressure has to be increased (10 ^-4 mbar range) in the XGM in XTD2 to reduce the error on the measures. 

General

• The next dispatch meeting will take place on 20.08. It was discussed with AE that the existing time slots (9.30 on MOndays and Wednesdays) can be used voluntarily with the SASE3 experiments to exchange information with the technical groups.

SPB

• Interaction region cabling being reworked. By the end of the week shoud be done.
• For the 1st week of operation the Pulse Picker has to be ready -  on a good track.
• For the 1st week of operation the coordinated motion and the Peltier system has to be integrated in the loop1.
• Karabo Middle layer for coordinated motion being developed.
• AGIPD z-motion encoder problems being addressed by AE/CAS
• uTCA for timing being installed.

SQS

• Trigger has been made available by AE. CAS devices ready today.
• ITDM and CAS have to meet to get the DAQ running.
• EPS signals being defined with AE.
• Loop 1 update to be agreed after the radiation safety and othe important tests have been done.

CAS

• Today between 12 and 14 the Karabo server will be unavailable in the SASE2 tunnels. 

General

• Questions about the tunnel access and tunnel access schedule should go to Thomas Tschenscher and the PRC.

SPB

• Controls for the pulse picker being prepared. High priority for SPB
• Coordinated motion tests ongoing. High priority.
• ATT motors need to be re checked in XTD9. The closing dates for the tunnels are not known excactly. SPB should phone BKR for the last information

SQS

• Radiation safety test preparation ongoing. 
• Loop 5 motors being tested.

CAS

• in XTD6 the VAC loop applied labels for the max excursions of the bellows. 
• PBLM commissioning ongoing
• CRLs for MID to be repaired.

General

• Starting from this Wednesday the dispatch meeting will be shifted back to 9:30.

CAS

• 3 MID CRL motors in XTD6 need to be repaired. The work is being planned and the clean tent is being prepared.

SPB

• Mirrors installed. Lid closed.  Las vacuum connection will be closed today. Then the electronics can be connected. 
• Mock up pulse picker system ready
• Some crates in the tunnels will be switched on today after updates of the interlock definitions to be ready for photons.
• Train on demand technique is currently being discussed by FXE with DESY. SPB should be also involved.

SCS

• Some problems occur during the test of the PLCs, plus after firmware update new problems showed up. Patrick Gessler will organize a meeting with SCS and Bernard to improve the communication and identify the error sources.

SQS

• Tests ongoing. No relevant issues so far.

ITDM

• Working on new proposals format.

CAS

• XTD 6 open until the end of September.
• This week due to lack of resources by AE only the last slit and one PBLM will be commissioned. The rest will be done next week.

SPB

• Mirrors being installed. Tests continue. Karabo updated.

AE

• Works in the tunnels SASE3 and 2 continue
• Works on the SASE3 experiment preparation continue

General

Works to ensure the redundant power for Racks in SASE3 have to be performed. A suitable date has to be found. PSPO will coordinate that.

Dear All,

I would like to inform that we released last Friday (22.06) the Karabo 2.2.4 version. We got around five days delayed to prioritize issues regarding Karabo-DAQ integration and MDL devices that were found in the last steps of the test cycle. However, fortunately, those issues are solved. 

Based on this run, the new dates that we (CAS) are proposing to start the deployment in operation will be:

• SA3: 28.06, Thursday (CAS references: Chris, Wajid, and Valerii)
• SA2: 02.07, Monday (CAS references: Chris, Wajid, and Valerii)
• SA1 + SPB + FXE: 03.07, Tuesday (CAS references: Alessandro, Dennis, Gabriele, Wajid)
  • SPB deployment should not impact AGIPD firmware test week, as agreed before.
• BKR: 28.06? (Wajid and Jan, could you please agree offline?)
• LAS, SQS and SCS dates have to be agreed with the respective contact persons.

-
Hugo

Dear All,

Please, see below the updates regarding Karabo 2.2.4.1 deployment:

• SA3: 05.07, Thursday (references: Chris, Wajid, and Valerii) - Done
• SA2: 06.07, Friday (references: Chris, Wajid, and Valerii) - Done
• SA1 + FXE: 09.07, Monday (references: Wajidi) - Done
• SPB: 10.07, Tuesday (references: Gabriele) - Done
• BKR: 10.07, Tuesday (references: Wajid, Jan, Harald) ToDo
• LAS, SQS and SCS dates have to be agreed with the respective contact persons. (references: Cyril, Riccardo, and David) In Progress
• ITDM: We are working to understand and solve a problem reported by Nasser in Karabo 2.2.4 Configurator (references: Dennis, Cyril, and Nasser) In Progress

SPB

• Mirrors installatiostarting now. Preliminary motion tests conducted last friday were successful. Still closed loop for encoders signal has to be implemented. AE will work on that.
• AGIPD firmware updated. Remaining tests ongoing. New Karabo release will be deployed tomorrow.
• Peltiers being tested. 

SQS

• End to end tests procceding. Laser incoupling and NQS chambers being set up.
• Loop 5 needed for next week.
• M pot firmware issue is being adreesed by CAS
• It is not possible to switch the ion pumps on and off remotely using Karabo. AE and CAS to look into that.

AE

• SASE3 exp. Loops preparation ongoing
• SPB Loop 4 update being prepared
• SASE3 tunnels interlock definitions are being updated for some diagnostic component
• SASE2 MID CRLs being tested.

CAS

• SPB 2.2.4.1 Karabo update scheduled for tomorrow morning between 7 and 9.

SPB

• Mirrors installation ongoing. Lid is open. Movements are being locally tested. Test of synch. motion with PLC to happen likely tomorrow.
• AGIPD firmware being updated. On friday all the tests should be finished and on Monday the deployment of Karabo 2.2.4. could happen. 

AE

• SCS support ongoing
• SQS Loop 4 being prepared
• New Interlock definitions are being / will be added for SASE1 and 3 
• SASE3 VAC loop updated today
• MPS plc has to be rebooted and the kernel has to be updated. The intervention will be done today between 16:00 and 17:00. No timing during the intervention. B. Fernandes will send a mail to the readiness mailing list before and after the completion of the work.

ITDM

• Maxwell cluster foxing postponed until 15:00

CAS

• An issue with the middle layer ofthe new release was found. Hotfix being developed and will be released today or tomorrow. NO ITDM migration affected.

Info about tunnels accessibility

concening the XTD10 + XTD2 situation: 

According to my information these two tunnels should be open and will be closed tomorrow at 15:00.

XTD10 will be opened again on Friday 8.00 and then stay open until next week latest until Thursday July12, afternoon (depending if Martin can find somebody to search). 

XTD6 + XTD1: as said before: It will be opened tomorrow at 8:00 and closed again at 15:00 

Both will be opened again on Friday. 

Concerning the interlock test in general, the old table from MPS is still valid (if you add the info that the tunnels are searched the previous working day after 15:00). 

SPB

• MIrrors installation ongoing. Almost all the beamline is vented. Only exception being the AGIPD, that will be vented this eveving.
• Mirrors movement test to happen AE to agree with SPB about a point in time.
• Motor parameters could be retrieved after the power cut last week.

AE

• Tunnel commissioning ongoing
• SQS/SCS loops being prepared.

ITDM

This Wednesday the Maxwell cluster will be not available from 9 to 12 due to mainteinance.

CAS

• Karabo 2.2.4. will be deployed in the SASE2 tunnels today. Control system will be down until 15:30. PLCs will keep running. 
• Karabo 2.2.4 deployment date soe SPB will be agreed offline.

Dear colleagues

 electrical power to SASE1 rooms, except FXE-experimental hutch OUTLETS, is

ready to use. FXE experimental hutch outlets will need about one more hour.

Kind regards

Dipl.-Ing. Torsten Schön

SPB

• Electronic and instrumentation will remain OFF until the electricity comes back tomorrow (will be announced by TS on the logbook)
• MIrrors installation will start next week
• Mirrors movement test to happen, then Loop 4 and 1 to be updated 
• In order to make the calibration of the motors easier after the powering up, they should not be moved before.

SCS

• New cables arrived. Testing ongoing.
• Loop 1,2,3, up. 1 has to be updated. Coordination AE/MTeichmann ongoing
• Loop4 and 5 being prepared by AE

AE

• XTD6 DPS and MCP being commissioned.
• SCS issues with Beckhoff modules being fixed.
• AE and CAS to agree on interlock reset procedure. 

CAS

• Karabo 2.2.4. will be deployed in the SASE3 tunnels tomorrow at 10:00. It will take approx. 2 hours. 

SPB

• The power shutdown has been delayed to tomorrow and power is supposed to come back on Thursday at 12:00. Torsten Schen will send an E-mail to readiness when the power comes back on Thursday.
• Pumps are in a safe state since they currently get external power.

AE

• Loop 4 SQS up and running. New SASE3 loops being prepared.

ITDM

• GPFS space issue. Meeting with the experiments on Thursday.

• MID controls, timing and signals cables have to go through the drag chain (40 m). ITDM and AE will discuss the issue with MID to evalueate problems due to the cables length.

CAS

• XTD6 closed tomorrow. This will generate some delay of the technical commissioning. 
• Karabo 2.2.4. deployment schedule proposed by Hugo Santos with an entry in the ELOG this morning.

SASE1 Experiments

Due to the interlock PLCs reprogramming, the emergency off circuit will be triggered tomorrow and power cuts will occur. Due to that, the AGIPD in SPB is being put into a safety state. ITDM and AE are informed. FXE and Optical lasers were also informed. The order is: Experiment in a safe state==>PLCs switched off==>Servers switched off. 

SPB will put all the motors to zero position and the parameters will be saved by a script prepared by CAS (AS). 

Next week Mon-Wed the planned power cut will also occur. 

SCS

• Control room being put into operation - New cables to arrive soon. AE addressing the so far known issues.

AE

• SASE2 MOV2 loop is up.
• Master timing will be down due to maintenance by DESY on Thursday, from 9:00 to 17:00

ITDM

• GPFS ran out of space in the week end. ITDM/DET/CAS meeting tomorrow. Then the experiments will be presented a way to go.

• On Monday we would like to replace the dns/dhcp server in the balcony room. In principle we have the secondary one working at DESY but experience shows everything gets slow and in particular Karabo may have big problems. Due to the power off in SASE1, this will be scheduled for monday afternoon.

CAS

• Issues with DAQ observed. Conflict on data aggregators. This was found to be a typo in the DAQ code. Fixing of the DAQ required but the release of Karabo 2.2.4. can proceed.
• Coordinated motion for SPB, SCS, SQS: Follow up tests by AE to be done this week.

SCS

• Tests  continue - List of issues sent to AE. Bernard will take care of that as soon as he's back.

SQS

• Issues with KB motors (ELMO) reported and will be addressed by AE. Also the list was sent to BB.
• Loop 4  is not up yet. Will be probably by next week.
• MC2 motors configuration procedure has to be clarified with AE. SCS also interested. Meeting will take place. 

AE

• SASE2 MOV2 loop will be up on Thursday. XGM already up.
• Master timing will be down due to maintenance by DESY on Thursday, from 9:00 to 17:00

ITDM

• GPFS ran out of space in the week end. This was due to the calibrated data from the users proposal being completed in the last weeks. ITDM and DET groups will meet to discuss size and strategies to deal with calibrated data.

XRO

• In SASE3 a slit is reported tomove left if the command "move to right" is given. CAS to look into that.

CAS

• Karabo version 2.2.4. Test being ultimated.
• Issues with DAQ observed. Work ongoing to fix it.
• Coordinated motion for SPB, SCS, SQS: There was a demo (successful). RBean will send minutes around.

General

• Next Dispatch meetings will take place on Wednesday, at 13:00.

SCS

• Tests  continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

• Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
• Loop 3 - Motors test continue
• Loop 2 is up and running.
• Loop 4  probably will run tomorrow.
• In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

AE

• EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

• Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

• Karabo version 2.2.4. being tested with PLCs. 

EET

• Firmwares fo all the SCS loops have been generated.
• Firmwares up to Loop 5 for SQS have been generated.

Hi All,

Just adding more info:

CAS

• Karabo version 2.2.4rc. being tested with PLC 1.27. Waiting for AE support to update the test environment to 1.28 (Jan, Nerea, and Bernard are aware).

SPB

• Beam delivery issues.
• After several days of usage the Karabo GUI slows down. Servers are reported to have crashed a couple of times in the last shift.
• Same situation with data aggregators. Once per shift they have to be restarted. This in known to ITDM.
• Interlock system seems to work fine now, but the impression is that the people in BKR cannot correctly interpret the interloch signals coming from the experiments. The issue relates to the Interlock panels to be developed in BKR (Issue list, Friday's Operation meeting)

FXE

• Reading the logbook it seems that some problems with the optical laser's shutter forced to stop the laser operation at about 3 AM. 

SCS

• Tests  continue

SQS

• Loop 1 one more ion pump will be implemented today by AE
• Loop 3 - Configuration of the rest of the  motors expected today
• Loop 2 should be up by tomorrow according to AE
• Loop 4 and 5 are supposed to go up in the next 3 weeks 

AE

• EPS in SASE2 should be ready today

ITDM

• FXE reported missing images from Gotthardt detector in the H5 files. The data does not reach the DAQ. Tests have to be made, but cannot be done while SPB has beamitime. Tests foreseen for tomorrow.
•  

CAS

• Some issues with scanning tool reported - working on it
• Some issues concerning data acquisiion from imagers were reported from SPB  - Valerii on it.

SPB

• Ready for the Users run. No big issues.
• The first 2 weeks of the shutdown will be used for cleaning and rearrangement of the hutch by SPB, in preparation of the mirrors instalaltion. PLCs update and Technical interventions by TS or external companies have to be carried out in this period. 

AE

• XTD6 VAC loop running.
• CRL XTD1 working
• Loop 2 SQS experiment still not running due to small issues being solved.
• Loop2 SCS was restarted yesterday after some fix

ITDM

• Gpfs should be fixed by now.

SCS

• List with the problems encountered so far during the end to end test was sent to AE. Bernard will get the info and eventually inform EET if problems with cables are found.

SQS

• Elmo motors for KB system were ocnfigured and are being moved right now. If problems arise AE will be contacted.
• It was not possible to control the turbo pump controllers via PLC. Bernard will have a look. In general also tirbo pump cables can and will be checked with the test box.
• Loop 2 is being prepared including the MC2 motors. Should be ready by tomorrow.
• SQS would like to start setting up the digitizers environment in Karabo. uTCA crates can be used on the HW side. AE acan support that.

AE

• XTD6 VAC loop is being patched. Small problems with the cabling of the CRL occured but solved. 

XRD

• XTD6 components being cabled. 2 Imagers still need to get the EPS signals. in 1/2 weeks ready for local tests.

​ITDM

• One IBM storage server failed. Reasons not clear. No impact on the performance of the system.

CAS

• This Week VAC loop in XTD 6 will work.
• For the MOV 2 loop still  EPLAN exports are needed.
• MOV1/MOV2 interfaces have to be defined.

AE

• CRLs in XTD1 being implemented
• GATT and EPS loop in SASE3 being updated

SCS

• Tests continue - No issues so far

SQS

• ELMO controllers have to be configured for the mirror motors test. Slit motors have lower priority. Cables can be tested w/o being physically connected to the motors, but to configure the ELMO controllers the cables have to be connected to the real motors. Tomorrow AE is available to support these activities.
• Basler cameras set up and working.

​ITDM

• SPB AGIPD test today or tomorrow

CAS

• Hot fix for SPB released yesterday. This will address the issue of crushings when instantiating a new device. Has to be tested with SPB. 
• XTD 6 will have no electricity for the racks today since Wille is completing the cabling. MTCA being shut down accordingly.

General

• For the upcoming emergency off test in SASE1 SPB and FXE should propose  a possible date in the next shutdown to be then discussed with AE.
• The machine is down due to the failure of a cryocompressor. Beam will be probably coming back tomorrow morning. Tomorrow morning will be also confirmed whether SPB and FXE cold use the beamtime for the user run.

SPB

• In the week end there was a failure of the PP laser shutter. It was possible to find a very temporary solution to circumvent the problem.
• The beam has been unstable for most of the time. No drift, mostly gitter. Reasons are unclear. The instability was also measured with the XGM in XTD2. SPB is analysing the data collected to be able to quantify the instability. 
• DAQ tests for AGIPD were planned for tomorrow but the request is to shift them to today to take advantage of the missing beam.

FXE

Some issues were reported: LPD readout of the data, Fast digitizers mainly. The details have to be clarified with the DET and ITDM groups.  

AE

SQS loop 3 is up and running. 

SCS

• Tests continue

SQS

• FMB Oxford will be working on the mirror tank starting form tomorrow until next week. Acces to the hutch is limited.
• There will be a meeting with AE to discuss the set up of the motors for testing (especially for the mirrors). SCS will also participate. 

ITDM

• SASE3 projects updated. Performance increased.
• Possible to move the AGIPD test for SPB to today

CAS

• Camera device had to be reinstanciated 4 times during the week end. Investigation ongoing.

General

• Please remember to send the list of the tasks you planned for the next shutdown to Monica Areste by the end of the week.
• Please remember to send the list of the alignment works needed until July to Gerd Wellenreuther.

AE

SQS loop 3 is almost ready. 1 crate is currently in the AE lab but should be installed soon. The loop 3 is needed by SQS for the motor tests with FMB Oxford next week.

SQS

• Due to a pump that stopped working (Caverion) this morning there was water on the floor of the laser hutch and part of the experiment hutch.  Some cables (not powered and packed) saw water. These are being checked. Also lasermet boxes on the wall saw water. Lasermet is checking that.

• For the testing SQS requested the test box to AE 

SPB

• Final DAQ test ongoing. Slow data acquisition OK. This is needed for the users program this week. Fast data acquisition (inc. AGIPD) will be tested next week.

ITDM

• Patch released for Slow data acquisition FXE/SPB. LPD acquisition OK. AGIPD issues being investigated.

The MPS comm for SASE 2 and FXE were tested and the only remaining part is to test the SPB to MPS changes. Tomorrow morning a  timeslot will be agreed between AE and SPB.

​​AE

• SASE3:
  • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail ==> AE updated the loop. Nerea is making sure that the MPS signals are enabled in the BKR 
  • Stepper motor for the mirror bender was found to be miswired. Has been already fixed. Similar issue (inverse wiring) was found in SASE1. Cannot be fixed now. It will be done in the next shutdown. For the time being one has to drive the morot the opposite way.

• SASE2:
  • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail ==> DONE. Not clear if some motor parameters for XRO devices were saved before. D. La Civita and CAS are checking that

• SPB:
  • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people. ==> DONE. 

• FXE:
  • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

ITDM

• DAQ has to be upgraded to thelast verison of Karabo (just hotfix). Will be checked with Gero what is the best way to go.

SQS

• Loop1 still being worked by AE. Not yet up and running. 
• Loop 3 is needed to be up and running in 2 weeks since the mirror motors ar eincluded in this loop and they are needed for the final check with FMB.

SCS

• Bernard has been added to the SCS operation logbook and he is getting al the updates concerning commissioning.
• Testing of the Loop 1 motors ongoing., 2 out of 20 motors have problems. Being investigated.
• Some connectors names are reported not to be consistent with the EPLANS. Martin Teichmann will check that with EET.

​​AE

• SCS:
  • loop 2 running. Bernard has recommended that you start with commissioning of KBS_VDM, _HFM, _BFL, VFM motors because they are using MC2MOTOR soft device class (controller is Elmo).
  • Crate BKF00157 can be removed and refresh terminal will be added without interrupting the PLC - only equipment connected to this crate will not be available during this period of change.

• SASE3:
  • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail

• SASE2:
  • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail

• SPB:
  • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people.

• FXE:
  • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

DET

• Working of the optimization of the python piece of code allowing the display of online images. Right now 30 images per second rate is possible.

ITDM

• GPFS server fixed.

CAS

• XTD1 CRLs will be integrated iin the control system in KW 22
• 20.07 is the goal date for beam in MID and HED optics Hutch.  

SPB

• Test on the mirrors peltier showed some issues due to the fact that the test boxes were different from the actual stuff. R.Bean will clarify it with FMB.
• Loops 12,13 update to be done.
• Magnetic turbos control: the serial communication has some issues. Being checked with AE and CAS

SQS

• Loop1 cabled up. Some names haad to be corrected in the firmware. As soon as the export is delivered by EET to AE the formware can be uploaded, Probably today.
• Upstream turbo pumps running using controlers in the rackrooms but still no PLC control.

SCS

• Some issue with the movement of the motors found. AE is on it.
• 1 crate does not have enough power to power up all the terminals. Will be solved today.

​AE

• The valves will be included in the interlock systems (SASE1 issue) - Loop ready to be updated - The MPS has to be disabled before updating the loop to avoid tripping of the beam.
• FXE loops update started

DET

• Hotfix for AGIPD ready to be deployed. Can start today. Details will be sorted out with SPB.

CAS

• SASE3 Soft Mono is now technically commissioned. PBLMs next in the line.
• XTD1 CRLs will be integrated iin the control system in KW 22
• XTD6 priority on the MID beamline. 

General

It is not clear if and when the experiment in SASE1 could get beam during this week. Has to be checked with the PRC.

SPB

• Loops 5,7,11, update and Karabo deployed.
• Loops 12,13,8 update ongoing.

SQS

• Loop1 cabled up. Today the formware will be uploaded.
• Loop 2 will probably be ready on Wednesday

SCS

• Out ov 50 valves tested 5 showed problems. Investigation ongoing to understand where is the problem (PLC, Cables, Valves)
• In Loop 2 a problem with a crate containing Elmo controlers was found. AE on it. Next loop to be tested will be Loop 4.
• 1 crate does not have enough power to power up all the terminals. CR already prepared. Will be solved soon.

​AE

• By the middle of the week the valves will be included in the interlock systems (SASE1 issue)
• The EPS interlock definitions for the tunnels in SASE2 are being finalized by AE, XPO, XPD

ITDM

• Failure of one IBM server for online storage. Fixing ongoing.

DET

• Online preview and charge injection issues are being solved restarting the DAQ and re-deployng Karabo.

SPB

• After re-compiling the relative karabo device, the configuration procedure of the AGIPD is now smoothand reliable.
• PLCs update ongoing. AGIPD is in a safe state to allow the PLC update.

SQS

• Meeting today at 11:00 with AE, EET, CAS to define the next steps of the end to end tests.

AE

• SASE2 tunnels - Only VAC conditions are transmitted from the EPS loop to the MPS. All the other conditions were supposed to be not necessary for first lasing and were not implemented. Nevertheless this was not clear for the people working in the BKR (especcially for diagnostics) during the last days. It has to be discussed with the PRC (H. Sinn this week) whether these have to be implemented on a short time. In pronciple AE can do it if requested. Otherwise the definitions will be implemented in the next shutdown.

ITDM

• Failure of one IBM server for online storage. Thanks to the implemented redundancy this will not affect the operation. The performances have to be checked.   IBM has been contacted.
• On monday evening ITDM OCD received a call from BKR because of some problems with the DAQ in the tunnels. It seems that this was due to the fact that the system was not properly tested by the "users" before. Seems to be solved now.

CAS (C. Youngman)

Following the Technical commissioning phase for the SASE2 tunnel components, the need of a technical coordination activity is strongly pointed out. This is in particular important to make sure that the proper final tests are performed by the components owner  before e.g. first lasing attempts.

SQS (T. Baumann)

Lopp 1 being brought up. First tests to be started soon. The CAS and AE contact persons should be in involved in the tests as early as possible.

INFO

As reported by the PRC (J. Gruenert) during the operation meeting this morning, due to the first lasing attempt in SASE2 no lasing of SASE3 is to be expected until the middle of next week.

SPB

• Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
• Smaract motors for PAM are needed and Nerea is on it.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions  have to be checked next week when HSinn is back. 
• During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it. 
• Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
• Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

• Access  of XTD1 today @14:00 to fix some EPS switches
• SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

AE

• SCS exp: first loop running. 2 creates missing but loop is up.
• MPS connection tunnels SASE2 done

ITDM

• Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
• MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

Regarding CAS Support for SASE2 Lasing:

CAS is happy to support in the frame of the On Call Duty system as it has been provided in the past.

I will update the SASE3 MOV1 loop in 30 minutes. Please let me know if you would like me to postpone the update.

Best regards,

Marijan

SPB

• Timing tool for optical laser installed
• Sample chamber's PLC will be updated next week due to ongoing tests until monday. Pending.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Pending. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Pending.

AE

• SASE2 EPS Loop running
• Updating PLCs for MPS in the tunnels. The update was stopped last friday due to a problem with the on/off funcionality of some ion pumps. The problem was found and the fixing requires an update of the VAC loops whoch will be done in the next hours/days. In case the loop update is not possible due to specific requirements the problem can be bypassed in Karabo. 

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

Dear all,

below please find the hottest and most up-to-date times for the closing of the tunnels after the shutdown:

Cheers, Gerd

SPB

• Karabo has been deployed  without major  problems! 
• Now PLCs update follows. Sample chamber's PLC will be updated next week due to ongoing tests until monday.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.
• Updating PLCs for MPS in the tunnels 

ITDM

• Network stability problems in SASE3 are understood and a workaroung is being implemented.

General

Closing time of the tunnels on monday still not clear. To be clarified by PSPO.

Dear all,

 the emergency power off test has been successfully performed in SPB.

Nobody can instantly cut the power to non hazardous racks (see racklist.xlsx managed by Ulf Brueggmann) by pushing the emergency off button anymore. The outlets in the experiment hutch do not go off. This is due to some missing cables. These will be installed in the next weeks and a new test will be planned for next shutdown.

Furthermore three racks for D.02 (SPB optics) and two racks for D.13 (SPB laser) cannot be powered off anymore by pushing the emergency off in D.09. The missing two emergency power off switching devices will be mounted in UV-XHEXP1-01-13-AV in the next shutdown. Until then the power to those five racks (D03-UH1 - D03-UH3, D13-UH1, D13-UH2) cannot be cut by emergency off buttons in D.02 or D.13.

Summary of Karabo 2.2.3 deployment dates:

• Karabo 2.2.3 deployed in SASE3 tunnels last friday ==> no major issues. DAQ can be deployed
• Karabo in SASE2 Tunnels to be deployed today at 14:00
• Karabo in SASE1 Tunnels to be deployed tomorrow at 08:00
• Karabo in SPB/SFX experiment to be deployed today and tomorrow
• Karabo in FXE experiment to be deployed on Wednesday

SPB

• By tomorrow the mirror installation will be done.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.

ITDM

• Infiniband port fixing ongoing
• GPFS bug to be fixed with a patch that should be issued on 24.04. To be checked whether it is still OK to install the patch or it is too late.

General

• PSPO will ask HSinn for a detailed schedule concerning closing of the tunnels to fine tune the activities going on this week in the SASE2 and 3 tunnels

Dear all,

I was just informed by Ulf Brueggmann that the emergency power off test was successfully performed in FXE.

The  reality is now reflecting the planned logic, exeption being the power plugs in the experiment hutch. They do not go off. This is due to some missing cables. These will be installed in the next weeks and a new test will be planned for next shutdown.

The tests will continue tomorrow in SPB/SFX.

This entry is on behalve of Frank Babies, if you have any doubt about it, contact him:

On 16.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from FXE
"fxe-rr-sys-utc-1"
"fxe-rr-sys-utc-2"
because of the Power Down issue on Monday.

On 17.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from SPB
"spb-rr-sys-utc-1"
"spb-rr-sys-utc-2"
because of the Power Down issue on Tuesday.

This will affect timing system and MPS communication.

Interlock systems SASE3 - TÜV test

The pre test by D3 will be held on 20.06.

On 05.07.The TÜV Test for SASE3 will be done together with the SASE2 test and the repetition of the SASE1 test

Emergency power test will be performed at the following dates:

FXE - Mon. 16.04.

SPB - Tue. 17.04. 

Schedule

• 10:00 Power will go off in SASE1 - Only redundant power line will stay ON
• Reprogramming of the PLC of the Emergency power off system - Estimated 1.5 Hrs.
• Power ON again ==> Emergency OFF test will be carried out. (info from AE: AE PLCs will not go off. uTCA crates will be shut down before)
• About at 15:00 everything should be back to normal.

For technical questions contact T. Schoen and U. Brueggmann.

changes according to the racklist on the electrical power distribution cabinets for rackrooms E.21 and E.22 are done. Power supply should be stable now. Please note that there will be emergency power off tests prior to the final acceptance test of SASE3 electrical installations around mid of May. You will be notified about the exact time and date.

Hi all, for your information below a message from Chris Youngman concerning Karabo 2.2.3 installations in SASE3.

"preparation work has concluded and we will definitely try to deploy Karabo 2.2.3 on SA3 machines TODAY 13.04.. SA3 s/w will not work from 8:00 on. Hopefully we will finish during the morning.

SA1 and SA2 installations should continue as normal."

Die Arbeiten an der Kühlwassersteuerung konnten innerhalb des genannten

Zeitraums abgeschlossen werden,

so dass die Kühlwasserversorgung nun schon einige Zeit wieder geregelt

läuft.

Großer Dank an alle Beteiligten, die den Ablauf der Abschaltung meines

Wissens auswirkungsfrei gestalten konnten.

Wir bitten über kurze Rückmeldungen zur Optimierung, sollten

Schwierigkeiten im Zusammenhang mit der Abschaltung entstanden sein.

Mit freundlichen Grüßen

Jörg Hanik

SPB

1. Detector z motion would be needed. AE checks the status of the crate. still pending
2. MIrrors installation ongoing. NO ACCESS to D.02!!!!
3. Laser Interlock change of logic by K.Witt will be done on monday at 9:00
4. Karabo update installation dates being agreed with CAS. Probably early next week.

FYI

Below the E-Mail from Hugo Santos with the Karabo deployment dates.

Dear All,

I would like to call your attention to the fact that we made Karabo 2.2.3 available last Monday (04/04, and the changelog 06/04). From the last Readiness Meeting, I got informed that we agreed and are allowed to proceed with the Karabo rollout planning on SASE3 and SASE2. (SASE1 still needs agreement):

1. SASE3
   1. PRE-CONDITION: (AE) PLC version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed. (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 13/04/2018 (Friday)
   3. Should be the first one because of the completeness of this system: VAC, PES, XGM and GAT loops, for instance, that have to be checked and run properly afterward.
   4. Downtime expected: 9:00 to 14:00. 
   5. (AE) PLCs version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed.
   6. No major change regarding DET systems
   7. CAS Members for this specific task: Chris and Wajid.
2. SASE2
   1. PRE-CONDITION: SASE3 Deployment task must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 16/04/2018 (Monday)
   3. Downtime expected: 11:00 to 13:00.
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET system.
   6. CAS Members for this specific task: Chris and Wajid.
3. SASE1
   1. PRE-CONDITION: SASE2 and SASE3 Deployment tasks must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues).
   2. Date: (17/04/2018 - Tuesday - to be agreed)
   3. Downtime expected: to be defined. 
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET systems.
   6. CAS Members for this specific task: Alessandro and Wajid

Finally, please, also see that as soon as we have the Karabo 2.2.3 running during the shutdown, we will have the most comfortable time to perform the setup, configurations and stabilize it, still, during the shutdown. This way we can handover the control system properly to the operators.

Please, if you do not agree or are not comfortable with something in this message, let me know as soon as possible. Otherwise, I will assume that it does not contain any showstopper and we (CAS) can proceed.

H.S.

General

Emergency off test in SPB is shifted. Date to be agreed on Friday. FXE test stays on 16.04. TS will confirm that.

MKK will shut down the cooling water for Tunnels and XHEXP1 on Thursday startin from 11:00 for maintenance purposes.

1. Based on the information we have the intervention could last up to 2 hrs. UBrueggmann is trying to check whether is possible to limit the duration of the intervention to bunches of 45 min. This could avoid the shut down of the Turbo pumps.
2. UBrueggmann is taking care of opening the rack doors in the tunnels (all of them!) and close them again when the water is back. 
3. The experiments should take care of opening the doors of the racks in the rackrooms. Also the main door.
4. AV will speak with the important groups involved and then send a summary e-mail to staff today afternoon.

SPB

1. Detector z motion would be needed. AE checks the status of the crate
2. Quadrant motion - Suggestion from AE is to perform tests and implement it in the next shutdown.
3. MIrrors installation ongoing. NO ACCESS to D.02!!!!
4. Laser Interlock/Rad. Saf. interlock issue still being investigated.  No news from SRP.

SASE2 -CYoungman

Goal is to stop the HW changes by the end of Wed. Thursday final loops and firmware will be updated. Both in SASE2 and 3.

Due to some delay in the installation of some necessary cables, the emergency power off test foreseen for today 06.04. in the SPB experiment will not take place.

The details will be reported by Ulf Brueggmann today in the dispatch meeting at 13:00. The needed cabling should be done this week, so that the test in FXE planned for Monday, 16.04 CAN take place.

TS will confirm the FXE date and propose the nex date for SPB this Friday in the Readiness meeting.

On Friday, 6.04, there will be a maintenance of the online storage system. This will tackle the issues we had just before the last user beamtimes, when we decided to apply workaround to keep the system working. The availability of the storage system is not guaranteed, especially when faulty h/w parts needs to be exchanged.

Best, 

Janusz

Tomorrow, 5.04, between 9:30 and 12:00 there will be urgent maintenance of one of the two redundant main switches for the control network. The network connectivity during that time is not guaranteed, but also any major interruptions are not expected. For the two switches in sase1 tunnels with only one uplink, there could be short (~1min) interruption.

Best, 

Janusz

SPB/SFX:

• Mirror installation next week

FXE:

• Replaced broken diamond-window

SCS:

• Installation of alignment laser
• Mirror tank installation next week

SQS:

• Not present / no issues.

SASE tunnels:

• C. Youngman is coordinating the works in the SASE1, 2 and 3 tunnels, daily meetings are at 9:30 every morning.
• Works have been scheduled and planned, most work to happen in SASE3 and SASE2.
• Staff should be aware that works e.g. regarding the vacuum loops have been scheduled and are ongoing, in doubt contact C. Youngman, N. Jardon or V. Bondar

CAS:

• Release test with Karabo 2.2.3 are concluding probably today, including test with PLC firmwares 1.26 and 1.27.
• This would lead to a release for ITDM & Detectors by this evening. A timeslot is required to test the new Karabo with ITDM / DAQ.
• Deployment at the beamline could potentially be in two weeks, if the test with ITDM and Detectors will conclude fine.

XPD:

• Email to xfel-op-allhands was received, indicating that there could have been a shortage of cooling water for the magnets and racks supplied from XS4 and also XS2 (should be SASE2-branch only). If you have electronic racks in this area, check the temperature, email will be forwarded to readiness.

Detectors:

• A single supermodule of the LPD-detector is flickering, is being investigated.

ITDM:

• A rise of the temperature in the balcony room from 27° to 39° was observed in the morning from ca. 6:00 until 6:30.
• This lead to damage at two power supplies.
• There was no indication by MKK or TS about any kind of shortage or shutdown of the cooling water supply.

SPB

• Sample delivery installed. Tonight will be commissioned. Experiment set-up starts tomorrow night

FXE

• Digitizers were already implemented in the DAQ but there is an issue related to the Karabo device. CAS already looking into that.

AE

• Follow up to the issue of re-setting pumps after interlock is triggered: loops updated. 
• SASE2 MOV loop patched
• Imagers in XTD1 have to be tested with Karabo

ITDM

• On monday afternoon problems with the storage servers were reported. The issue is a defective board, whoch is meanwhile been deactivated. IBM is working on that. The system seems to be working in this state.

CAS

• The next Karabo release will be able to handle better big loops. ASilenzi points out that the next update is really important since it adresses problems like the restart of time servers issue.

Readiness Meeting Issue list

• Item 2 (XGM SASE3): After restarting of the CPP servers the XGM turbo pumps stopped, valves closed and it required a manual intervention to restore them. ==> Firmware updated. CLOSED

Operation Meeting Issuel List

• Item 1 (Karabo rollout process): Testing steps of a new Karabo release have to be discussed with AE-ITDM-DET ==> Meeting took place. Closed

Update regarding Digitizer:

----- Forwarded Message -----
From: "hamed sotoudi namin" <hamed.sotoudi.namin@xfel.eu>
To: "Sergey Esenov" <serguei.essenov@xfel.eu>, "andre" <andreas.galler@xfel.eu>
Cc: "Djelloul Boukhelef" <djelloul.boukhelef@xfel.eu>, "Dennis Goeries" <dennis.goeries@xfel.eu>, "Leonce Mekinda" <leonce.mekinda@xfel.eu>, "Patrick Gessler" <patrick.gessler@xfel.eu>, "Hugo Santos" <hugo.santos@xfel.eu>, "Krzysztof Wrona" <krzysztof.wrona@xfel.eu>, fxe-instrument@xfel.eu
Sent: Wednesday, 28 March, 2018 17:37:28
Subject: RE: adqDigitzer

Hi everybody,

Today morning me, Andreas and Leonce have tested digitizer with software in 
FXE hutch. We tried the same configuration as it was and checked the log 
file, we observed that software was missing synchronization, that means 
either packets from digitizer have some problems ( missed or corrupted) , or 
buffers are too small in software. We reinitialized everything and problem 
disappeared ( at least for synchronization).
This afternoon I tested digitizer in our lab in APD mode with Raw data mode 
and also checked values; everything was fine. Because till today we have 
tested digitizer in fully load and enabled APD and raw data for all channels 
we observed no issues. I tried to set the same configuration which Andreas 
applied in their system ( APD for one channel and raw data for all channels) 
I encountered the same problems that he had mentioned in Email. Conclusion 
is that Digitizer has no problem when Algorithms are enabled for all 
channels. But when one algorithm is enabled for only one channel there is 
problem in packets, or even there is no packet from digitizer sometimes. 
Which gives timeout error.
Current solution is that to enable required algorithm for all channels and 
ignore result from unwanted ones.
I do really apologize for this inconvenience caused and will do my best to 
solve the problem in Firmware as soon as possible.

Best,
Hamed

Info from BKR:  the southern tunnels were opened, thus XTD1 can be accessed now.

General

Due to the triggering of the emergency power off in FXE caused by electrical works being done on Sunday, FXE and SPB lost beam and electronics was switched off. In general the efficiency of the communication between groups has to be impoved. An important first step is the publication to a easy accessible site of the OCDs phone numbers list. PSPO will organize that.

Comment by PGessler

"It was clear to all, that currently the impact on operation due to emergency power off is severe, since the Racks with the PLC CPUs as well as the MicroTCA systems will be also switched off, what is not what was defined/specified. Therefore it was agreed, that no work on the power distribution/emergency off will be done until the defined date (for FXE 16th April). That work was done there nevertheless is extremely careless. The fact, that this was done on a Sunday and also during user time makes it even much worse then that.  It has to be completely followed up and made clear, that this is not acceptable and will not be repeated again!"

SPB

A run of 14 hrs acquiring data was a sort of stress test for the DAQ. It worked. ITDM would like to remove the data. Should be ok.

Due to the triggering of the emergency power off also SPB lost the beam. This was sort of strange since the power off signal was triggered in FXE. AE reported that this behavior is to be expected with the current configuration. This has to be discussed.

FXE

DAQ is recovering from the power off. 

Tripping of the interlock caused again by control unit of ion pumps in the week end. During the shutdown the responsible units will be replaced.

In general is is agreed to prepare an overview interlcok panel in the BKR. AE is on boat, CAS to be checked.

500 kHz operation is problematic for the machine due to heavy losses. They are only able to deliver 1 bunch at this rate. FXE will discuss this directly with the PRC (this week is Jan Gruenert)

AE related

Follow up to the issue of re-setting pumps after interlock is triggered: VAC and EPS loops need to be updated. HSinn was pushing to have it done as soon as possible without waiting for the shutdown. This is not riskless for the experiments. This will be discussed in a small round (AE, PRC, SPB, FXE)

ITDM

Due to the power outage many devices do not have proper train id. CAS should check it. Time servers also need to be checked.

FXE needs to use digitizers for the upcoming measurements session. These were not implemented in the DAQ. ITDM/FXE/CAS meeting needed to clarify and solve the issue .

 DET

LPD calibration had a problem but should be ok now. 

More supermodules coming.

Communication with upcoming userve concerning data format and other issue has to be improved according to the DET group. SPB suggests to tackle the issue offline.

Dear all,
apparently the Emergency Power-Off at FXE was triggered at around 10:00 - 10:30 (my estimate). As far as I know power has been switched on again. BKR / PRC have been informed. More information will be provided by TS / Torsten Schön soon.

Edit: T. Schoen is on site and can be contacted under 0160 977 287 97

Regard, Gerd

SPB

On monday there was an issue with a detector related PLC due to misconnection of one EtherCAT cable. The issue has been solved and it "just" resulted in 3 of hours delay in  the AGIPD calibration activity.

FXE, SCS, SQS, XPD, XRO,ITDM no issues reported

AE related

uTCA in XTD 6 are installed 

VAC-Related

Marc Messerchmidt reported  abouit an entry in the DESY Logbook about a pump that triggered the interlock in the SASE1 tunnels. According to Nerea this this seems to be related to the old issue already known. VAC group is working on that. 

 CAS

CYoungman is working on the priority list for the components in SASE2 tunnels. Double checking with HSinn happening soon.

Access for Tue/Wed next week to XTD1 will be requested by CY through HS

Beam is in SPB/FXE this week due to the HERCULES project.

No new issues to be discussed where reported by any groups.

The status and issues reported on friday are still valid. 

Photon Run Coordinator (H. Sinn):

1.) SASE1: User experiment SPB could be finished sucussfully after several problems on the weekend (thank to FXE for providing night shift). Big progress in improving throughput of FXE insrument. Next D3 material test with 300 bunches on April 3, 9am. 

2.) SASE2: Transmission of electron beam to T5D dump (SASE2 beamline) this week sucessful. Switching between electron beamlines 

works nicely (minor problem witth dark current during switching).

Now air coil measurement going on. BBA needs to done before first lasiing starting on May 2nd. 

To be clarified today: To we need access to XTD in week 12/13.

3.) SASE3: Photon commissioning (with 24/7 shifts) finished sucessfully. Now specific tasks are done parasitically to SASE1 operation 

during regular working hours (organized by photon run coordinator). 

4.) General (stimulated by DESY operators): EPS system can lead to shutdown of DESY accelerator through MPS by many different channels (PLC in tunnels, Fast valves, PBLM etc). 

On DESY side, MPS provides overview of inputs, but structure difficult to understand and labels are not always conclusive, 

On XFEL side, no general overview panel for all SASEs+Instruments (scene) exists, but it required for facility operation. 

SPB

No issues/no report 

SQS:

- Cleaning of the hutches is ongoing. From 19.3. on, access will be limited. Furthermore, shoe covers have to be worn in E.06 and E.07.

- Electricity in E.22 is delayed. AE and ITDM will be informed about the status by PSPO.

- Vacuum will start the installation of the transfer pipe between the tunnel and E.07 on 19.3.

- SQS will start the installation of their instrumentation on 19.3.

- Some of our beamline components are currently stored at Hera South and must be transported to Schenefeld next week. However, there is an issue with the crane at Hera South. Kai-Erik has to look into this!

FXE (P. Zalden / A. Galler):

No issues/no report 

SCS (J.T. Delitz)

No issues/no report 

PP-Laser:

"The laser group has nothing particularly noteworthy to report, only that installation is still going as planned"

Vacuuum:

Installations in XTD6 are ongoing. Damaged vacuum pipes have been 

exchanged/ordered. Vacuum in XTD1 is closed. Control System in XTD1 is 

working and locally tested, Interlock-tests will happen after Easter.

XPD:

No issues/no report 

Advanced Electronics:

No issues/no report 

SASE2 Tunnels (C. Youngman)

Going according to plan. 1 week after Easter the movement and EPS loops are supposed to be brought up. 

ITDM (K. Wrona):

No issues/no report 

Detectors (J. Sztuk-Dambietz, P. Lang):

A module for AGIPD was replaced. 

No issues/no report

CAS (H. Santos):

DEMO for the coordinated motion was done during the week. Further tests needed. 

Schedule and discussion about the Karabo rollout process. See presentation bz H. Santos given in the technical meeting.

SPB (Weinhausen/Round/Mancuso):

• Again on the fast valve was triggering the MPS interlock system. For XRO was not completely clear what happened. for AE everything is clear and it depends on the wrong MPS configuration with respect to missingl cables. The will discuss this offline and eventually bring back the issue. Linked to this, it is reported that the MPS panel is difficult to decipher for the experiments...

• FXE (Galler):

• 2 ion pumps went in error and the interlock was triggered. Issue clarified.

SASE2 Tunnels (Youngman)

• Access schedule sorted out. Works in the SASE2 tunnels will be carried out in the period 4-19.04 plus the week before Easter. Discussion with AE on the status of the loops ongoing.

AE (Jardon)

• It is reported by SPB (March Messerschmidt) that people in the BKR are confused about MPS signal that they are getting from XFEL. The different lables assigned to the signals have to be sorted out and communicated. Nerea and Marc will try to start this clarification offiline.
• GATT Loop in SASE3 will be updated today
• Check of the inficon gauges ongoing- eventually update required.

ITDM (Wrona)

• Lots of data (14 Tb) collected during the last SPB shift.
• Issue concerning speed of offline data transfer reported by SPB
• For the user runs in May data will need to be deleted from the offline cluster. A procedure has to be established to clearly communicate with GLs and User groups.
• SASE3 IT installations have been handed over to XFEL. IT setup ongoing. Network will be available in the experiments by the end of the month.

SPB (Weinhausen):

• During the night there was an interlock issue. The fast shutter was triggering the MPS interlock system. After several calls and request of support from SPB to the photon run coordinator and the machine run coordinator the problem was found and the interlock signal bypassed. This was due to some incomplete cabling of the fast valve. While this was knwe, after the last updated the correct configuration was not set up in the controlling PLC.
• In general one has to note that the experiments do not have access to the MPS panels and there is no well established procedure when MPS interlocks are concerned. This will be reported in the PSO meeting.

FXE (Gawelda):

• Before the beam shutdown FXE observed that the beam was very unstable.
• 2 ion pumps went in error and the interlock was triggered. Again in this case is not clear who is responsible for the re setting of the interlock and the pumps.
• Servers had to be restarted due to wrong train id after the cooling water accident
• 0.5 MHz operation of the LPD successfully tested. 

SASE2 Tunnels (Youngman)

• It is reported that the technical coordination of the necessary activities is missing.
• A list of activities was sent by CYoungman to TTschentscher concerning SASE2 tunnels. In total 10 working days are needed to complete these tasks. This includes 1 week implementation and 1 week commissioning. Carefully planning with DESY is needed to avoid that the necessary 10 working days are scheduled around Easter, which will result in less effective days.

CAS (Santos)

• HSantos is preparing a deployment plan for Karabo whoch includes the requirements and deadlines to be fulfilled by the other support groups (ITDM, DET, AE). The idea is to have this plan presented on Friday for discussion.

Photon Run Coordinator (L. Samoylova):

Commissioning during machine studies Tue and Wed and parasitically during SA1 user operation at 
 SASE3 on Thursday and Friday 

SASE1

SPB

User operation, current status (Adrian): got few data from a test system on Saturday. The users’ 
sample delivery technology now seems to deliver nice & stable droplets, so if SPB 
get beam early enough today to set up for a measurement, a full data set could be collected. 
 

FXE

• Instrument Commissioning
• Measure beamline transmission (unfocused 550 µJ per pulse at the sample position, with lenses 
  115 uJ)

 Machine studies

• Wed afternoon: a set of K-mono FEL spectra taken

https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/07.03_a&pos=2018-03-09T17:56:20 
 SASE3

 Thu, Friday - a first experience of the SASE3 commissioning in 30 bunch mode during user operation: 

• minimal number of undulators (12) for lasing, low intense SASE (XGM shows 10 uJ, linear 
  regime), SRA closed to 2 mm
• MCP2 @ SASE3 - signal from 30 SASE pulses, consistently decrease with closing the slit down 
  to 0.5 mm

https://ttfinfo.desy.de/elog/XMLlist?file=/XFELelog/data/2018/10/08.03_a&xsl=/elogbook/xsl/elog.xsl&picture=true#2018-03-08T18:24:14 

• Fri afternoon GATT calibration with XGM and MCP ongoing, in 30b mode, optimizing SASE3 lasing 
  for these measurements needed - too weak and not homogenous signal in the center, see 
  https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/08.03_a&pos=2018-03-08T18:26:50

  Problems:

 'unexpected' from photon side closing XTD1 two working days earlier, explicit schedule 

• for the south branch tunnels access would be appreciated
• power off in XHPSC on Thursday at 21:15, no cooling water in experimental hall that led to 
  increasing the temperature in the rack with the MicorTCA crate, the CPUs of the two MicroTCA 
  crates in SPB and one of the two crates in FXE were automatically switched off. SPB crate 
  1 did not come automatically back afterwards, as the result: no train IDs for SPB PLC 
  loops, no MPS for SPB, no Clock and Control for AGIPD, no triggers for SPB systems on 
  Friday
• Saturday morning the MPS interlock takes into account the signal from one of three fast valve 
  on SPB shutter although these valves are disconnected to SPB EPS interlock. The MPS 
  bit had to be masked

Correction from SPB [12.03.2018 10:10]: This issue was not related to the XTD9 shutter, but to the 
optics hutch shutter. The fast valve in XTD9 is connected to the EPS, the fast valve 
in the OH is not yet connected to the EPS. After the restart of the EPS/MPS on Thursday 
/ Friday after the cooling water issue, the condition for the fast valve in the optics 
hutch was wrongly included in the MPS. 

SPB

1) Thursday 08.03.2018 (8 am - 8 pm) - SPB/SFX commisioning shift

    Notes:

          - everything running

          - RAW data overview very responsive with  data filtering as suggested by Steffen

          - Corrected data preview started very nicely and seem to provide only limited update rate after a few hours running

          - karabo DAQ deices (dataaggregators) need several rests, resulting in a full DAQ restart around 7pm

          - sometimes updates in KARABO were lost

          - preparation for users experiment - over night

2) Thursday 08.03.2018 (around 9 pm) - "cooling water" incident

     Notes:

         - vacuum pumps affected

         - AGIPD warmed up

         - uTCAs have tripped/shutdown (some infrastructure needs to replaced)

3) Friday 09.03.2018 - morning (User beamtime)

         - damaged uTCAs equipment was replaced but trigger setting needs to be input/ adjusted =  loss of the user beamtime

         - User experiment progresses

SQS:

Hutch cleaning Mid. March

KB tanks installation in parallel with SCS

9-14 air filter will be installed. After that date only clean works have to be carried out

Collision with KB tanks installation and other activities like Rad. Interlock test to verify

FXE (P. Zalden / A. Galler):

Setting up the experiment

IPM and XGM data manualy correlated. Would be nice to have the train id working to have the "live" correlation

SCS (J.T. Delitz)

• cleaning of hutch next week (wk11)
• we will start moving first components wk12-13, ie. ALAS and DPS
• KB installation (positioning only, no AC+filter required) beginning April-depends on commissioning of precision climate zone (currently discussed with PI)

PP-Laser (M. Lederer):

SASE1 commisisonin gonoing.

Some misalignment due to the cooling water accident. To be investigated 

09.04 SASE2/3 laser tables will be installed.

Vacuuum (R. Villanueva):

SASE2--------->Good progresses on XTD1 control system deployment: Differential Pumping sectors are ready for first beam // First five vacuum sectors under vacuum and locally tested (already with Karabo). XTD6 tunnel installation phase progresses at good pace.

SASE1--------->In general, nothing additional to remark. We are working in a more robust interlock definition to prevent false triggering due of ion pump control signals.

SASE3--------->Complete vacuum system is working normally. Gas Attenuator is now under routinary operation up to 2-3 mbar with a UHV interface pressure after its DPS sector < 1*10e-9 mbar.

XRO 

SQS/SCS/XRO collaboration for the mirror coating ongoing

Installation of M5 in preparation

XPD (T. Maltezopoulos ):

XGM@XTD2, @XTD9, and @XTD10 are online.

XGM@XTD1 is in the loop and accessible from Karabo

(together with AE+CAS).

Currently I test XGM@XTD1 components from DOOCS.

XGM@XTD9 shows slightly higher energy then in XTD2.

I calibrate all XGM components twice, but still it remained.

Note that the beam pass the XTD9 XGM with -7 mm offset

(in beam direction 7 mm to the left, to the south).

With this high offset absolute calibration is not guaranteed.

Advanced Electronics (P. Gessler):

1 CPU for SPB broken due to the cooling water accident. Already replaced. Being configured

FXE grating integrated in the vac loop

SASE2 Tunnels (C. Youngman)

More coordination is needed, especially concerning the access to the tunnels. For the technical details, please see slides on the indico page https://indico.desy.de/indico/event/20115/

ITDM (K. Wrona):

No problems with overrheating

Problems with Train ID reliability after the heating. Should be checked and monitored online

Documentation of single devices have to be improved. + there is need for a tool to make the sanity check

Detectors (J. Sztuk-Dambietz, P. Lang):

LPD commissioning OK + DAQ working. Now LPD off because of cooling water problems.

CAS (H. Santos):

• Sandor is in a meeting with the Detector group to align requirements regarding AGIPD and DA
• We started to prioritize and address the (agile) missions of SPB and FXE. As part of this effort:
  • Alessandro started to develop the scenes link on the GUI. This feature is really required and has been present in several SRDs (For example, to link vac sections/loops).
  • We implemented the GUI combo box to select different GUI Server - it is part of the SPB Mission (Leonce)
• Hamed and Sergey made good progress on the Digitizer. They will work on a demo/talk for the next PBS Meeting.
• Wajid, Sergey, and Dennis fixed the X2Timer issue reported by FXE (hash/data inconsistency). It is already deployed and running.
• Regarding the cooling incident. Wajid was the first contacted yesterday. After that Alessandro at 01:00. Today, during the morning, we reinstall Karabo at SPB-RR-SYS-UTC-1.

SPB:

• Monday morning Karabo failed. It was not possible to operate anything. Aftrer restart it took 2 hrs. to have the tunnel loop and Lopp 5 again operable . Other issues connected to that. CAS is on it.
• AGIPD can be operated with some workaroung concerning the control system. No final fix available yet.
• CSS can be operated locally.
• Coordinated motion test probably on 14.03.

FXE:

• IPM/XGM correlation data process ongoing. 
• DAQ worked reliably
• Scan tool operational for almost every component. Data storage works. Test ongoing.
• PG points aout that no MPS definitions were given and no MPS is available for FXE yet. To be checked.

SQS

• Corrdination of installations in the hutch with the TUV tests for Rad. Interlock needed.

SCS

• Nothing ton report

ITDM:

• FXE LPD slow data separation will be done today after the meeting
• Some devices are setting the train id as "0". CAS is informed.

SASE3

MCP HV - Crate restarted. Now ok. There was a misconfiguration of Power/Karabo settings. This meeds to be avoided increasing the communication between people performin operations on the XPD components and the technical support groups.

AE

• uTCA crate in SASE2 for timing being prepared, as well as PLC for VAC loop
• Replacement of a uTCA crate in the balcony room for SASE2 needed

Below some clarifications from R. Bean on the issues reported in today's dispatch meeting:

- AGIPD suffered many problems which affected the operability (not really a problem of the scene). It was just operable, although not at all efficiently. Problems with the pipeline / DAQ and Karabo controls for the AGIPD suggest that some reasonable amount of work is required to find and solve the problems which may not be independent from each other

- CAS OCD point already addressed with previous email: They were available, just we needed them rather more than we should have done

- The CSS is the large rail that carries the detector. We cannot move it via the integrated system because of a misunderstanding in the implementation of limit / safety switches into the eplan design of the crate. The cabling between rack room / device seems to be fine (but of course not really tested properly because the crate is not correct). We have to use the test crate to move it right now, which is rather tedious since the encoders / limits cannot be handled properly by that system. I am available to discuss the proper implementation of switches / buttons such that we don’t need a fourth attempt at this crate

- For the smaracts we still have axes that are not controllable via the PLC. It is these that would be beneficial if they are fully integrated 

SPB

• Last night the beam disappeared between 4 and 8. 
• AGIPD scene showed many problems, both concerning Karabo and Data preview. Detector is not operable.  To solve the issue a joint effort of all the groups involved, as well as the physical presence of experts during the shift is requested. 
• CSS has to be moved out of the beam. As of now there are problems with cabling that need to be solved
• Smaract stages are misbehaving. More support from AE (Mahadi) is requested
• Status of the coordinated motion test was asked. Should be ongoing but the experts were not there.

FXE:

• Testing fast intensity monitors and LPD 
• DAQ worked very well
• Scan tool must me tested
• Correlation between LPD and fast intensity monitor is problematic due to train id problems. There is a new device to be deloyed on the XGM. Today or tomorrow will be seen whether this solves the issues.
• BKR reported that the beam tripping was due to some FXE device close to the M3, but FXE is not aware of that. Info will be asked to Harald today during the PSO meeting.

Advanced Electronics (P. Gessler):

• MCP HV problem in SASE3 - the power supply failed. The fixing shoud be done according to the priorities, Photon run coordinator decides.

ITDM (K. Wrona):

• FXE slow data DAQ issues: these will be solved splitting the LPD slow data from the rest. small update of the software needed. 
• Simultaneous acquisition from LPD and cameras is now available and working well.

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIPD will operate with the old firmware due to problems with the new one
• No calibration data available as of today
• AGIPD is in a safe state

Photon Run Coordinator (N. G. Kujala):

• SASE1
  • Machine delivery 300 pulses upto XTD2 without M1 mirror in beam on Wednesday evening.
  • On Thursday 300 pulses deliver to FXE for radiation test.
  • Collected data using DAQ with 100, 200 and 300 pulses.
  • Large area Detector40 x 40 mm was set-up for radiation test at FXE for .
  • Damage test with 300pulses in FXE hutch with 0.5mm steel, 0.2mm copper and B4C block   
  • Vacuum at XTD2 absorber was stable and temp was also stable for 300 pulses
  • Harald setup piezo motors for M3 of SASE1.
  • Yesterday night beam delivery to SPB with single bunch.
  • ZZ for XTD9 on Tuesday by SPB for CRL test/check.
  • Vacuum on FXE branch in XTD9 was tripped on Tuesday early at 1am, which lost beam permission. Manually I have reset interlock.
• SASE3
  • SASE3 commissioning is going on.
  • Commissioning the XGM and IMG correlations with Gas ATT and without Gas ATT;
  • SoftMono mirrors is commissioning
  • Gas attenuator commissioned and Raul made look-up table for pressure setting
• 30bunches were operated in SASE3 with Gas attenuator

SPB

• Ready for first commissionig
• AGIPD power and cooling procedures OK
• Feedback problems between Karabo and AGIPD. Being worked out

SQS:

• Hardware installation starting Mid. March after  cleaning of the hutch
• FMB will install the mirror tanks in April. Has to be synchronized with the remaning work in the hutches
• Meetings with AE soon to strat planning the commissioning

FXE (P. Zalden / A. Galler):

• Ready to accept first beam
• Radiation tests done

SCS (J.T. Delitz)

• Removing of wood cover: 05.02.2018

• Cleaning: end of week 11.

• End of march beginning of the installation of the first components (ALAS and TIM)

• Waiting of the exact date for Interlock test to specify more exact the dates for the KB mirror system installation.

PP-Laser (M. Lederer):

• LAS is now commissioning 3-stage NOPAs for both FXE and SPB.
  ​so far all in plan to finish in March

Vacuuum (R. Villanueva):

• SASE1 
  • Nothing remarkable: the whole vacuum system works in a very stable way. We only had one incident on the output of 1-2 ion pumps that trigger once the interlock. For that we are investigating the causes .
• SASE 3.
  • The whole vacuum loop (as required for the first lasing) down to the instrument front-end is under control and no special issues have appeared yet.
  • The SASE3 Gas Attenuator is under manually operation and most of the Photon Commissioning Members are familiar with it. Next steps are aligned with the deployment of the Pressure Feedback Control PID control available at Karabo level and to continue with the software-related tasks.
• SASE2.
  • XTD1------->The complete vacuum string down to the front-end in the shaft building is under vacuum but in local mode operation (including XGM subsystem)
  •          ------->During this almost past week we already initiated the PLC local tests (i.e. US & DS Differential Pumping are already done), and in KW10 we aim to move on with the rest of the system (ion pumps and vacuum sector gate valves).
  •          ------->Absorber, V0 and Fast Valve wiring are in good progress and we think they could be ready for local tests by the beginning of next week.
  •          ------->The interlock tests are planned also for KW10.
  •          ------->Depending of the progresses of AE and CAS, we may even be able to deploy the first control scenes in Karabo.
  •  
  • XTD6 ------->The assembly works of the whole beamline there progress adequately and without remarkable "show-stoppers" that couldn´t be managed.
  •          ------->Vacuum Group and Optics group (N. Kohlstrunk) are having also special attention and dedication to the SDU vacuum chamber conditioning and preparation works for its upcoming in-vacuum mechanics installation.

XPD (T. Maltezopoulos ):

• XGM in XTD1 is connected to DOOCS (together with DESY)
• Synchronization of XGM@XTD1 microTCA next week
• Local tests at XTD1 XGM were finished (together with AE)
• XTD2, XTD9, and XTD10 XGMs are running 24/7
• Change to 300 bunches went without problems
• XTD9 XGM need SRG recalibration ... done meanwhile
• Wajid, Joakim, Tommaso, and Theo will repeat  SASE3 DAQ Imager/XGM correlation today (need single shot operation)

Advanced Electronics (P. Gessler):

• Broken terminal between SASE3 MOV2 PLC Loop and Master Loop were replaced last Friday without any side effects. The Train ID is now also provided from the MOV2 loop.
•  So far, no PLC related export from the SASE2 Tunnel eplans of the vacuum, beam transport and diagnostics PLC crates are possible. Therefore we will build up the XLS file for the start up of the VAC and XGM loops next week. However, the related group plus AE and EETF should meet to evaluate, if an export is possible until end of March in order to use it for further installation work. Otherwise XLS files have to be crates manually again with support from all involved groups. Also consistent documentation is required between diagnostics on one side and vacuum and beam transport eplans.

SASE2 Tunnels (C. Youngman)

• Table with the set priorities and deadlines available in Indico.

ITDM (K. Wrona):

• Run Configuration Groups required for the user experiments for FXE and SPB/SFX have been defined and tested
• Discussion after the meeting: Since the Karabo-DOOCS interface for XGM will be changed and it will affect also how the data arrive at DAQ data aggregators we need to modify and (re)test the BEAM_COND group.

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIPD will operate with the old firmware due to problems with the new one
• No calibration data available as of today
• AGIPD is in a safe state

CAS (S. Brockhauser):

• Gotthard for HIREX being operated
• Progress on GATT commissioning by Valerii

SPB:

• Ion pumps running in the Mirror chamber. Vacuum level OK to run.
• D.02 is now interlocked and is a clean area.
• Mahadi and Alessandro are setting up a test system for the coupled movement. The test will probably be done between the 2 users periods.
• 2 hours of parasitic beam can be used today to check the alignment of the CRLs. The CRLs will be then moved out of the beam to allow the FXE radiation tests tomorrow.
• Primary limits for coupled motion not included in E-plan, safety switches are implemented but SPB prefers to have the primary limits included. CSS crate to be modified after clarification with AE/EET.
• Test of the karabo brigde will be performed with users. It is not yet clear exactly when since this needs synchronization with several groups involved (CAS, ITDM, DET). 

FXE:

• Radiation test tomorrow
• LPD test measurements in thhe WE
• Von Hamos spectrometer alignment until monday

ITDM:

• SASE1 and 3 have the last DAQ update now. Stable in SASE1 so far.
• Data sources arrangement being worked out. Today a 45 min session is foreseen with SPB.
• SPB needs slow data acquisition from 2 cameras for users, apart from AGIPD.

VAC/Detector/XPD/CAS: nobody attended

SPB:

• Ion pumps running in the Mirror chamber.
• Starting test with mirrors together with CAS this week.
• In general everything in good position to receive beam.
• Primary limits for coupled motion not included in E-plan, safety switches are implemented but SPB prefers to have the primary limits included. This will suppose a hardware modification.

AE:

• SASE3: train IDs in loop2 are working, the problem was fixed last Friday.

ITDM:

• Measurements for commissioning continues.

CAS:

• H.Santos inform that CAS has not resources to run tasks for SASE2, this might affect the date for first lasing in SASE2. Priorities are SASE1 operation and SASE3 commissioning.
• G. Wellenreuther will bring this topic to the PSO meeting.

FXE/VAC/Detector/XPD: nobody attended

Photon Run Coordinator (L. Samoylova):

• SASE1 tuning is ongoing (350 microJ)
• SASE3 3000microJ
• SASE1 interlocks enabled, SASE3 only vacuum interlock enabled
• Soft monochromator is being commissioned
• K-mono studies were carried out
• HiRex was being commissioned
• SASE1 and SASE3 DAQ was used
• Gain curves should now be possible

SPB/SFX (M. Messerschmidt):

• Recorded data with detector integrated in Karabo
• problems with coupled motions persist, ran into & were not able to get out of end switches using Karabo
• leak at mirror tank still has to be localized

FXE (P. Zalden / A. Galler):

• Will be ready for beam by the end of the week
• chamber was pumped, crystals mounted
• I0 monitor was tested and commissioned
• only have permission for <=2 bunches
• scantool: Train IDs are not captured
• DAQ-tests look good for LPD

PP-Laser (M. Lederer):

• Final commissioning of the laser is ongoing
• integration of the laser into DAQ during the April shutdown

Vacuuum (M. Dommach):

SASE1

• nothing to report; Vacuum system ready of beam operation

SASE3:

• XTD10 was open last Tuesday
• Together with Diagnostic group Exchange of Turbo pump on XGM
• Gas Attenuator commissioning ongoing

SASE2:

• Working on cabling of XTD1 components and vacuum installation in XTD6
• Installation of cleantent at HED-SDU ongoing.

XPD (J. Grünert):

XGM SASE3 

• tuesday XPD made XTD10 access to repair SASE3 XGM vacuum
• one turbo pump failed Saturday noon, decided to repair during access tuesday, exchanged pump
• tuesday detected another vacuum trip in the differential pumping around the XGM
• wednesday afternoon VAC made ZZ-access to „repair“ what tripped the vacuum interlock
• wednesday to thursday night: checking proper and stable SASE3 vacuum operation - is ok
• XGM is back in operation since thursday morning 8am

XGM status overall:

• XTD2 and XTD10 XGMs in operation
• When will the XGM in XTD9 be needed? ==> Harald and Liuba believe that beam might be sent to XTD9 this weekend, therefore they request to start XTD9 operation today afternoon

Imagers 

• SQS Exit slit imager has been set to beam position and can be used for some basic commissioning (limited functionality)

thursday work with beam:

• SASE1 KMONO studies: two cells studies, height adjustment of single cells using DAQ (until it crashed at noon), data analysis today
• HIREX commissioning
• SASE1 beam conditions recorded with DAQ

Correlation of beam intensity monitored by XGM and imagers

• Required e.g. for the current Gas Attenuator Transmission Calibration
• need to prepare a simple table of calibration values for all imagers which are used as supplementary intensity monitors
• will discuss offline among photon run coordinators

SASE2

• need to rearrange space in one rack to be able to put our last crate into tunnel XTD1: Wolfgang will contact Ulf and VAC
• Request to CAS: 
• DOOCS devices shall be created for SASE3 (as in SASE1) for undulator and phase shifter and electron beam energy

Advanced Electronics (P. Gessler):

• SASE3: No train IDs are being delivered in loop 2. Repair could potentially / in the worst case affect all electronics. ==> Should be fixed right now to get it done.

ITDM (K. Wrona):

• Configuration / Testing of DAQ continues

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIDP ready for 100/64 bunch operation
• there will be a two-day training for AGIPD-operators
• LPD was tested with new Karabo / DAQ
• LPD has hardware problem regarding power line for one module, to be fixed

CAS (S. Brockhauser):

• The issue reported in the beginning of the week regarding the TrainIDs was identified. We have an integer overflow problem when connection is lost with the timeserver. That happens sporadically, and we are working to fix it since yesterday.
• We released Karabo Hotfix 2.2.2.2 to fix an issue on the RunConfiguratorGroup at SA1 (ITDM already checked and it is working)
• We have improved the communication protocol with AE on handling loops/PLC updates and related (re)establishment of the Karabo services.
• AGIPD library is extended to supply information if detector is in stop or acquire state. The next step will add the "acquisition time" to enable scantool integration
• The old digitizer is running and integrated with DAQ. Also, we will add the "acquisition time" to enable scantool integration
• We started to call the Sprint Meetings (for the moment we focus on updating the priorities of FXE and SPB topics)

AoB:

E. Boyd: A test @ FXE with Cu-Target and 300 bunches is required. This will affect the areas / hutches around FXE

A. Mancuso: Will try to work around this, but in general a message one week beforehand would be required / appreciated.

SPB

• KB chamber being pumped down
• Smaract commissioning ongoing
• Coordinate motion for VAC ready to be tested (probably tomorrow). It has to be agreed with Richard Bean
• Clean-up slits Karabo middle layer deployed and working. - The same hase to be done for the power slits.

 FXE

• DAQ test ongoing - fixing of bugs
• After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Still the case.

• ​XPD

• XGM in SASE3 fixed (A turbo broke - replaced)
• One exit slit Imager in XTD 10 commissioned

AE

• PBLM moto problem was found to be due to a wrong Karabo-PLC linking. ==> FIXED
• Soft mono commissioning started and AE support was required. DONE

ITDM

•  Main problems with Train ID solved. Some problems still remain for some beckhoff loops in the tunnels==> AE on that.
• Some way to document the change of data format for devices has and will be found by IDTM
• Data source running for FXE. Changes have to be clearly communicated. 

CAS

• Soft fix improving the GUI deployed. No 2.2.2 ready.
  

• ​Issue with Karabo device returning the sensors data (e.g. cooling AGIPD) is being investigated. The Bug was found also in test environment but still random.
   

SASE3 - CY

• GATT works
• Inficon device to be improved (loop restart needed)
• CAS will deliver a method to deal with the configuration changes asked by the experiments, redirecting the changes to the proper platform (Firmware (AE), Karabo...)

Small correction:

The coordinated motion is the KB-mirror's, not the Vacuum.

The postfix is 2.2.2.1 and includes features that were not included in 2.2.2

Karabo 2.2.2 is ready

 FXE

The FXE beamshutter key has been installed into the interlock system after satisfactory completion of the FXE Chicane and Radiation Protection Sticker Checklist. The FXE Experiment hutch is now ready to receive xfel beam as required by the FXE instrument group.

However, the bunch mode MUST be limited to 1 or 2 bunches until the new user chicane is tested with thirty bunches using the copper target. This can be done whenever the parameters can be practically put into place. The test will be similar to the hutch commissioning where once it has been confirmed that the beam is on target. I will take radiation measurements at the chicane. This is the only place where radiation measurements will be required. Please provide me as much notice as possible prior to the date and time that 30 bunches will be brought into the FXE Experiment hutch.

I have also place 3 additional area monitoring dosimeters in the FXE Experiment. One on the robot arm where the D3 pandora place holder is. One above the old user chicane on the D.05 side and one inside the new user chicane on the control room side.

SPB

• KB chamber being pumped down
  

• interaction with AE is ognoing concerning update pf Loop 5 and AGIPD cooling procedure ongoing
• Smaract commissioning ongoing
• The goal is to be ready to start commissioning with beam on 01.03.
• Slit Karabo middle layer - integtration and final fixing being finalized by A. SIlenzi

 FXE

• DAQ test ongoing - fixing of bugs
• After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Being discussed with CAS
• Permission to use parasitic beam next week requested to the radiation safety group.

​XPD

• XGM in SASE3 requires physical intervention to be fixed. Access request for tomorrow.
• DAQ groups definition ongoing and being clarified with ITDM

AE

• PBLM moto problem was found to be due to a wrong Karabo-PLC linking. If the tunnel will be accessible tomorrow the issue will be solved. HSinn to decide on priority.
• Soft mono commissioning started and AE support was required. It is not clear what priority does it have. Mahadi is anyway working on that today ad 14:00. If the intervention is found to require more time one has to see.

ITDM

•  During the FXE DAQ tests many problems were found:
  • issues with digitizers
  • issues with scan
  • issues with images acquisition. In this case the problem has been identified (Train ID was 0). KWrona is in contact with CAS to fix the bug.

CAS

• The issue reported by SPB about no data being read from AGIPD temperature sensors has been identified by Gero (CAS) and the fixing is ongoing. 

Report of the XTD10 interventions from last week - Tuesday and Wednesday - C. Youngman 

Many thanks to those who contribute to the "SA3 'further technical coordination push" onTuesday (no-beam XTD10 closed) and Wednesday (XTD10 open) mini-shutdown, below is brief summary. The status items were not highlighted during PSPO's readiness meeting AoB section but are useful to know. Those issues that were raised are contained in the conclusion section. 

We leave the SA3 beamline with a functioning interlock, but you should additionally watch and test your component and ensure that beam operation is possible.

The principle task on Tuesday was

• for ITDM to shutdown and update f/w of the control server CPU hosts in the balcony room, and
• for CAS to replace the Karabo 2.1.18, running since the Oct-Nov shutdown 2017, with Karabo 2.2.1 which is already running on the SA1 installation. 
  • A number of problems during the installation were addressed and a stable deployment was finally reached early in the evening.

The principle task on Wednesday were

• test the updated GATT Ebara pump updated f/w and its s/w counterpart
• test that GATT Bronkhorst Mass-Flow valve f/w worked: correcting conversion errors and the incorrect setting of property r/w attributes
• test that GATT Inficon gauge f/w did not crash the loop and correct  conversion and r/w errors as with the the Mass-Flow valves
• perform Beckhoff-local tests on one or more the DOD motor axes as they were expected to be wrongly wired (as seen in the Jan-Dec shutdown in the VSLIT)
• update the s/w to reflect changes in the GATT f/w
• address difference in the s/w introduced by the move to Karabo 2.2.1

The key-point task listed above have been attained with the following caveats:

• There is a h/w fault which prevents the second Ebara pump being fully controllable remotely, it is currently locally controlled.
• Bronkhorst f/w and s/w is usable, the conversion is correct, but additional s/w work is required to align the s/w
• The Inficon f/w works (it does not crash the loop), the f/w conversion error is identified but not enough time was available to modify it.
• some s/w issues remain which are being addressed (tagging of Ebara, Bronkhorst and Inficon s/w, correct configuration of the vacuum-section..)
• the automatic update of project and scenes was not completely successful so project and scene owners should check and correct their scenes. The auto update procedure will be improved.
• Some data logging information was lost, but e now have regained a standard implementation (not hot-fixed) which should improve the this.
• After measuring that the DOD axes (1 of 10) were incorrectly wired it was concluded that all were incorrectly wired (as had been the VSLIT in the Dec-Jan shutdown)
  • Benoit and Daniele managed to rewire all 10 DOD axes during the mini shutdown and test 1 axis with Karabo.
• PES tests of multi-board digitizer s/w and f/w were continued (with measurements at the crate), these are promising as 'raw' and 'processed' data are now concurrently available.

In conclusion - good progress was made and this type of 'commissioning' mini-shutdown (anything longer than ~4 hours which includes somebody elses longer ZZ) should be performed as often as possible.

This requires

• that the photon run coordinator team know that there is a "list of task" to be performed which are not in the ZZ list
• that a selection of these tasks can be performed within the time budget of a mini-shutdown and that these do not compromise SA3 tunnel operation
• that the photon run coordinators inform the SA3 technical commissioning team (contact a member of the "SA3 tech coord" CAS team setup by Sandor, e.g. Chris, Valerii or Wajid ) so that they can organize the work in time (or report that they cannot as something or someone is not available).

During the next few days a "list-of-task" will be created. This could include technical commissioning actions for SA3: PBLM, ExitSlits, very end of beamline imagers where the status needs updating, etc.

Also remember that the CAS team tries hard to satisfy as many requests as possible, but prioritization will be done, so make your case thoroughly and don't expect it to simply happen (it has to be on the radar screen). In particular h/w, f/w and related s/w tests needing access will be done first and non-critical s/w work will be left until after tunnel closure. And remember to check your projects, scenes, and macros before you need them.

Here is a information from Sandor on CAS OCD Numbers
(this info is copied from the SASE1 commissioning logbook): 

during the day: 98053   (8:00 - 16:00)

overnight: 98054  (11:00 - 8:00 next day) 

during the weekend it is #53 or #54  (Saturday #53 8:00 - 8:00 next day, Sunday #54 8:00- 8:00 next day)

Photon Run Coordinator Report to Readiness Meeting

The maintainance on Wednesday was very successful and all goals were reached (and some extras). 

Thanks to everybody involved in planning and execution! 

- Installation of Karabo 2.2.x in XTD10 went smoothly. 

- SASE1 tuning studies are ongoing. Still there are some things to be learned on compression and 

optimizations will go on Friday. Currently around 250 uJ.

Recent achievments in commissioning: 

SASE1: 

-Peter aligned CRLs in XTD2 to beam. 

-Naresh could work on Hirex

-Liuba and Ilia started pointing studies with imagers (?)

Problem: We found out that gate vales of vacuum are not connected to DESY MPS

(waek beam was hitting one closed valve behind Hirex for about 2 hours)

SASE3: 

- Gas attenuator commissioning started

- Calibration of SRA to optical center of beamline

- The undulator beam was steered 4 mm horizontally. The vertical steering requires movement of undulator center, 

which will be tried possibly on Friday

- Cross calibration of XGM to FEL imager started last night. 

SPB

- KB mirrors

        - installation of horizontal micron KBs finished (approx 8 working days were required)

        - pump carts attached, expected that one week is required before valves can be re-opened (Vacuum group needs ZZ to reconnect valves on the XTD9 side)

        - installation of vertical micron KBs in April

        - D02 will remain cleanroom during the intervening time, i.e. all companies / groups (internal and external) coming on site should be briefed. No access without permission and accompaniment

        - controls tests on elmos completed via Karabo, help from Nerea / Mahadi to get everything working as expected

- configuration of smaract motors

        - commissioning setup will be used during first beam week in March

        - Re-config of smaract controllers by Marcin / Grant / Mahadi mostly completed

- AGIPD

        - firmware update scheduled for this week will not be completed as DESY detector group firmware expert is not available

        - remain limited to 32 patterns per train

        - automatic cooling and powering procedures are not working reliably (requires investigation poss. CAS?)

        - AGIPD chiller fell into an error state after removal of a separate crate BKF00072 (SPB_IRU_LIC), loop 5 redundancy requires investigation (AE)

- Karabo

        - GUI is slow to update or does sometimes not update (request investigation by CAS)

        - devices (e.g. those reporting pressure / temperature) have to be re-started before updates are seen in Karabo (request investigation by CAS)

        - Users have questions about any changes in the implementation of the 'Karabo bridge’. Technical information on how information is broadcast from Karabo is required in order to ensure that users can adapt their monitoring software 

- data trial run

        - Robert suggested trial run for DAQ / user software plugin before the first experiment. SPB can contribute to the execution of these tests and involving users. Requires also action from CAS and ITDM teams

FXE

• Diamond grating in XTD9 can be controlled remotely now
• DAQ test ongoing
• LPD calibration to be  performed

Laser:

•  Installation according to the plan, no major delays. 

Vacuum

SASE1:     ZZ access to connect SPB valves

SASE3:     tripping of interlock seems to be understood, interlock conditions changed,

                if it happens again, the two effected gauges will be removed from interlock until April

           Gas Attenuator is operational, Raul did some tests yesterday, up to 1 mbar

           No feedback control on gas pressure, only manual setting of gas flow

           The device can only be operated by experts so far

Optics

• see PRC section 

XPD

Last wednesday tunnel access:

• „repaired“ SA1 IMGSR trigger (with AE)
• „repaired“ SA3 XGM, restarted (but intentionally without HAMP)
• better routing / fixing of purple Beckhoff loop Ethernet cables and fiber connections is needed

SASE3 vacuum interlock trips (which eventually led to XGM failure on Saturday and reoccured monday night again) were strongly investigated by VAC : now understood ?

work on cross-calibration between XGMs and Imagers (as backup intensity/position monitors when XGM is out of order) with Laurent

DAQ: work with Krzysztof and Luis to get SASE1 DAQ back in operation, and to start SASE3 DAQ (can we test today ?)

Current status of SASE delivery (max. µJ in SA1/SA3): yesterday 390 / 200, this morning 190 / 600

Request to CAS: please plan to send for "training" or "experience" some CAS people to BKR during beam operation, especially the people in CAS who do OCD.

On behalf of Natalia : report to AE on detected issues with the ELMO controller settinsome hours gs of the PBLM motors in SASE1

AE

- stuck trigger timing module in XTD2 restarted (related to SR imager)

- Gas Attenuator PLC loop updated

- PLC modules in SPB rack with SmarACT system rearranged according to updated documentation

- Two bugs in PLC Framework identified, which are fixed now in the current version, but is not deployed yet to most systems:

     - Access rights are not correctly transferred to Karabo. This leads to the fact, that in cases of very special parameters only read access is possible via Karabo (these are automatically injected parameters in Karabo). If you come to the problem of not being able to write such a parameter, please let AE know and we can set it directly to the PLC. This will be solved with the next PLC loop update.

     - A bug was identified, which could in the worst case lead to a full crash of the related PLC. This could be caused by a „re-init“ on a device on the PLC. Therefore do no re-inits on PLC devices. If you really see the need for a re-init of a certain device, please let AE know about this and we see, if this is required and how this could be done. Also this issue will be fixed with the next scheduled update of the PLC loop.

ITDM

•  Cameras issue solved.
• SASE3 DAQ testing to start
• XGM data acquisition to be tested due to karabo/DAQ possible lost of information recently debugged.

CAS

+ Karabo 2.2.2 is released

+ 3rd control pc is set up in BKR

+ SASE3 is running Karabo 2.2.1

+ Karabo 2.2.1 is tested with DAQ

+ as we moved into weeks of commissioning, CAS OCD is regularly (basically every night) involved in and provides support for partly software and partly hardware issues

+ as experiments are coming, Data Analysis integration requests (eg. on Karabo Bridge) shall be channeled to Hans (and to the newly setup DA contacts: Thomas for SPB and Cyril for FXE) directly

+ the gui slowness problem reported by SPB needs to be checked if it comes from the GUI client or from the server and shall be tested with the new Karabo hotfix 2.2.2, too.

DET

LPD canlbration analisis ongoing

LPD Long integration time acquisition being tested

AGIPD: colntrol optimized and tested. Firmware being tested, the aim is to ioncrease the acquisition speed.

AGIPD: next week calibration data to be taken. 

AGIPD: Sensors in the centra area that underwent scattered beam to be checked.

CYoungman - SASE3

 - good progress was made and this type of 'commissioning' mini-shutdown (anything longer than 6 hours) should be performed as often as possible (Wed. shutdown). Next work items would involve: PBLM, ExitSlits....

LBatchelor - XTD6

XTD6: both CRLs (MID CRL2 and HED CRL2) ready for installation (lenses installed and aligned, chambers sealed). Installation should happen next week.

SPB

• MIddle layer karabo devices are responsible for the det. Karabo problems. Issue being addressed.
• 2 mirrors in D.02 installed.
• Test of the mirrors movements is done.
• Mirrors cooling bars being installed

Vacuum

• SASE3: 

  • Spikes of the downstream DPS gauges are responsible for the interlocks tripping. Probable cause is related to the beckhoff terminals of the DPS (different from the other gauges). Ongoing.

XPD

• Pop-in monitor filters in XTD10 being changed
• Exit slit imager after Mono XTD10 has ot be commissioned and scintillator has to be put out of the beam way.

AE

•  It can happena that some parameters cannot be set using Karabo due to lack of writing rights. A fix has been found but we would need to update all the loops. Since the problem is just occurring from time to time, AE asks to report specific cases for whoch a solution will be found. Same things happens with the re initialization of some devices. In this case the PLC crashes.
• Timing device in XTD 2 stopped working. Will be fixed today either by remote re start or by local intervention.

ITDM

• Broken server replaced
• Tomorrow there will be an update of the cluster between 9:00 and 10:00. DAQ will be down.
• Reboot of the online cluster needed. To be agreed with Detector group. 1 hour downtime also in this case.

CAS

•  In SASE3, deployment of new (2.2.1) Karabo happened yesterday and seems stable, but brittle. Analysis of the reason ongoing.
• Today GAT final fixes plus testing will be performed

SPB

• During operation Karabo crashed. As soon as detector is being operated the problem appears. Fixing is still ongoing.
• 2 mirrors in D.02 are being installed.
• Test of the mirrors movements has been performed.

FXE

• Debug of implemented devices ongoing
• Diamond grating in XTD9 has to be moved out of the beam way manually (no remote controls implemented due to long lead time of required beckhoff teminals). Access requested for Wednesday.
• Vac group will check whether beckhoff terminals are available to allow the remote control of the dimaond grating

Vacuum

• During the weekend the XGM in SASE3 was damaged. Details still unknown. 

AE

•  SASE2 XGM local test ongoing
• GAT loop update in SASE3 being done.
• Local checks in XTD10 on Wed.

ITDM

•  Slow data acquisition issue affecting imagers: solved
• One server died over the week end. ITDM notified only today.

General

"In the beginning of this meeting SM congratulated the whole staff of the European XFEL and in particular the commissioning team with the very important achievements 
- the first SASE3 lasing / bi-lasing at SASE3/SASE1. It happened yesterday late evening. More information will be given during our PBS meeting next week."

Photon Run Coordinator Report to Readiness Meeting

• parasitic operation in SASE3 (closed 3 undulator segments) for photon commissioning with SR-beam
  • V0 opened, vacuum open down to XHEXP-shutters
  • photon beam detected down to FEL imager, yesterday until SR absorber beyond mirrors M1/M2 and MCP
• photon systems supported tuning/search for SASE1 lasing
• Progress in SASE3: 
  • First Lasing in SASE3/XTD10 
  • simultaneous lasing SASE1 and SASE3

SPB

• During operation Karabo crashed. The problem is not clear yet. Det. loop was affetcted ==> AGIPD tests stopped!
• 2 mirrors in D.02 are being installed.
• Test of the mirrors movements is ongoing
• Motors issues are being checked by AE (Mahadi)

FXE

• Gotthard det. and digitizers back to work
• Leak close to the diamond window being fixed

Laser:

• Timing system has been upgraded - currently we can have any repetitive laser pattern for each laser, number of laser pulses in not limited to number of electron bunches, also we can have lasing when there is no x-ray (electron bunch)

•  Installation according to the plan, no major delays. No commitment to experiments for laser availability in March.

Vacuum

• SASE2:

  • XTD1 under vacuum up to XS2 shutter

• SASE3: 

  • interlock problems solved, runs without problems since yesterday

  • GasAttenuator loop needs update. when is the best time to do that? to be agreed with PRC

Optics

• issues mainly in the XPD section 

XPD

• SASE1 : photon systems supported tuning/search for SASE1 lasing
  • tests of DAQ in SASE1 ongoing
• parasitic operation in SASE3 (closed 3 undulator segments) for photon commissioning with SR-beam
  • V0 opened, vacuum open down to XHEXP-shutters
  • photon beam detected down to FEL imager, yesterday until SR absorber beyond mirrors M1/M2 and MCP
• karabo update to 2.2.x in SASE3 tunnel scheduled for next tuesday morning
• SASE3 DAQ urgently required
• issues with in-vacuum goniometer motor control of KMONO
• issues with IMGFEL to be understood - found a scintillator which works when we have FEL radiation

AE

•  The need to properly plan the activities close to the tunnels closing is pointed put.
• The need of a specific technical coordinator as in SASE3 for the SASE2 tunnels is pointed out

ITDM

•  Sase3 DAQ not ready since Karabo is not yet up-to-date
•  Slow data acquisition issue affecting cameras: being worked out.

CAS

•  In SASE3, deployment of new (2.2.1) Karabo versionwas  delayed because of many people getting the flu.
•  2.2.2 release test cycle ongoing

SPB/SFX

- An AGIPD crate providing power broke down and has been fixed already by Kai-Eric Ballack. This delayed the data acquisition test planned by Jola. These are being done today.

 -Synch motion of mirrors will be tested with fake mirrors in D.09 as soon as the control is ready. Work is ongoing.

-First mirron in the SPB optics hutch will be installed today. 

- New Karabo release working smoothly so far.

FXE

- CRL in the tunnel being commissioned

 -Leak close to the diamond window being fixed. If everything goes fine ready for parasitic beam on Wed next week.

VAC

-Vacuum interlock tripped twice in the last 12 hours. Issue has to be clarified with CAS.

AE

-Reconfiguration of the SmarACT controllers between rotary and linear motion is being carried out. Marcin and Mahadi are the main players here.

ITDM

-Major safety update of the cluster has to be done. ITDM proposal is to do it next week. SPB and FXE opted for the early morning (between 8 and 10). Exact date will be announced by ITDM as soon as possible.

SPB

- MIrrors installation (2 horizontal mirrors) in D.02 has started 

 -Synch motion of mirrors will be tested with fake mirrors in D.09 as soon as the control is ready.

SASE3 - CYoungman

- After SASE3 tunnels were closed it was discovered that was not pobbile to set values of the massflow (GAT loop). This required a remote intervention (firmware) that was supposed to be fast and easy  but it was not, and this required work until friday evening. Current status is: The gas attenuator secion closed and in static vacuum since the shutdown of a turbo pump was required. The system is in asafe status.The issues are being analized and solved with experts right now.

-It is important to establish a procedure to set priorities for ZZ access. The issue will be brought up in the PSO meeting

ITDM

-AGIPD p[roject is OK. Jolanta will start taking data already today

SPB

- Loops update OK

- Issue with SmarACT controllers. AE is on it (see below).

- Synch motion of mirrors motion being worked out with AE.

FXE

- Loops update OK

- DAQ updated

- CRL in the tunnel have two encoders with problems. Can be disabled and are not a problem for operation.

- A leak before the diamond window was found. Being closed now.

- Preparation for timing tool commissioning ongoing

- The new user chicane will be probably cut next week.

Laser:

- SASE 1 commissioning and upgrade moves according to plan

- SASE 2 and 3:  so far we still have green light concerning access control, laser safety installations, network and sync for installation of laser tables starting at the beginning of April.

Vacuum

- SASE1 und SASE3 vacuum systems are ready for restart of beam operation

- update of SASE1 vacuum loop on Wednesday caused many problems (all ion pumps in error state)

Optics

- Test piezo motors XTD9 today

XPD

- XTD10: K-Mono, XGM, PES, filter chamber, imagers operational except:

- Karabo tests for Exit slit imager.

- Camera cables to be exchanged for 2 imagers, ZZ access 1h.

- PES, issues with digitiser and MPOD scenes.

- XTD2 and 9: All components operational, Hirex also including Gotthard.

- Next steps

- Progress on scenes and projects implementations.

- Compatibility with new Karabo, Karabo GUI.

- Test and improve usage of DAQ.

AE

- Coordinated motion of MC2 Based Motors were implemented and presented to first users in SPB and XPO as well as to CAS

- The issue with the SmarACT motor controllers observed in the past was due to an issue in the controller. It was solved with an firmware update and requird configuration with dip switches. For this the later usage (e.g. rotary, linear, etc stages) has to be defined. The modifications for SPB is ongoing.

- AE observed, that in some cases PLC related electrical work was carried out until the very last day of open tunnel, which require a PLC firmware update and successive test. Since there is also a risk in the firmware update as it could be, that during tests issues might be observed, there is no access anymore to the actual system and thus could not be solved without ZZ or new access time. We had issues due to this, where at least temporary workarounds were possible - but it could also be different. This is a high risk, which should be know to everyone involved and decided on, if we want to continue like this in the future. AE would prefer to stop with any electrical PLC changes at least some days before tunnel closure and allow firmware updates (possible Karabo changes) and full tests by the users of the components with Karabo in order to identify and solve possible issues.

- Another issue took place, where a change request for a PLC change was issued, but instead of waiting for the finally modified designs to then do the related hardware modification, the modification on hardware were done without any new designs. This creates many problems: no CAD design according to hardware exists, the current revision state of the module is not defined, no firmware structure could be exported and generated for the current design, etc. Therefore modifications without new CAD design and consulting AE is not allowed!

ITDM

- Sase3 tunnels: cabling issues solved.

- DAQ so far stable. Has to be tested at full load now with the new Karabo.

- DAQ update for Tunnels in SASE1 and 3 ongoing. Today and next Monday will be test days.

- Gateway update done: small issue with SSH connections, easily solvable==> the procedure will be posted in the logbook

- The Maxwell cluster will need to be shut down for 2 days next week due to upgrades.

CAS

-  2.2.1 has been deployed without real problems on FXE, LAS, SPB, SA1.

-    In SASE3, it is still 2.1.18 and it will be updated early next week

-    2.2.2 release test cycle almost finished. The release will be published early next week - This release will not be mandatory, but recommended based on the new GUI benefits.

-    SASE3 Tunnel most critical tasks were done, including the gas attenuator by Valerii. It was a really complex task that Valerii went deeply and get it done for us all!

-    Sergey keeps working in Digitizer with Hamed (AE) but based on the deadline they decided to go for the old protocol.

-    Andrea (Alessandro and Steffen) has been working on the AGIPD Library to report when AGIPD will stop to acquire data

-    Gabriele started discussions to address the DAQ for LAS with ITDM.

-    XGM's DOOCS integration has been updated to address TrainID issues. We must allocate 2-3 days to test it with Beam.

Today late in the evening (~22:00), the exflgateway will be migrated to a new hardware. All users will be logged out before migration and home folders will be copied.

After migration is finished, you may experience ssh warning about ssh host key. For all centrally managed Linux this should be fixed the following morning. For all other, like Windows, or yellow desktops, you need to remove the old key and/or accept a changed one. 

Best 

Janusz

The migration has finished, to avoid temporary problem with ssh you can use

ssh exflgateway -o StrictHostKeyChecking=no

until it will be resolved in the morning ( the usual update of all hosts ssh keys).

Home folder has been changed to /data/<user_name> and it is slightly bigger then before...

Let me know if there is still anything missing.

best, Janusz

SPB/SFX:

• Fixing of Karabo scenes ongoing
• Loop 5 need to be updated after discussion with DET group (Jolanta Stuk)
• A meeting with AE will take place today to discuss the synchronous movement of the mirrors motor. 
• The test on the NKB chamber of the indipendent movement should take place. It has to be double checked that such motors are implemented in Karabo.

CAS SASE3 (C. Youngman):

• SASE3 tunnels closing today at 14:00. All the critical tasks were done. 
• The day before the first lasing, a ZZ access is strongly requested. The request will be discussed in the PSO meeting.

ITDM:

• Tomorrow the DAQ will be available for the AGIPD. LPD to follow. The goal is to have SASE1 ready next week as a whole.

SPB/SFX:

• Problems with data loggers ==> no statements about the past.
• Loop 5 upgraded
• Alle the other Loops are not at the 100% of their potential but sufficient for the users.
• Test on nkb mirror movement to be done.

XPO

• In SASE3, 4 cameras have trigger problems. AE is working on that.
• In SASE3, 4 cameras have accessibility problems. ITDM is working on that.

CAS SASE3 (C. Youngman):

• Interlock for VAC, GAT and XGM loops are tested and OK
• The rest is so far as it could go. For details refer to C. Youngman report.

ITDM:

• Broken Linecard  replaced .
• Update/switch for the gateway to the control network will happen in one late evening during this week. ITDM will write an entry in the logbook just before. 

Dear all,

the closing of the XTD10 tunnel is shifted to Wednesday (January 31st.) at 2 p.m.

Best regards,

Martin

Dear all,

please enter your access requests to the tunnels into the list provided by Winni Decking & Co.:

https://ttfinfo.desy.de/elog/XMLlist?file=/XFELelog/data/doc/Access-Requests&xsl=/elogbook/xsl/elog.xsl&picture=true

Can be also found through the usual XFEL-Logbook, and then clicking on "Access Request" on the left hand side.

Cheers, Gerd

SPB/SFX:

• Preparation for the mirror installation were ongoing as planned.
• The Karabo and server update went smooth, but scenes were broken a bit more then anticipated so we are still testing on that part.
• ITDM was working on DAQ update and Jola checked out the detector with the new Karabo successfully.

• did not attend / did not send any report

Vacuum:

• SASE1:

  • Vacuum system ready for restart

• SASE3:

  • loops were updated

  • final interlock test ongoing

  • GATT tests with Karabo

    • Vacuum possible

    • manual attenuation of the beam possible, no PID control

    • Controls of Ebara pump needs commissioning.

• SASE2:

  • XTD1: under vacuum, leak test done, all pumps running

Dear all,

the update to the SASE1 Timing system has been perform with no problems. The Timing board is now running firmware 1.24.0 and the doocs server was updated to version 7.8. I also updated the MCH firmware to version 2.19
 

SPB/SFX:

• Preparing D.02 for mirrors installation (next 2 weeks no access to D.02)
• Karabo being updated - Detector being tested, seems fine so far.
• DAQ tests foreseen for today
• NKB movement (Loop3) should be tested to make sure that the controls work fine before interfacing the MKB with real mirrors inside.
• AE will check the status of the CSS crate. If ready it will be mounted and tested after the MKB.

Advanced Electronics:

• ATM for timing has to be replaced in the balcony rooms.  downtime expected to be  15 minutes. Will happen Today at 16:00

CAS:

• Deploy SASE1 topic in the tunnel this week.Tomorrow at 13:oo CAS would like to make the switch. A.Silenzi will write an entry in the logbook to make sure everyone is on board with that.
• Release of the version 2.2.2. being prepared. This would mainly improve the GUI performance.

XPO

• FXE Mono local test performed and OK. To be tested through Karabo in Feb.

CAS SASE3 (C. Youngman):

• Gas attenuator controls being completed and tested.
• Soft MOno local test done and only small problems found
• EPS loop in good state - interlock test tomorrow.

General remarks:

• Stable beam only to be expected starting from 1.03. for SASE1 instruments commissioning
• There is no guarantee for parasitic beam delivery to the experiments. If it happens at all it  will most probably be during the last week of Feb. This has to be agreed on the fly  with the BKR and H.S/ T.T.

SPB/SFX:

• Preparing D.02 for mirrors installation (next week)
• Karabo being updated

Advanced Electronics:

• Trigger delivered by DESY should be back and stable at 10Hz. 
• ATM for timing has to be replaced in the balcony rooms. Bruno Fernandes is trying to find a way to reduce the downtime down to 30 minutes. Will be scheduled in the next days.

ITDM:

• 1 line card has to be replaced. Probably happens on Wednesday this week during the DAQ tests.
• Tuesday afternoon the online storage units will me remounted with the same path of the offline ones.
• This week a proposal for the in-house commissioning is going to be circulated.

Hi everyone,

there was some miscommunication between us and DESY: only the Timing firmware and doocs server need to be updated, the RTM was already been replaced. This require update is similar to the one perform two weeks ago (see entry #449).

The time slot for this update is next Wednesday, 23rd January, between 16h and 17h. The update will affect all Timing Devices in SASE1. This operation should not take more than 10 minutes; during this time no timing signals will be generate (clocks/triggers/bunch patterns/etc.). Naturally, problems may occur during this update which will lead to increase down time.

If somebody sees a problem with the current schedule, please contact me.

SPB/SFX:

• 2nd AGIPD intervention will be finished today.
• SE worked on a reliable procedure for nozzle exchange and trained the instrument scientists
• SmarAct motors are in error, Mahadi / AE is informed. These motors are needed for user operation / commissioning.
• loop 5 update still outstanding
• Karabo update for loop 1-4 was done, seems to be ok. Full update is planned for beginning of next week.

FXE:

• JJ X-rays finalized installation of the last components at FXE:
  • Diamond grating (in XTD9)
  • Spectrum analyzers
  • Timing tool
  • repositioning of the sample manipulator and working on better cabling
  • minor work on cables of FXE CRL to allow full movement along the beam direction
  • Johann spectrometer
• FXE got the information from CAS that the Karabo update was successful and it’s ready for testing
• FXE decided not not pursue the further commissioning of the new equipment due to the emergency power off tests
• FXE is working on a more universal sample environment allowing faster and more reliable changes from solid to liquid experiments.

Vacuum:

SASE1:

• vacuum system closed,
• PLC loop update done,
• ready for restart

SASE2:

• XTD1 all devices in the tunnel and aligned

SASE3:

• Soft Mono chamber closed yesterday, pumpdown, leaktest later today
• GasAttenuator comissioning ongoing
• Karabo scene developement

XPD:

• SASE3 MCP detector technical commissioning:
  • HV tested OK (MPOD operational), today ADC/signal test
  • motors: rewiring done --> correct directions and limit switches
  • motors were intermediately ok, but are not stable (now there are again issues)
  • several PLC / karabo issues were sorted out, ok
• SASE3 XGM:
  • test with Kr gas supply yesterday/today
  • test with gas plus HV
• SASE3 PES:
  • ordered grounding works (cabling) to be done similar to what was done for the XGM
• SASE2 XGM:
  • vacuum incident: vacuum operation of the XGM in XTD6 was unauthorized interrupted for some time, The rack had been powered down without prior information to XPD.
  • Reminder to all that rack near the XGMs shall be powered down only after informing and coordinating with XPD.

Laser:

• SASE1 PP-laser commissioning and upgrade is continuing according to our plan.
• Karabo upgrade to 2.2.1 was rather uneventful and successful.
• Hutch readiness milestones for laser hutches in SASE 3 and 2 seem to consolidate:  15. March. If this continues to hold, we will start our Phase II installation of PP-lasers in E.04 and A.04 with laser tables at the start of April. Laser tables for the ILHs will be installed at the same time.

Detectors: Did not attend.

Advanced Electronics:

• Replacement of the microTCA for the PES was done
• microTCA update of the timing system was done
• local test of the CRL can be done after the cables and the hardware have been fixed by the Vacuum group
• All loops of FXE updated (firmware etc.)

ITDM:

• Network ports for SASE2 in the racks have been fixed
• DAQ in SASE1 needs fixing of hardware, foreseen for CW 4
• Test of DAQ with new Karabo release is on its way, should be done by Wednesday 24.1.

CAS:

• Karabo 2.2.1 has been deployed on FXE, LAS-LAB, LAS-PPL, and 3 SPB loops without any real problems
• migration continues next week on SPB
• BKR will be equiped with a 3rd Karabo Control PCs (we work with Tim already on it)

Event trigger #16 is now back to 10Hz and it will still that way.

Dear all,

I communicated today on the 13h meeting that the Trigger #16 was back to 10Hz; this is no longer the case, the trigger signal is back to 1Hz.

Dear all, 

following up from the message #449, the MicroTCA Crate in rack 1.2 has been rebooted. Timing system hardware is now running firmware v1.24.0.

SPB:
- one mfpga board was found malfunctioning and was replaced with the old one. Concerning the mc the new version can be used.
- Liquind jet test ongoing.
- Waiting for the Loop 5 to be updated by AE
- Old version of Karabo was sluggish as of yesterday.
- 2.2.1 Karabo verion being rolled out for SPB - test ongoing.

FXE:
not represented
Vacuum:
- SASE1 vacuum in the tunnels closed (CRL chamber closed).
- KB mirror chamber in D.02 (SPB optics hutch) will be opened at the beginning of February
- In SASE3 the gas supply is now decoupled from the vacuum loop
- Soft XRay mono vacuum to be closed today.
XPD:
not represented

Detectors:
not represented

AE:

-XTD 2 CRL to be tested tomorrow
- Update of loops ongoing
- Emergency off button test in FXE showed that the relative controlling PLC was not configured correctly. In addition, communication problems between TS and FXE made things more difficult. PSPO will talk to TS and decide how to proceed. In the meanwhile the test foreseen in SPB for next monday HAS BEEN CANCELLED. GWellenreuther has informed Torsten Schoen.



ITDM:
 - Meeting this week to decide whether to use or not the train builder for 
AGIPD and LPD detectors
 - in 3 racks in XTD10 some network connection added last year do not work 
properly. The responsible company solved the problem without bringing down the loops.

CAS:
- 2.2.1 Karabo was deployed in FXE experiment and tests are ongoing.

- 2.2.1. being rolled out in SPB exp and Laser

- SASE3 tunnels  tunnels will get the new Karabo version next week. SASE1 tunnels will follow (probably in the same week) 

SPB:
- The exchange of the mc and mfpga boards has been done and some problems were found. The boards are now being debugged at CFEL.
- Sample enviroment's group is testing the liquid jet this week.
- Wait for the Loop 5 to be updated by AE

FXE:
not represented
Vacuum:
not represented
XPD:
not represented

Laser:
not represented

Detectors:
not represented

AE:

- Timing devices update done. One crate in XTD2 has a problem and cannot be operated remotely. Fixing ongoing.
- The clock, VETO and Start signals for the AGIPD could not be brought correctly from the utca in the rackroom to the patch panel in the hutch due to the excessive cable length. Workaround found using a signal replicator at the patch panel position. The problem seems to be solved.


ITDM:
 - LPD recabled with Infiniband connection. Test succesful.
 - Meeting this week to decide whether to use or not the train builder for 
AGIPD and LPD detectors
 - in 3 racks in XTD10 some network connection added last year do not work 
properly. The responsible company is comeing back next week to check the 
cabling. Interventions have to be planned accordingly since the Beckhoff 
loops are running! ==> no news.

CAS:
- not represented

Dear all,

with the exception of one board, all boards in SASE1, SASE3, FXE, SPB and PPL have been successfully updated. All Timing boards are now running firmware v1.24.0. The doocs-x2timer-server has also been updated in all CPUs; running version is 7.8.

The problem occur in the Timing System of rack 1.2 located at XTD2. The MCH board is not reachable via ssh/telnet/web interface; thus I am not able to power cycle the timing board, which would make the hardware load the latest version. The board is currently running (i.e. generating all the timing signals) but with an older hardware version. During the next week this problem will be tackle this problem with a quick visit to the rack.

Cheers.

Hi everyone,

after talking with the responsabile people, here's the timetable for the Timing firmware/server update:

In the meantime, all of SASE1 and SASE3 machines have been updated (with exception of the MicroTCA in the SASE1 Balcony room) with no problems.

Dear all,

an update of the Timing boards firmware and doocs server is require. This operation should not take more than 10 minutes per device; during this time no timing signals will be generate (clocks/triggers/bunch patterns/etc.). Naturally, problems may occur during this update which will lead to increase down time.

I will perform this necessary update in the devices locates in SASE1 and SASE3 tunnels today at 11H30. If somebody sees a problem please contact me.

Concerning the devices in the Experiment Rack Rooms, I will contact a responsible person to schedule a time slot where this update can be perform. If are one of these people contact me.

The MicroTCA crate used in rack 3.4 (XTD10) has been replaced. The system is up and running. Timing information is properly distributed from this system to the MicroTCA system in rack 3.2 and XGMXTD10.

Hello everyone,

The main Timing System at DESY, which distributes all timing information to our systems, is right now configured to generate 1Hz trigger signals on Event #16. This is the trigger that is deliver to most detectors/cameras on our site.

DESY is testing a new Gun and wanted to use a slower trigger frequency during this evaluation.

We were not inform in advance about this reconfiguration. After a few colleagues from SPB approach us reporting this problem, we contacted DESY and got the update. We will take steps to make sure that future Timing changes done by DESY are communicated to XFEL ahead of time.

Hi,

the FXE Mono's won't be available via Karabo from 15.30 onwards due to an update from CAS (karabo) and ITDM (server).

Best, Dennis

PSPO:

- G.Wellenreuther inform that MKK is planning to test the Umschaltautomatik, this will affect XHE1, XHE3, XHPSC, XHEXP, XHVAC. Oleg Engelmann will send a mail with detailed information.

SPB:

- AGIPD is fully occupied again and restarted this morning as scheduled. Everything works well so far and it is expected to finish the test on time. Further test with new Microcontroller are scheduled for next week.

- Most of the loops that required modifications are updated or on going.

- Module BKF00128 will be modified this week, update of Loop 5 will folllow.

- Loop 12 is ready to be deployed, to confirm if modifications in the modules are finished and reorganization on rack UH2.12 is ready.

- It is necessary to correct an error in module BKF00220,  AE will agree with SPB  when this will be possible.

FXE:

- New Vacuum device in operation.

- Karabo 2.2.0 being deploy today for FXE.

AE:

- Replace one Micro-TC in SASE3, a cooling unit is failing and one module will be replaced by a new one today. This is providing timing for XGM and K-mono. AE will inform by the logbook before to start the replacement.

Dear Monica,

Just one small correction at "Karabo 2.2.0 being deploy today for FXE." The version is, actually, Karabo 2.2.1 that we released yesterday.

Hugo

Dear all,

we got the information from MKK, that they have to fix a leak in the main compressed air line. Therefore they will close the valves to the tunnels for approximately one hour starting today at 1 p.m.

Please do not use the compresed air system in the tunnels during that time with air driven tools or air guns, as the volume of the distribution systems acts as a buffer.

Regards,

Martin

Dear all,

according to H. Sinn the closing of the XTDs 2, 4 and 10 will be delayed until the 30th of January.

Cheers, Gerd

PSPO:

- G. Wellenreuther inform that DESY is commissioning a new gun in the injector that could take longer than expected. Harald Sinn is in a meeting today to disccuss this issue, any change in the scheduled will be reported.

SPB:

- Lost motor positions in the tunnel, CRL in defined state. Good state in general no major problems expected.

- Detector was clabled again.

- Loop updates still pending, SPB request to have loops ready during this week.

FXE: No one attended

Vacuum:

- SASE1:

• Some works in CRL for XTD1.
• Working in FXE grating chamber, necessary to update Vacuum loop.

- SASE3:

• M1 and M2 mirrors were installed before X-mas. Mirrors installation in K-mono to be finished next week.
• Created new loop this affects Vacuum loop, update necessary.
• XGM loop need to be updated.
• Motion loop 1 is working, some works in MCP motors on going.

Diagnostics:

- Technical commissioning of MCP on going together with the Russian collegues.

- MCP SASE2 will be transported to XTD6 on Wednesday afternoon.

AE:

- Working to implement redundancy into FXE loops.

- To agree a date with FXE to work in the grating chamber at the end of XTD10.

- Motion loop for SASE1 will be interrupted this afternoon at 14:00 to add some terminals for mirror 2.

Software:

- Deployment of Karabo 2.2.0 will start this Wednesday in FXE.

- Next dates for Karabo deployment are: 15th January for the Tunnels,  22nd January for SPB.

TS:

- Torsten Schoen inform that it is necessary to carry out an 'Emergency off button' test during the Shutdown.

- SPB agreed to perfom the test the 22nd January in the morning.

- For FXE the proposal is to perfom the test the 17th January, to be confirmed with AE and FXE.

Dear all,

This is a reminder to inform you that Readiness meeting will resume today at 13:00.

Best regards and Happy New Year !!

PSPO:

• M. Areste presented a report about the progress so far achieved in the shutdown (see Indico for details).
• Most tasks are ongoing, on schedule.
• Few still have to be scheduled, mostly because support groups wait for feedback.
• C. Youngman does detailed coordination of SASE3tasks.Will organize meeting on Tuesday 9:30 for SASE3 tunnel only
• We will review the progress again in Mid-January!

SPB (M. Sikorski):

• AGIPD was tested.
• Roll back to crystallography configuration is ongoing.

FXE: No one attended.

Vacuum (M. Dommach): 

• 1st mirror M1 in SASE3 installed, chamber closed, pumped down & leak tested
• Preparations for 2^nd mirror in SASE3 installation ongoing, to be installed next week.
• MCP taken out of SASE3, put in parking position and pumped down again.
• Grating chamber from FXE was installed by JJ X-ray, is to be pumped in CW 51.
• SASE2 XS2 beam pipe behind shutter and lead wall installed, ready for TÜV inspection next week.

Diagnostics (A. Koch):

• XGM in XTD1 is almost cabled, more cabling was done in XTD10.
• Gas supply system in XTD1+6 was worked on.
• Photonics Science camera was tested.
• Trouble with the motion loop in XTD2 and XTD10, reported to AE.

AE (P. Gessler ):

• MCP motion and k-monochromator were tested.
• Motion loop was split in SASE1 tunnel.
• Many tests, lots of work in progress, need to act quick as time is short.
• T. Wilksen (DESY) is going to update the timing hardware è devices would have to switch to local timing, and switch back later to be operable.

Laser (M. Lederer):

• Further commissioning of the PP-laser was started.
• Upgraded Karabo and PLC firmwares. Works are on schedule.
• Measurements on the dependence of humidity are ongoing.
• DESY is working on the timing system, laser synchronization will be down until the 21.1.18.
• PP-laser pattern selection with machine timing:DESY is now pushing for the originally planned implementation. DESY will have to do some software development in this context.

Detectors (J. )

• Tested the AGIPD @ SPB, but 1 board was not working. Detector consortium could not provide a replacement board, so only half of the detector was tested before SPB/SFX had to close the detector vessel again.
• LPD was tested with train builder, but only tests using 2 memory cells could be conducted (2 images per train), so train builder was not tested with all memory cells.
• More tests with full load (i.e up to 510 images per train) still have to be performed.
• Test with DSSC will take place in HERA South in CW 51.

ITDM (K. Wrona)

• Monday til Wednesday there will be no DAQ due to extension of the Infini-band bandwidth, upgrades, firmware updates, recabling etc.
• DAQ up again on Thursday

Software (S. Brockhauser)

• CAS-People are currently working only in SASE3. In the future, they can either work on SASE3 or SASE2, but not both!
• Slides with Project-plan provided by Hugo Santos (see Indico)
• Release of Karabo 2.2.0 on the 18.12. (in the meantime shifted to the 20.12.)
• Testing by ITDM and consecutive deployment in tunnels and at the instruments takes until 22^nd of January
• Karabo 2.2.1 to be expected mid of January
• Further decisions will be taken mid of January

The LENS_3X limit switch (switched when driving towards the chamber center) is broken and must be replaced, hence further work is required on the next JJ visit 9/10 Jan (TBC)

The in vacuum components also need further work during the above visit. The lens that lost alignment during commissioning was also found to be more severley damaged than anticipated - the B4C guard is cracked and this deposited some fragments under the lens. All lens cassettes will be exchanged, rebuilt and realigned in Jan.

The patch panel has been removed for rework (to switch motor direction)

The Be lenses seem to be in exactly the same condition as 'new'

General

• Meeting on Friday will take place at 1300 due to overlap with XDAC
• M. Areste shall ask for updates on task list on Thursday

Instruments

• SPB
  • Work on AGIPD is ahead of schedule. Closing of detector should happen at the beginning of next week
  • Additional installations for roughing pump system is going on and will probably finish this year

Tunnels

• Diagnostics group currently working in the Southern tunnels, cabling components and aligning imagers
• Colleagues from Russia are coming 3 - 20 Jan to work on MCP in SASE3
• KMono local tests in SASE3 should also happen in early Jan
• Diagnostics requests scenes for XGM in XTD1

AE

• Vacuum loop in SASE3 will be modified. Some components shall be moved to a separate loop, e. g. Soft Mono
• SPB tunnel loop will be re-deployed to include the attenuator
• The motion loop in SASE1 will also be split to allow more rapid updates for newly integrated components
• Z-motion crate for detector in SPB will be done until next week or early next year

CAS

• Currently working with AE and ITDM on refining release schedule for KARABO
• Testing the version for release next week. This will then first be used by ITDM and is needed for the firmware upgrade of AGIPD
• Work is going on on the XGM train ID problem
• FXE will be moved to a separate topic (broker) to disentangle some dependencies at least temporarily
• FEL imager performance problem was investigated. This runs on a server that is very busy and should be moved to a different server

FYI

Author: F. Saretzki

English version below

Ankündigung:

Aufgrund von Sachverständigenprüfungen wird es am folgenden Termin zu Auslösungen des Evakuierungsalarms kommen:

Gebäude: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 und 8
Datum: 13.12.2017 ca. 09:00 Uhr – 11:00 Uhr


Sie brauchen bei ertönen der Sirene in diesen Zeiträumen das Gebäude nicht verlassen.

============================================================================
Announcement:

Due to expert testing there will be an evacuation alarm on the following date:

Building: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 and 8
Date: 13th of December 2017 between 09:00 a.m. and 11:00 a.m.


You don’t need to exit the building while the alarm occurs.

 

General

• Starting from next week shutdown activities will be tracked using M. Aresté's list
• Dispatch meeting on Monday and Wednesday at 13:00
• Next Friday Readiness meeting also at 13:00 because of conflict with XDAC

Instruments

• FXE:
  • Currently focussing of implementing all default parameters into the next firmware release
• SPB:
  • Major detector work with AGIPD disassembly started

Tunnels

• Currently testing firmware and software in SASE3
• Installation of first mirror starts next week. Mirror installation takes about 1 wk/mirror
• CRL in XTD2 will be vented to align internal mask and fix stuck arm
• Drilling takes place at the end of XTD9 for the installation of the FXE diamond grating
• Mirror cooling will not be installed this shutdown. It is not needed for the planned bunch rates and might introduce unwanted effects/vibrations
• Activities of Diagnostics:
  • Photonics Science camera in SASE1 exchange.
  • Tests are being carried out in SASE3 together with J. Tolkiehn
  • Cabling is going on in XTD1 on the XGM
  • There is a noise problem on the Gotthardt detector in the HIREX. Need support from AE to find the origin

AE

• Motion loop in SASE3 will be split into mov1 and mov2. This will allow to sort components into more and less stable sets and keep one of the loops up most of the time

Laser

• Laser activities had to start late due to problems with AC and laser interlock (delays of 6 and 5 months respectively)
• Availability of trigger pattern from the machine was also a problem
• Specs could just be reached for the Nov user period
• Eventually the PP laser was not used for any of the experiments
• Work on the laser is far from finished and will require 3 more months to come to full funtionality
• Currently, the Karabo version is being upgraded and the most up-to-date PLC firmware is being installed
• These updates had intentionally been delayed until the end of the user period.

Detector

• Detector group not present at the meeting
• Information from SPB: major work is ongoing on AGIPD

AE

• Train builder is being tested as it was not used so far. This required re-cabling
• PLC motor loops are also being split in SASE1
• Many activities in SPB
  • Remaining modifications of modules have started
  • Export of input for firmware generation for loop 8 has been provided by EETF
• Waiting for go in FXE
  • Need to work on the interface for the EPS system
• No tasks currently from laser group
• New information available on SmarACT stages: The observed problems arise from a cable problem and can be solve by configuration
• The problems with movers in the sample chamber in SPB (non-functioning encoders) arise from cable cross talk at the vacuum feed through. This require hardware re-design
• In SASE2 there are tests coming up together with the diagnostics group

ITDM

• Train builder tests are scheduled next week
• Recabling of Infiniband network happens on 18/19/20 Dec. No DAQ and controls available on these days
• A summary of DAQ statistics was shown (see slides on INDICO)

CAS

• Release of KARABO 2.2 on 18 Dec
• Freeze of functionality on 21 Jan
• Discussion reveals that a discussion on specific deployment dates and needs of different groups is urgently needed (Instruments, Tunnels, DAQ)

Please note following info from DESY:

Author: F. Saretzki



Ab sofort werden die Rack-Löschanlagen in den südlichen Tunneln in Betrieb genommen. Die Anlagen bleiben danach scharf.



 

The Motion loop in SASE1 is totally operated again.

General:

• Shutdown started.

Instruments

• SPB:
  • AGIPD prepared for exchange modules intervention
  • AE is in contact with SPB for crates issues fixing
  • The schedule for next year, especially when the experiment has to be ready before the users run has to be known as soon as possible.
• FXE
  • Starting loops update in the next days. Reason for the small delay is that FXE is checking all the default parameters to be able to restore them after the update.
  • In order to solve some DAQ issues, FXE requested to have their own topic. AE agrees, ITDM agrees, CAS is trying to find out how to do it and to do it before the next users run. Technical details have to be discussed offline with FXE.

ITDM

• in KW 51 (18-22 Dec.) the DAQ and online cluster will not be available due to recabling works in the balcony rooms.
• Update of the servers in the experiments has to be done. ITDM requested to know from CAS when the major update of Karabo is taking place to coordinate the effort and minimize the downtime. CAS will present a schedule concerning Karabo this Friday.
• 700 Tb of data are stored on the online cluster. It has to be communicated by the experiments what can be moved from there, since there is almost no space left.

Diagnostics

• Implementation of Gotthard detector in Karabo is still a problem. S.Hauf will invite the main groups involved for a meeting to find a solution.
• The data correlation with time stamps when using the Karabo DAQ has to be done carefully and contacting WP74 since there are still issues present. 

AE

• Activities on the Hirex in SASE1 will need the MOV loop to be brought down in the afternoon. An entry in the ELOG will be written.
• SPB loop 8 prepared. 
• Full list of new crates belonging to the motion loop in SASE3 tunnels has to be delivered to AE
• Full list of new crates belonging to the IPS loop in SASE3 tunnels has to be delivered to AE in order to be able to close the loop.
• Technosoft controller issues in SASE3 will be attacked once the SASE1 activities will allow it.

VAC

• VAC testing ongoing.
• 2 mirrors will be installed before Christmas
• Motion crates for the motion lopps and test of the VAC interlock will happen this week.

General:

• User operation with good beam conditions over the weekend
• One interlock trip in the FXE branch in XTD9. Restored by CAS
• Shutdown starts tomorrow at 08:00.
• Next dispatch meeting takes place Wednesday at 13:00. A reminder will be sent.

Instruments

• SPB:
  • Started with the actual sample during the night shift
  • Some DAQ issues
  • 108TB data was recorded
  • Problems appear to occur between 01:00 and 02:00  and between 13:00 and 14:00 
• FXE
  • Running is ok. 
  • Temporal and spatial overlap with optical laser established
  • Currently taking pump-probe data
  • People on shift were uncertain about how to recover from interlock trip
    • Statement from Vacuum: Pumps can restarted/valves open if the respective button in KARABO can be clicked

CAS:

• Delay between data taking and "online" analysis in the "Hummingbird" software is about 0.3s

ITDM

• On Friday, there was a global network outage affecting the Office Network on the DESY site.
• The transponder system in the experiment hall was down
• SRP organised a guard at the hall entrance so that the hall had not to be evacuated
• The control network (XDAC) was not affected

General:

• User Operation ongoing until Tuesday 5 Dec 8 AM
• CAS shall present a release plan for KARABO 2.2 next week
• Dispatch meetings shall move to a Mo/Wed 13:00 schedule starting next Wedneday 6 Dec

Instruments

• FXE:
  • Scantool was successfully used to establish overlap with laser beam (Tangerine) 
  • Some problems with slow DAQ in particular also for tunnel components
  • Comment from ITDM: There are known issues with the slow data acquisiton. A new version exists that was intentionally not deployed before the end of UOP
• SPB
  • Just finished first shift of laser user experiment
  • No big problems
  • All goals in the optimistic schedule were achieved

Tunnels

• Currently preparing for the shutdown
• Sticking condition of mirror bender could be reproduced in the lab
• WP74 just installed the last missing pieces in XTD1

Laser

• Running
• temporal and spatial overlap with beam is achieved
• But now instruments do not intend to use the PP laser for the last experiment

Detectors

• Both detectors are working
• The DAQ for slow data is crashing occasionally
• Gotthard is fully operational but there problems getting the data out. Help from PSI is needed

AE

• No problems

ITDM

• No problems

CAS

• CAS-XPD Running:
  • We will finish next Monday
  • We have been addressing issues related to MCP, Karabo-DAQ, Cameras, and K-Mono
  • Deploy K-mono today
  • Review DAQ Groups in Karabo: Cyril (interface to ITDM), Liubov, Koch
  • Gotthard we have to address in a separate and more systematic discussion: trigger, operation, calibration...
• XGM memleak - we already have a fix to get it stable but have to check when we will be able to upgrade (only device).
• Internal discussion about CAS/Karabo priorities for the winter shutdown

General

• Preparation for the user's run starting tomorrow (until 7:00 Tuesday morning)

Instruments

• FXE:
  • On Monday night the DAQ seemed to be slow. This problem was caused by a misconfiguration in the DAQ which could be solved by ITDM once they were informed. In general both the Configuration understanding by the users and the communication with ITDM has to improve.
  • The Gotthard detector could not be used. The Greateyes detector is being used now and works properly.
  • PP laser is now synchronized but not commissioned yet. This is going on now (crucial point is the pulse width measure). It is not yet clear whether the PP laser will be used in the next users run. 
• SPB
  • AGIPD was moved 5 m downstream for the next experiments. Vacuum is OK. Alignment checked with the ALAS, will be checked with beam probably today evening.
  • A change in the DAQ configuration has to be (is being) done today (CAS/ITDM)

ITDM

• Upgrade switches today at 9:00 will be checked if there were problems right after this meeting.

General

• User operation ongoing in SPB/SFX and FXE

Instruments

• FXE:
  • User operation started during the night
  • Some concerns over running LPD as the sample chamber is from diamond and direct reflections could cause damage
  • Generally instrument worked very well
  • Synchronization with PP laser is now in a state where it can be tested
• SPB
  • Many data taken for detector calibration
  • Severe issues with KARABO
    • Response is very slow
    • Major crash this morning
    • Detector controls complex and instable
  • A failure of AGIPD related to FPGA not understood
  • Strange interlock issue: EPS interlocks appeared disabled

Tunnels

• L. Samoylova spent many hours on Wednesday to set up the DAQ in the tunnel
• With the help of many support people she managed to get the system to work
• The system is instable and also seems to record many more parameters than are needed
• Use of the system is very clumsy
• Comment from ITDM (L. Maia): The data sources were specified by the users. It is acknowledged that the system needs a lot more work
• Diagnostics is working with CAS in a squad to work down a list of open redmine tickets
• The DOOCS - KARABO bridge is now working. The FEL imager is routinely available in DOOCS to the BKR crew
• DESY set up a statistics server to help collect data about beam pointing instabilities

Laser

• Local pattern generation synchronized to machine timing is working
• Overlap finding together with FXE was done but has to be tested under real condition next week
• A few suddent shutoffs of the front end occured. The origin might be understood
• There are large timing drifts in the ADCs that are currently not understood (up to ms)

ITDM

• Next Wednesday there will be a 1h shutdown of the load balancer in the DESY RZ. This should affect mostly the web services
• It should be noted that the DAQ ran quite stably in SPB and FXE. More than 20 runs were collected in each case without a restart

CAS

• Two hot fixes are out (2.1.18.4/5). These address  very specific issues
• A major new release (2.2) is planned for the winter shutdown. This will be mandatory
• Comment from instrument: It must happen early so that wrinkles can be ironed out!)

General

• User experiments resume tomorrow morning
• Beam should be coming back late this morning or early this afternoon
• After the Operation Coordinaton Meeting H. Sinn, L. Samoylova and the instruments will align the mirrors
• Beam to the instruments should be available directly after this

Instruments

• SPB:
  • The last night with beam on Monday was used to take flat field calibrations with AGIPD
  • The timing for overlap with the optical laser was adjusted
  • 5 out of 6 SmarACT rotational stages are now operations. Some problems remain and AE is talking to the manufacturer
• FXE
  • Request for use of the beam after 9 PM
  • Can be given to SPB starting at 5 AM tomorrow for final setup 

Detectors

• Connection to the ONDA system was set up for the detectors
• Significant latency develops quickly an saturates at approximately 2 mins

ITDM

• Simultaneous use of the DAQ system by both instruments should be done carefullly

AE

• no further comment

CAS

• no further comment

General

• Intensity and availability of beam was good over the weekend
• Large drifts and jitter were observed that are significantly higher than previously
• Beam has two horizontal lobes as if it is too large for the mirrors

Instruments

• SPB:
  • Many tasks were achieved during the weekend, e. g. instrument alignment, detector calibration
  • Issue with encoders whose cables are close to motor cables persists. These encoders are being disabled
  • Rotational SmarACT stages still do not work and are needed for this week's user experiment
• FXE
  • At shift start on Saturday after reloading beam parameters there was no beam in the hutch.
  • Beam had to be brought back by hand. This was a large effort
  • This situation did not repeat on Sunday
  • LPD detector had problems: Only one super module correctly recorded data. Comment from S. Hauf: FXE should have called the detector expert
  • PP laser synchronisation does not work properly yet: Increments when adjusting timing are too coarse. Scan range is limited to 18ns

Detector

• Detector group was not informed about the LPD issues in FXE. 
• AGIPD calibrations were successfully taken on Sunday

ITDM

• Still work is needed for slow data DAQ
• DAQ behaves differently in SPB and FXE. In FXE it appears overloaded and in SPB it works fine
• Comment from CAS: 
  • Many changes were made in the KARABO framework to allow asynchronous operation
  • These changes have to be also pulled through to DAQ. This will only be done during the winter shutdown
  • Until then, DAQ is very vulnerable to any delay in the system

CAS

• No further comments

AE

• Currently working on problems with rotational stages in SPB

General

• Beam is back with good intensity (up to 1 mJ) 
• Next user period starts on  23 Nov
• Time between now and then should be used to optimize FEL, beam lines and Instruments

Instruments

• FXE:
  • Measurements of pulse intensity were done using an XGM from DESY in the hutch
  • A  pulse intensity of ca 40 uJ was found. This is in rough agreement with the expectations
  • Spectrometer was aligned
  • GreatEyes and Gotthardt detectors are working fine
  • LPD still to be checked
• SPB:
  • Using beam to do alignment
  • Not such a good focus (too big)
  • Plans are to continue with instrument commissioning today and Saturday
  • Work is currently being done by Mahadi on the SmarACT rotary stages
  • FXE points out that most likely SPB operated with the FXE CRLs inserted into the beam
  • Note: The standard "switching script" from Harald should be used to make sure parameters and devices are properly adjusted when beam is switched from one instrument to the other
  • Detector calibration on Sunday

Tunnels

• Last Friday there was a Bender failure on mirror M2
• The mirror was removed from the tunnel during the weekend and the SASE2 spare was inserted 
• In the process one Beckhoff controller was found broken
• In the removal process a further problem was caused in one of the Beckhoff loops
• This problem could be fixed from outside the tunnel 
• After this 20h were spent to redo Beam-Based Alignment in 7 iterations
• The new orbit deviates from the old by up to 80 um.
• After using the new orbit, the FEL lased quickly with good intensity
• The replaced mirror M2 could be re-aligned 
• On Thursday beam was available to FXE
• Remark from J. Grünert: Instruments should make sure that most up-to-date DOOCS panel is loaded to see proper XGM info
• Update of DAQ caused DAQ in tunnels to be broken. This is now fixed according to K. Wrona
• HIREX commissiong is going in parallel 

Detectors

• AGIPD:
  • Before the bender failure a few X-Rays could be taken on AGIPD
  • It looks like 30 X-Ray pulse can be taken now
  • Additional flat field calibration is needed now
  • On Sunday a full shift is foreseen for commissioning with beam
• LPD:
  • One quadrant was disassembled to find the cooling leak.
  • Leak now appears to be fixed
  • No sign of dead spots on detector

Laser

• We are operating in shifts:  early / late, although experiments have been cancelled, performing shift change and other excercises for commissioning.
• Operating the laser with machine timing was so far hampered by the fact that the transmitted laser patterns from the machine were limited to the actual number of x-ray pulses, i. e. mostly non-deterministic 1 or 0. Optical alignment and characterization is not possible under those conditions. Therefore, on the weekend we have switched to local timing to give guys some power.
• Following this experience, we have decided to attempt local generation of laser patterns under machine timing in order to be independent of the machine and not have to switch back and forth between local and machine timing. This has been implemented and is under test today and over night. We still observe spurious long term drifts, which are under investigation.  However, if finally successful, this can only be a make shift solution for the coming weeks. Mid to long term we need proper implementation by DESY, allowing pattern changes on short notice and independent of the machine. This is in preparation (time line ca. mid 2018).
• Currently, beams are sent to FXE and SPB with 30 pulses each, 800nm broadband, should be compressible to 15fs. This is under machine timing but with locally generated patterns.
• There are other curiosities concerning machine timing, which are related to the increased T0 arrival time jitter compared with our local timing. Debugging is ongoing. We are also testing various features under above conditions (machine timing, local pattern):  phase shifter, trigger shifter, shift change. Time line: depending on outcome, daily updates will be provided.

AE

• When the breakdown of the interlock occured in conjunction with the broken Beckhoff terminal it became obvious that crucial emergency phone numbers are not available
• Better emergency procedures are needed
• The special EPS operation mentioned to align the mirrors should be written down so that it can be implemented an no override of the interlock system is needed
• The list of shutdown activities compiled by PSPO contains a lot of work for AE. This is now being discussed. 

ITDM

• Some problems occured this week with DAQ of slow data. This should have been fixed yesterday

CAS

• Doing 24/7 OCD support 
• During switch between instrument CAS comes in to provide support
• The cancellation of user operation last weekend was not properly communicated so that people came in unnecessarily
• Next week additional work for WP74 is being done

Instruments

• SPB:
  • Waiting for beam to come in a usable condition (500 uJ and pointing within aperture of optical elements)
• FXE
  • Similar

AE

• There was an interrupt of the vacuum loop in XTD2 due to a faulty pump controller. This was fixed by ZZ
• Rotational SmarACT stage and slits in the sample chamber for SPB are being worked on

CAS

• Due to change of beam schedule, now working on implementing more devices in DOOCS-KARABO for Jan Grünert

ITDM

• Will upgrade DAQ software today. Note: Still based on KARABO 2.1.16

Dear all,

I received updates from DESY (M. Hüning) about how the access to the southern tunnels is being restricted in the next months, due to interlock tests etc., and included it in the TAS. Please check, if you have work to be done in these tunnels:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

General

• No user operation.
• During the week end the accelerator was not able to deliver reliable and stable beam for the user operation.
• The M2 mirror bender stopped working on Saturday morning and had to be replaced. The mirror and blender were replaced as a whole durig the week end. Normally it would have taker weeks but a "spare" system was already set up for SASE2 and could be used on a short notice. Pumping of the M2 section is ongoing and leak test be performed today.
• The non working M2 does not show apparent problems so far, and the reason why was not working is not clear.
• Some sections of the LINAC were warmed up due to some maintenance required for the cold compressors.
• LINAC is being cooled down and ready for operation today afternoon.
• The technical support groups were not informed about the problems until saturday afternoon. The communication channels in these cases have to be improved.

​Instruments

• SPB and FXE:
  • The goal is to be able to strat the secondo users run on Thursday (Day FXE, Night SPB).

Tunnels

• XTD2,9 Ready to take beam in the afternoon (if no leak in M2 is discovered).

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on Wednesday

General

• User operation is delayed by 48h due to rocky startup of FEL.
• FEL still has not reached acceptable levels of stability and power

Instruments

• SPB:
  • Instrument was brought back to a somewhat running state
  • There are still problems with motors in the sample chamber
  • Alignment with beam is ongoing at this time
• FXE
  • New scan tool was tested
  • Currently working on user-specific alignment tasks
  • Laser was routed to the sample position
  • The beam is still drifting around

Tunnels

• FEL had trouble to find lasing
• There was a controled access on Wednesday to fix various problems. However, further problems appeared
• It was decided to roll back the changes made for a new injector laser, but this did not improve the situation
• At some point 400 uJ were reached and lost again
• There are many problems with the undulator system
  • Incorrect values in lookup tables
  • Difficult operation of the entire system
• FXE should be able to start up tonight
• SPB would follow Saturday morning since time is needed for alignment during the day today
• Shifts for photon diagnostics were largely cancelled due to FEL problems
• KMONO gave some interesting results pointing to the problems in the undulator system
• Next week FEL Imager and KMONO will be installed in SASE2

Laser

• Both SASE1 instruments have laser beam with local timing. 
• Tests with machine timing start today at lunchtime

Detectors

• AGIPD is in best possible state. 
• Firmware and DAQ were upgraded
• Some commissioning time with beam is needed today
• LPD had a cooling leak  and a power problem last night
• At 6 pm yesterday, this was still not fixed

AE

• Continuing small activities for user operation

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on this Monday

CAS

• Dedicated squads for the setups of SPB/FXE, DOOCS integration, Scan Tool closed now
• Tools delivered but with significantly reduced scope
• All necessary things for DOOCS Karabo integration were delivered. Now waiting for DESY

General

• There is currently no beam less than 24h before user operation is supposed to start in SPB
• The affects everybody (Instruments, Detector, Laser 

General

• Recomissioning of the FEL has been bumpy and up to this point only very weak lasing could be achieved. 
• So far SPB had only little beam and FXE had no beam at all. Both instruments state that the commissioning time for the up coming user run is insufficient

Instruments

• FXE:
  • First SW and DAQ stress test on Friday went well.
  • Some problems appeared with the slow data that should be fixed by now
  • K. Wrona states that these problem originated from missing permissions due to change of userid
• SPB 
  • Entire instruments needed re-alignment with beam since the CSS was moved during the shutdown
  • Very low intensity and not very stable beams were available on the Weekend. The beam could not be properly focussed
  • The bender for M2 cannot reach the flat condition. Is this the temperature problem again?
  • One pump was overheating
  • AGIPD tests with new bunch patterns started this morning
  • Significantly more time needed before user operation
  • There were some problems with the interlock system that trigger beam aborts

Tunnel

• Commissioning program was very limited in the last few days
• KMONO got some first interesting results
  • Rings and Kicks could be seen
  • Some wrong undulator settings were identified

Detectors

• Firmware update was done to record all 30 bunches. However this still has problems
• Need beam for calibrations
• The planned update by ITDM of the DAQ was not yet done

Advanced Electronics

• The mentioned interlock issues in SPB require an update to loop 7
• In loop 4 need to reset the "all valves open" signal by hand if it trips
• The SPB Master loop needs updating both for MPS and timing

Software

• Tests of the scantool took place with hardware. These tests continue today
• A new data model is being worked on for the scan tool but this is not available yet
• SPB states that at this point they will not make use of the scan tool as they first have to consolidate the instrument
• FXE is involved in the CAS tests

ITDM

• As mentioned before the DAQ version is still old and needs an update. This contains many improvements
• IBM discovered a bug in GPFS and published a patch. The bug could lead to data corruption
• Decision: DAQ update and GPFS patch will be installed today in the morning.

Dear all,

I updated the TAS to include the upcoming user runs up to and including the winter shutdown. As you can see, there will only be access to the XTDs in cases of urgency. In these cases, please contact Martin Dommach, Harald Sinn or Thomas Tschentscher.

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Cheers, Gerd

4.11. from 08:00 until 5.11. 08:00 - Jan Gruenert by email (work mobile defect)

5.11. from 08:00 until 6.11. 08:00 - Naresh Kujala 9-6840 

General

• The rack layouts and rack placement were finalized during the PRR in August. The result is the basis of the network planning. The network is currently being installed and will most likely be finished by the end of the year. Changes to the layout that affect the network planning are no longer possible. 
• The TC/XHEXP1 meetings shall be combined into one bi-weekly meeting starting on 9 Nov. A separater new invitation shall be sent

Instruments

• SPB
  • Currently preparing for the next user period
  • A lot of work still needs to be done for bringing the laser to the interaction point. This mostly concerns the beam pipe
  • Apparently there was an interlock problem related to SPB components in XTD9. SPB components were taken out of the EPS loop without SPB being informed
    • Comment from T. Freyermuth: The condition of SPB devices was preventing beam operation and noone of SPB could be reached
• FXE
  • A lot of progess was made together with CAS
  • This afternoon a DAQ stress test with several detectors is planned
  • The encoders on the CRL lens stack were disabled as this complicates the operation at the moment

Tunnels

• XTD9 was closed yesterday. Beam could arrive this weekend
• About one shift worth of work remains in XTD10 before readiness for lasing
• KMONO measurements took place on Wed/Thurs from 3PM - 3AM
• Some DAQ and KARABO issues surfaced
• But it was possible to record donut radiation patterns
• Now commissioning of HIREX is being prepared
• XGM requires some minor re-commissioning

Laser

• Beam is in the ILHs. 
• Still problems dealing with the capricious AC system
• The promised specs of the laser beam are met
• Synchronization is still to local timing. Synchronization to the machine timing only comes next week and no time remains for testing before users

Detectors

• LPD:
  • 15/16 super modules are operational
  • This week calibration data from charge injection were taken
  • The detector is ready for users
• AGIPD:
  • Detector is fully working again after a big rearrangement of cables
  • It was found the the clocking-control cable is too long and needs to be re-routed during the winter shutdown. The detector will be fine for the next users
  • Work is ongoing on the firmware in order to allow 30 frames/train rather than only 15

Electronics

• SPB:
  • Updated the tunnel loop for SPB
  • Loop 12 is up-to-date
  • Loop4: The mirror mounts were tested locally
  • The is no feed back on interlock behaviour in loop 5. If there should be problems individual conditions can be disabled
  • For loop 7 zoom and focus and mirror mounts to be tested
  • Same applies to loop 2
• XTD9:
  • want to update EPS loop to include removed SPB components today
  • Vacuum loop to be updated next week
• XTD10
  • Some ion pumps show an error condition in KARABO. J. Tolkiehn is investigating

ITDM

• User handling for log books has changed. Please consult the help log book for details or consult J. Szuba
• The meta data catalogue was updated so that instruments can now annotate with their own credentials
• Folders for the next two user proposals are created, also in the online environment
• A new version of the DAQ software needs to be deployed in order to fix a bug in slow data daq

CAS

• A number of issues need to be resolved with Diagnostics as already mentioned earlier
• The Gotthardt Detector is now working in FXE
• The promised reduced scope for the scantool has been achieved. It will be made available to the users asap, most likely today

Hi

Logbooks for the first two upcoming proposals in FXE and SPB instruments have been set up and all missing users in the past 6 proposal have been corrected. From now on, user registration is disabled, logbooks for next proposal will be set up automatically once the proposal info is migrated from UPEX to MdCnix. Any subsequent changes (like a new user added in MdCnix to a given proposal) has to be done on request either by admin of particular logbook or ITDM.

Standard setup of beamtime logbooks is the following

• Admin list contains ITDM staff, instrument support group (as given by relevant instruments), PI and Main proposer as given in UPEX
• User access is given ITDM staff, instrument support group and all participants of a given beamtime.

All subsequent customisation, including adding extra users, should be made by logbook admins.

Regarding logbooks for the past proposals in September, not all was corrected according to the above defaults, especially there may be wrong email addresses put at that time by users during registration -this can be fixed as well, taking info from UPEX.

Also HELP logbook was updated.

This change affects also XFEL internal usage of ELOG, as from now on, a new user has to be added by a relevant admin.

Cheers, 

Janusz
 

Instruments

• SPB:
  • When PLC loops are restarted, still many device parameters are not properly recovered. This generates a lot of work for the team. Recently an entire day was lost.
    • The procedure now foresees a notice to the relevant group before each loop update. In particular, CAS will be asked for the last version of the parameters export to be re-implemented. The relative script was also updated and some bugs were fixed.
  • Loop 4 (maybe also 3) - MKB system - needs to be updated. This will be done as soon as the last eplan export is available.  
  • XTD9 loop needs to be updated with proper encoder names. CAS takes care of the missing names.
  • AGIPD Interlock system (loop 5) shows some issues concerning valves. It is not clear whether this is due to real problems with the interlock definitions or just to the fact that the system is not yet in vacuum. Chember will be pumped today and then the final test will be done.
• FXE:
  • Preparation for the users run ongoing. PP laser treatment test ongoing.

Tunnels

• XTD10/SASE3
• XTD9/SASE1
  • Si-Mono: work has been done during the week end. The idea is to have it running without tilt stages.

Laser

• Delay control is available through KARABO but requires ca 2 days testing: ongoing.

Electronics

• Currently the work that is still needed for SASE3 is being evaluated
• Planning for the next shut down is going on

CAS

• Working on DOOCS/KARABO integration together with DESY. Ongoing. General frame being built. Then priority on the FEL imager needed for first lasing.
• A. Silenzi and S. Hauff worked on the AGIPD procedures: Cooling down procedure successfully tested.
• Work continues on reading and writing data for the GreatEyes detector: The idea is now to test the parallel operation of LPD, Gotthard and Greateyes detector.

ITDM

• The folders for the next user run are being set up next week. GPFS is now fixed.
• SPB requested 18 new cameras. This effort needs to be carefully coordinated. CAS and ITDM are working on that.

Instruments

• SPB:
  • When PLC loops are restarted, still many device parameters are not properly recovered. This generates a lot of work for the team. Recently an entire day was lost.
    • Answer from AE: The parameters are extracted by CAS using KARABO. It is possible that not all necessary values are returned. AE is also investigating whether there are some bugs in the firmware procedure. It is pointed out that the change from 24V to 48V led to a huge amount of changes. 
  • Injector motion is now working along all three axes
  • AGIPD automatic power up procedure is now working. Tests continue
• FXE:
  • The spurious data losses with the Gotthard Detector were further investigated. They are not resolved but they do not limit operation with 30 bunches

Tunnels

• XTD10/SASE3
  • All vacuum sections are connected, locally tested and ready
  • Part of the interlock system is also ok. A global test of the entire integrated system still needs to be performed
  • The biggest challenge was the gas attenuator section with 7 turbo pumps. All pumps were locally test and integrated into the control system. The vacuum is very good (10-9) Now they are put into a  safe state.
  • R. Villanueva would like to be informed in case whenever the interlock in XTD10 is broken so he can inspect the system
  • XGM is connected to the overall vacuum and tested. The interlock is working and it is also connected to DOOCS
  • The two imagers requested for first lasing (SR/FEL) are available
  • Approximately one shift would be needed to take care of a few remaining tasks before first lasing
• XTD9/SASE1
  • The HiRex is fully operational
  • The Si-Mono needs work but this was de-prioritized as it is not needed for the next users

Laser

• Full specs were reached for both the SPB and FXE beams. The laser is in the respective instrument laser hutches
• The laser was covered in order to reduce effects of any remaining air drafts
• Differential mode compensation is available. Common mode compensation is not
• Work still continues on the 4 agree-upon bunch patterns. Another meeting is set up for today
• Delay control is available through KARABO but requires ca 2 days testing

Detectors

• For AGIPD both  for turning on power and cooling automatic procedures were implemented and tested
• LPD was moved to Karabo 2.18. 
• One charge injection run was done and the system operated for 2 days without interruts

Electronics

• Currently the work that is still needed for SASE3 is being evaluated
• Planning for the next shut down is going on

CAS

• Currently planning next OCD period
• Very good progress was made in SASE3
• Working on DOOCS/KARABO integration together with DESY
• Work on the laser is progressing
• A. Silenzi and S. Hauff worked on the AGIPD procedures
• Work continues on reading and writing data for the GreatEyes detector
• As reported by the detector group, the stability of the DAQ is much improved

ITDM

• A major problem caused the Maxwell cluster to be down for 3 days. IBM Technicians had to be called
• In this context it became clear that the communication with the users must be optimized
• Now the system is ok
• The folders for the next user run are being set up next week. This was delayed due to the technical problems
• For the experiment log books the users will now be automatically set up
• The update of the DAQ for AGIPD is somewhat delayed due to the long time taken for the recabling
• SPB requested 18 new cameras. This effort needs to be carefully coordinated.

The SASE3 tunnel XTD10 was searched by Janni and her team and interlocked about one hour ago.

It is now almost ready for first lasing in SASE3:

• The vacuum is completely connected and closed from V0 all the way to the end
• All valves, pumps and gauges are included in the PLC loop and can be controlled from Karabo
• The gas attenuator section was completely tested and is also under PLC control. The turbos are isolated from the rest of the vacuum and put into a safe state
• All three DPS's are operational, tested and included
• All imagers are operational
• The XGM vacuum is part of the general vacuum and controlled and monitored from KARABO
• The interlocks of the DPSs were succesfully tested

In order to allow beam, it is estimated that one access with approximately 8h would be needed to sort out some remaining smaller issues

I would like to thank the entire team from Vacuum, Optics, Diagnostics, AE and CAS for the big effort and in particular C. Youngman for coordinating.

General

• After discussion between XFEL Vacuum (J. Eidam) and DESY XFEL Run Coor (Matthias Scholz) it is agreed that XTD10 will only be closed on Thursday and that XFEL will search the tunnel between 15:00 and 17:00. (This information became available only after the meeting)
• This gives some extra time to finish activities in XTD10

Instruments

• FXE
  • Currently A. Parenti and FXE are hunting a spurious data loss in Gotthards
  • Otherwise the software is surprisingly stable
• SPB
  • Slow progress towards getting the Detector going again
  • The task to ease the acquistion needs help from the DESY developers

Tunnels

• XTD10:
  • A lot of progress was made by the team working in XTD10: Thank you very much!
  • The complete tunnel is under vacuum
  • Also the missing pieces between XGM/MCP and the rest were inserted 
  • Cleanup is done
  • Denis and Raul are still working on local tests of the gas attenuator system
  • Testing is somewhat slowed down because the GUI client no longer allows active manipulation of components at the moment. So the ikarabo interface is used
  • Work also continues on getting the mov1 loop going
  • Local tests are done on all imgers except on the exit slit imager
• XTD9
  • ​The FXE Mono is not finished. It needs a restart of the motion loop
  • AE will check when this is possible, but there are a lot of constraints from users and from resources
  • Since there has been a change in the interlock system, SPB needs to be in place when the shutter to XTD9 is opened. There is a risk of triggering the interlock and shutting down the machine.

AE

• Almost everything that was planned could be achieved in XTD10
• Some of the remaining work can be finished when the tunnel is closed.
  • Work is going on to finish the controls
• Even the interface to MPS that was originally not foreseen is now available

CAS

• The workaround that allows to modify parameters even though the protection does not allow this, is now in place. This should help with the trouble observed by the Vacuum Group.
• A new bug in the PLC was observed in this context. Changing a parameter whose name has changed leads to a crash. 

ITDM

• Currently the MAXWELL cluster is unavailable.  This is a serious software/hardware issued. The manufacturer is actually on site to fix the issue today

FXE:

• Pump-probe beam is now in the instrument hutch
• Currently working on setting up the remainder of beam steering and compression
• Gotthardt is working. Tests are going on a and some issues with the DAQ still need to be solved
• The sample stack needs to be moved up stream for the next user period

SPB

• Still working on AGIPD. Many cables needed to be disconnected and are being re-connected now
• AE finished all the scheduled modifications on the crates and are doing some more that were originally not scheduled
• A few details with the interlock need fixing
• The newest version of Beckhoff_Devices needs to be deployed
• The previously observed slowness of the GUI is probably fixed but needs more real life exposure. Final test are being done
• The full infrastructure should be back and operational Tuesday/Wednesday

Tunnels

• Work is going on on the gas attenuator
• Interlocks are ready for the WP74 DPS and the XGM
• Detailed steps to completion in XTD10 were re-iterated by C. Youngman. Additional coordination off line after the meeting
• L. Samoylova mentioned that the mirror encoders in XTD2 are currently in error. B. Baranasic will have a look

AE

• In the new version of KARABO, the rights for reading and writing parameters from and to the PLC are now being observed, differently from before.
• This causes problems. P. Gessler mentions that he is very concerned that such a major change is introduced so briefly before the next user period. This puts AE under great pressure and leaves almost no time for testing
• All the SASE1 loops were updated because of the new "feature" in KARABO
• A. Silenzi mentions that he can implement a workaround for this "feature" so that loops need no updating. However, this requires a new KARABO release

CAS

• KARABO2DOOCS: Meeting today with the DESY control people at 1430 in BKR
• ScanTool: Meeting foreseen Tuesday/Wednesday with the instruments
• The GUI slowness necessiated a hotfix which was verified within CAS. Now waiting for feedback from instruments

General

• SASE1 Instruments are asked to comment on progress with KARABO 2.1.18
• Tunnels XTD9/10 will be closed on 25 Oct at 1700

Instruments

• FXE (A. Galler):
  • KARABO 2.1.18 appears to be working ok. The device instantiation problem reported earlier in the week is no longer there. However, not all equipment could be tested yet
  • Work currently focusses on preparing for the next User Run and in particular on preparing the sample environment
• SPB (A. Mancuso)
  • A situation similar to the on in FXE is observed. Full proof of functionality of KARABO will only be possible once AGIPD is fully assembled again which is next week at the earliest
  • Currently, work is going on inserting the AGIPD cables into drag chains so that the detector can be moved on the CSS. This work should finish today

Tunnels (J. Eidam, R. Villanueva, T. Maltezopoulos)

• Leaky vacuum pipes were successfully welded
• Absorber section is closed and under vacuum
• MCP detector is connected
• DESY finished work on the upstream section in the dump
• The gas attenuator has very good vacuum (10^-9)
• Differential pumping is ready from the HW side
• Cabling for the gas attenuator section is still going on
• Alignment was done in view of the small apertures in the gas section
• Local tests can start now (This is 3 days behind the schedule made at the beginning of the week)
• PES is installed but will only act as a pipe.

Laser (M Lederer)

• NOPA Beam is now going to FXE
• Next week it will be brought to SPB
• The KARABO version used is several versions back with respect to the one in the instruments and there are still a few minor problems
• On Monday work on programming the laser burst patterns together with the respective people at DESY shall start
• Continue testing
• There appears to be a shortage of conversion crystals for the instruments. New ones have been ordered but are not there yet.

Detectors (P. Lang)

• AGIPD HW work is on track and finishes thie week. Software work starts next week
• LPD control system works ok with KARABO 2.1.18. 
• Some small issues with LPD calibrations

ITDM (K. Wrona)

• Two updates were done for the DAQ of LPD
• Some issues were found with the RC
• It was tried to install a new version. This was already done for AGIPD

AE (T. Freyermuth, B. Baranasic, J. Tolkiehn)

• SASE1 Tunnel:
  • Vacuum loop was updated without problems
  • The motion loop was updated a few times because there were problems with some motors. The work continues today
  • EPS loop was updated
  • On Monday/Tuesday the SPB loop will be updated to account for the terminal changes from 24V -> 48V
• SASE1 Instruments:
  • Loop 5 with the redundancy module is now ok
  • Loops 2, 4, 7 were updated
  • Next come loops 1 and 12
  • All modified modules (24V -> 48V) are installed but a few more might need to be changed
• SASE3 Tunnel:
  • Tried to get the vacuum loop up
  • But there are still many mistakes and inconsistencies with names
  • A big amount of work is still needed for the gas attenuator

CAS (S. Brockhauser)

• KARABO 2.1.18 was deployed
• Good progress was made with the Gotthard and GreatEyes Detectors
• For GreatEyes a specific driver on a specific computer will be used
• The problem with extra delays in the GUI client was resolved
• Scantool:
  • A plan of what to implement exists. This needs to be checked with the instruments
  • A first part of this tool is actually already implemented
  • A. Galler points out that he sees the problems predominantly in the integration with the rest of the environment
• DOOCS2KARABO
  • An example for pipelined data was sent to DESY
  • There is no showstopper seen, but the work is progressing according to priorities which are 1 Laser, 2 Undulators, 3 DOOCS2KARABO
  • Diagnostics points out that the will need both the scantool and DOOCS2KARABO for their commissioning period from 1 - 7 Nov. CAS should please provide information so that the tools can be tried

Dear all,

I updated the schedule for the tunnel access in the upcoming weeks based on the discussions in the PSO- and the Operation Board-meetings:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd