XO

FXE

• DAQ issue reported (data not being recorded). Comment from ITDM is that this was a miro issue solved with the re initialization of the proposal. More details will eb discussed offline.==> issue fixed (it was due to a misconfiguration in the DAQ server)

SPB/SFX

• Loop 12 has to be updated
• SASE1 M3 issues with the pitch motor lead to FXE losing beamtime. EEE to follow it up.

SCS

SQS

MID

• By the end of the week MID should send the requirements for the vacuum interlock to EEE. ==> tests are ongoing. AS will organize a meeting with CAS, VAC and EEE to define a test procedure to be followed by the instruments when testing the vacuum interlocks.
• Calibration server for mid has memory problems and needs to be replaced. Calibration pipeline has to be moved to a different servers. Will be discussed offline with MID.==>

HED

• Yesterday the loop 6 was updated. There were issue with pumps switching off and causing overhead for HED. PG will investigate why this happened... ==> the procedure was clear to EEE and foresaw the switching off of the pumps at some point. This was supposed to be clear also to the instruments buy clearly it was not the case. 
• Nanocube PLC has been integrated.

Optical lasers

​VAC

XRO

XPD

DET

EEE

• Crash of PLCs because of not a number values. The fix on the PLC side requires the update of the loops framework and can be done only in the August shutdown. By the end of the week a filtering tool will be availbale in Karabo.

CAS

• Karabo 2.5.1. hotfix is being prepared (Fixing 2 bugs in the framework and several GUI issues). ==> Done
• Tomorrow at 14:00 there will be a test of the pulse on demand feature (conditional triggers) in FXE. (AS/BF) ==> Done

ITDM

• Calibration server for mid has memory problems and needs to be replaced. Calibration pipeline has to be moved to a different servers. Will be discussed offline with MID.==> It is ongoing. In general this issue led to a loss of beamtime for MID. Alessandro suggests that in case of hardware failure there should be a joint ITDM-CAS decision/effort on what to do.
• Data migration: it was slow in the past days due to some script not being used for safety reasons. This is being fixed. Hovewer, due to the queue of old data to be migrated it will take some more time until full performance will be reached. 

SRP

TS

AoB

XO

FXE

SPB/SFX

SCS

SQS

MID

• Nanocube being used with the scan tool. Seems fine. Some test of the velocity scan has to be done (CY will coordinated with MID)
• Exapods have been integrated. Some feedback from TF is needed for EEE to complete the hand over.

HED

• Loop 6 will be updated next week. KS will talk to HED and EEE to make sure vacuum systems are safe during the update.

​Optical lasers

​VAC

XRO

XPD

DET

EEE

• Crash of PLCs because of not a number values. The fix on the PLC side requires the update of the loops framework and can be done only in the August shutdown. By the end of the week a filtering tool will be availbale in Karabo. ==> Filter is ready and can be deployed on Monday.

CAS

ITDM

• Some issues with the GPFS will results in slowness of the system. This is being investigated and probably already fixed today.

SRP

TS

AoB

XO

FXE

SPB/SFX

SCS

SQS

MID

• Nanocube being used with the scan tool. 
• Cleanup and preparation for the next experiments ongoing
• VAC interlock has been tested due to an accidental trigering of a vacuum valve. Some dcetails will need to be fixed.

HED

• Loop 6 will be updated as soonas the new interlock definitions will be available. KS will talk to HED and EEE to make sure vacuum systems are safe during the update.
• Loop 7 and 9 will be updated today.

​Optical lasers

​VAC

XRO

XPD

DET

EEE

CAS

• Crash of PLCs because of not a number values. Karabo client is now available to filer NaN values.
• 2.5.1. available. FIxes a bug for cameras (restricted ROI), and GUI machines. 

ITDM

• Some issues with the GPFS will results in slowness of the system. Has been fixed.
• Cleanup of the GPFS during the weekend. Petabytes of data have been recorded. This may reduce the time in which the data is available on the GPFS for the next user runs. 
• Xfel gateway will be shut down for a short time today around lunchtime. JS will send a mail to readiness with the precise time.

SRP

TS

AoB

XO

FXE

SPB/SFX

• Loop 12 has to be updated ==> Done.
• SASE1 M3 issues with the pitch motor lead to FXE losing beamtime. EEE to follow it up. ==> Basically fixed. Some remaining issues will be fixed during the shutdown week in August.
• Some issues reading the encoders of the JJ XRay CSLIT. EEE checks whether similar issues are also present in other instruments.

SCS

SQS

• Commissioning of smaract stages ongoing.

MID

• By the end of the week MID should send the requirements for the vacuum interlock to EEE. ==> tests are ongoing. AS will organize a meeting with CAS, VAC and EEE to define a test procedure to be followed by the instruments when testing the vacuum interlocks. ==> to be done.

HED

• New update of Loop 6 will be done propably today after new interlock definitions have been delivered.
• Hexapod: Loop 9 has been updated. Loop 7 to follow.

Optical lasers

​VAC

XRO

XPD

DET

EEE

• Crash of PLCs because of not a number values. The fix on the PLC side requires the update of the loops framework and can be done only in the August shutdown. By the end of the week a filtering tool will be availbale in Karabo.==> Karabo client has been deployed in the tunnels and FXE. The other instruments will get it during the August shutdown.
• XPD reported some issues with digitizers for MCPs. It should be fixed already by now.

CAS

• CAS received host alerts regarding sa2-br-sys-con-1, sa2-br-sys-con-2, sa2-br-sys-con-5 resources usage. To avoid the worst issues during experiments, we will do maintenance on these machines today from 12:00 to 14:00.
• CAS/SQS working on karabo scenes to display the LaserMet laser isnterlock status.

ITDM

SRP

TS

AoB

• RBean reported that by the end of the week a macro will be run by HSinn to decouple machine interlocks in SASE1 and SASE3. For more details AGaller should be addressed.

The meeting did not take place since only EEE was attending.

XO

FXE

• Photon3 detector has been fully implemented.

SPB/SFX

• Some issues reading the encoders of the JJ XRay CSLIT. EEE checks whether similar issues are also present in other instruments. ==> has been fixed.

SCS

SQS

​MID

• 2 Motors have to be checked: Rotation of the hexapod and the y motion of the AUX3 are needed for next week.
• DPS turbo pump 3 behaving strangely. EEE to have a look
• Loop 7 has to be updated. Offline discussion MS/JH

HED

• Grating holder motion HED OPT. Motors to be commissioned. EEE required
• Cable needed for the Jungfrau timing signal. KS BF to discuss offline.

Optical lasers

​VAC

XRO

XPD

DET

EEE

CAS

ITDM

SRP

TS

AoB

XO

The accelerator is now working at 2.25 MHz. Kicker can be used. DET informs that for repetition rates different to the ones been used so far the calibration can be significatively different.

FXE

SPB/SFX

• Hexapods are being integrated. 
• Master loop has to be updated. EEE to agree offline with SPB/SFX

SCS

SQS

• Commissioning of smaract stages done. 1 encoder cable is being checked (seems a problem of definition).

MID

• DPS turbo pump issue. A software workaround exists. The problem is that the used controller has not been commissioned firmwarewise before. This is not a showstopper and wil be addressed later. 

HED

Optical lasers

​VAC

XRO

XPD

DET

• Offline calibration update has been performed in SCS. Tests are foreseen for today.

EEE

• A new karabo device updating the digitizers with the beam parameters in real time has been successfully tested last week.

CAS

• 2.6.0. being prepared.

ITDM

SRP

TS

AoB

The dispatch/tech. integration meetings will be re organized in the near future. Slides are attached.

General

• Still recovering from the power loss occured on Saturday (The primary line was affected). In the hall the power was down for about 30 minutes, while in the tunnels it seems it was only a glitch.
• It is clear that it was a major system failure. Optimization of procedures and responsibility will be discussed within the DOD.

FXE

• Issues with the acquisition of images fromt he Jungfrau detector with the last version of Karabo. DET to follow it up offline.
• There are concerns about the replacement of the simple motor with the MC2 library, in particular about whether all functionalities will still be available (e.g. step-up/down). According to EEE this should be okay. CAS has already release the new library for testing. ==> will be done in thw WS for FXE, later for th tunnels and the other instruments.
• Remote control of the radiation shutter dows not work. IP of the new computer is needed. Will be solved offline between FXE, CAS and ITDM.

SPB/SFX

• Hexapods are being integrated. ==> This was rescheduled due to the issues with hexapods at HED and MID. Currently, there is no dire need to integrate the SPB/SFX hexapod, it can be run from TwinCAT. ==> postponed.

SCS

• Support from EEE is needed to make sure motors needed for the user experiments in 2 weeks are actually working (e.g. Elmo).
• If issues with the pulse IDs occur, AS, PG and BF are to be contacted (part of the trigger board)

SQS

​MID

• DPS turbo pump issue. A software workaround exists. The problem is that the used controller has not been commissioned firmwarewise before. This is not a showstopper and wil be addressed later. ==> The cabling is done differently. EEE is testing the device + firmware. ==> Ongoing
• Vacuum interlock worked fine following the power failure.
• Recovering from the power failure. Definition of check list in progress.

HED

• Recovering of Loop 9 is ongoing (motors fpor Mono1 and 2). EEE is supporting.
• Jungfrau has been fully recovered. Epix in progress.

Optical lasers

VAC

XRO

• There was an issue with a drift of the HED mirror. As a consequence, VAC and XRO have switched of the mirror cooling. HED and MID should report about the change of the situation. ==> still exchange of information needed.

XPD

​DET

• In general, it seems that no major issues occured after the power loss. 
• In FXE the power supply and the chiller for the LPD are connected on different power lines and this lead to the fact that, after the power came back, the power supply was back on while the chiller was off. This should be avoided anc hecked by the instruments.

EEE

CAS

• 2.6.0. being prepared. ==> Sneak peak was deployed in SPB/SFX. Actual deployment datas can be found in the Shutdown planing, will be discussed in the chapters and with the SASE responsibles. ==> tests with ITDM being planned.

ITDM

SRP

TS

AoB

XO

• A review of the procedures in case of power failure is in progress. Thanks to everybody for the efforts during the last days.

​FXE

• Remote control of the radiation shutter dows not work. IP of the new computer is needed. Will be solved offline between FXE, CAS and ITDM. ==> Done for FXE. Small issues being fixed for MID.

SPB/SFX

SCS

• Support from EEE is needed to make sure motors needed for the user experiments in 2 weeks are actually working (e.g. Elmo). ==> fine tuning of the support plan will be made between SCS and EEE.

​SQS

​MID

• DPS turbo pump issue. A software workaround exists. The problem is that the used controller has not been commissioned firmwarewise before. This is not a showstopper and wil be addressed later. ==> The cabling is done differently. EEE is testing the device + firmware. ==> Ongoing

HED

• The smaract stages with the current firmware can be either driven fast or precise. EEE will discuss with the isntruments involved how to modify the stages implementation to be able to have both options available.

Optical lasers

VAC

XRO

• There was an issue with a drift of the HED mirror. As a consequence, VAC and XRO have switched of the mirror cooling. HED and MID should report about the change of the situation. ==> still exchange of information needed. ==> Measurements from MID have been received. HED will do the same. Investigation of the problem will happen next week (shutdown)

XPD

​DET

• Calibration database haes been migrated. If problems are experienced please send a ticket to DET.

EEE

• Loop updates in the tunnels and instruments being scheduled together with the involved groups.

CAS

• 2.6.1. being prepared.  Deployment plan is still valid.

ITDM

SRP

TS

AoB

XO

• Today a test of the fire alarm system will take place. Mail has been circulated by SRP.
• From next monday the dispatch meeting time slot will change to 8.30. Chair will be with XO.

​FXE

• Leak in the LPD cooling system has been found. DET on it.
• Laser went out of synch several time during the last weeks. Being addressed with CAS.
• Mono middle layer is not working properly. Calibration parameters are not reloaded correctly after restart. CAS on it.

SPB/SFX

• CSLITS - discrepancies between the behavior of normal motors vs MC2 motors. CAS on it.

SCS

SQS

​MID

• During last WE big issues with the AGIPD were experienced. Probably calibration pipeline related. DET on it.

HED

• Hirex 2 commissionig ongoing.
• Mono commissioning - cameras were found not to work. CAS/EEE to check.
• Access to the experiment hutches will be restricted this week due to laser shots tests.

Optical lasers

VAC

XRO

• SASE1 FEL imager was damaged following the last power cuts and EPS/MPS issues. Will be replaced by Thursday.

XPD

​DET

​EEE

​CAS

• 2.6.1. ready to be deployed. Deployment schedule has been agreed with individual groups.

ITDM

SRP

TS

AoB

XO

FXE

• Leak in the LPD cooling system has been found. DET on it. Ongoing
• Laser went out of synch several time during the last weeks. Being addressed with the synch. group.
• Mono middle layer is not working properly. Calibration parameters are not reloaded correctly after restart. CAS on it. Ongoing.
• SA1 MOV loop has been updated. Minor issues with the CRL encoders are being addressed.

SPB/SFX

• CSLITS - discrepancies between the behavior of normal motors vs MC2 motors. CAS on it. ==> will be fixed once Karabo 2.6.1. has been deployed.

SCS

SQS

​MID

• During last WE big issues with the AGIPD were experienced. Probably calibration pipeline related. DET on it.
• Laser hutch A.07 is not supposed to be accessible in the next weeks because of the laser operation. SRP points out that they are not sure whether the interlocks are properly working and active. To be clarified offline between MID, HED and SRP.

HED

• Access to the experiment hutch, Optics and Rackroom will be restricted this week due to laser shots tests.

Optical lasers

VAC

XRO

• Change of middle layer devices for the attenuator in SASE2 will be discussed offline with CAS

XPD

​DET

​EEE

• Loops are being updated in coordination with the instruments.

​CAS

• 2.6.1. ready to be deployed. Deployment schedule has been agreed with individual groups.

ITDM

• Maxwell cluster maintenance on 2-3. 09.
• Control PCs wil be deployed during the shutdown week
• Claoud service (DESY platform) will be put into operation. An official announcement will details will follow.

SRP

TS

AoB

​FXE

SPB/SFX

​SCS

SQS

​MID

HED

Pptical lasers

VAC

XRO

​XPD

​DET

​EEE

​Controls

• Many IT/controls services will be down in KW48 (shutdown)

​ITDM

• Server 13 for SCS will be down today.
• General remark from Controls and ITDM: devices and functionalities (e.g. new devices in the DAQ, data aggregators…) that have not been tested during the normal working ours should not be used during shifts in the weekend or during the night. This creates a huge and evitable overhead for the people in OCD.

SRP

TS

AoB

​FXE

• Radiation tests ongoing.
• EEE informs that the PLC drawings will be exchanged in the WS

SPB/SFX

​SCS

SQS

​MID

HED

• During the first week of shutdown (KW47) the loop1 devices will be put into closed loop (e.g. for backlash compensation).
• An electrical shutdown seems to be necessary to upgrade the protection against the power glitches, especially to protect the optical lasers. This is now planned for the 17/18.19 for SASE2 and similar dates for SASE1 and 3. AV will try to collect more information and come back to the instruments ans technical groups.

Optical lasers

VAC

XRO

​XPD

​DET

​EEE

• EEE needs the updated interlock definitions from VAC
• The list of requested services during the WS is very long. NJ is trying to plan the works inside EEE and will come back to XO/PMO in case priorities have to be assigned. Also, late requests and changes are making things more difficult.

​Controls

• The overnight and weekend OCD service will stop during the shutdown. OCD only available until 18:00
• Karabo 2.7.0 deployment date (now forseen for KW51) is being verified. More info to come.

​ITDM

SRP

TS

AoB

​FXE

• Wille was asked by D3 to pull new cables for the Pandora in FXE on 19.11.. This was not known by FXE. Will be discussed with Safety/DNölle.
• Mono was fixed and is now ready for commissioning with beam

​SPB/SFX

• uTCA update planned for tomorrow#
• Chris Blöm will leave the company next week. His tasks need to be re assigned.
• Order of loops upgrade will be discussed with M.Stupar as soon as he is back.
• The drag chain will be opened for cables re organization. HV cables will be also replaced by Wille . SPB/SFX will get the new HV cables from Sonay Sayar.

​SCS

SQS

​MID

• Loop11 (AGIPD) needed. The task was not envisaged in the shutdown task list and is required by the detector group. Will be discussed with MS.
• It has to be checked whether the delivery of the SDL is confirmed and the patch panels infrastructure is ready for cables installation.

HED

Optical lasers

VAC

XRO

​XPD

​DET

​EEE

• Power supplies - firmware update will be performed in the next days. Mail will be sent around.
• Tming system update. No triggers will be available. Will be announced by mail.
• uTCA CPUs migration. The CPUs stored valued will be lost. Karabo devices will need to be re installed. Karabo stored values will not be lost. Will be announced by mail.

​Controls

• The overnight and weekend OCD service will stop during the shutdown. OCD only available in the time 6:00 - 18:00
• Karabo 2.7.0 deployment date for the tunnels is the 13.12.- Instruments to follow.

​ITDM

SRP

TS

• The power cut works will start at 6:00 in the morning. The electronics should ne put in a safe state the day before. 
• TS can provide support for temporary light and powerr during the power cut periods.

AoB

FXE

• No major issues - PLCs update has been started by EEE (BB)

SCS

• Hrixs and related components will be delivered on 13 and 14.01.2020. SCS is coordinating the works with TS in order to crane the chambers into the hall (collision with the scaffolding required for HED dipole laser)
• Puls picker update in XTD10. Requires update of loops. EEE is not informed. JTD follows the issue up.

EEE

• General work is ongoing.
• Next week the EPS loops will be updated in the framework of the full beam operation project.
• Issue with a PLC in SPB crashing after update to Win10. Beckhoff is working to a solution. If this does no work the PLC will be replaced.

Controls

• A mail will be sent out early next week to announce th deployment schedule of the new Karabo release.

FXE:

• No major issues.
• Karabo will be updated to the version 2.7.0. on Friday.
• uTCA update will be done on Thursday.

SCS:

• it was forgotten to involve EEE in the PLC update of the train picker in the tunnel. Now the issue is solved.

MID:

• SDL cabling ongoing.
• SDL preliminary tests ongoing
• Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 

 HED:

• IC1: cheking of the encoder is going on with the help of EEE (Hazem Ali)
• New interlock definitions for Loop 6 are giong to be delivered with some delay. NJ is informed.
• Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 
• XPD installed the diamond screens in the pop in monitors in the tunnel.

EEE:

• uTCA puippet migration  will start today in the optical laser systems. Next weeks it is planned to do it in the tunnel (XPD is informed). Info mail will follow.
• 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. Info mail will follow.
• SCS: new set of cables (microscope and others) will be ready for installation tomorrow.

Controls:

• 13.12. Karabo 2.7.0 deployment will begin.
• Normal motors will be upgraded to MC2. First tests are foreseen in MID tunnels. XPD and XRO are affected. XPD has been informed in this dispatch meeting. KS will inform MV (XRO). MS (EEE).

FXE:

• New optical table will be installed today. Access to the hutch could be limited.
• The new Airbox installed by D3 is colliding with the sliding door. It might be problematic to bring the new laser table in. SRP informed. It is being followed up. 
• uTCA update will be done on Thursday.

SCS

• The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP. It is not so clear who are the main players so far.

MID:

• SDL will be completed at the beginning of next week (inc. cables measurement).
• SDL tests ongoing
• Hexapod controllers will be moved to the rackroom. KS is waiting for the controller allocation in the racks in relation to the cables length from MFobian. 

 EEE:

• uTCA puppet migration: in MID it has been done yesterday. Today SQS and SCS will undergo the update. Tomorrow HED and FXE. Next week tunnels.
• 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. ITDM has to come back to BF to define a date.
• Update of loops in the tunnels are scheduled for next week

FXE:

• uTCA update has been completed by EEE on Friday evening. Information will be passed to controls for the finalization.

SCS

• The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP.
• MT will talk to VAC to clarify how to go on with the train picker integration in the tunnel.

SQS

• uTCA has been updated.
• A channel in a digitizer is bisbehaving. The issue is being fixed.
• Steinmeyer stages will be commissioned today with MS.

MID:

• SDL tests ongoing
• Hexapod controllers will be moved to the rackroom. Cables have to be pulled and only after the position of the controllers can be chosen.
• Loop12 will be updated today by EEE

 EEE:

• uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
• SASE1 and 3 EPS loop will be updated probably today.
• PLC in SPB with issues is being worked on.
• Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

 XPD:

• Some issue with the font size in the new Karabo GUI. Controls will have a look.

 Controls:

• Karabo updates ongoing. SASE3 today until 13:00. SA1, 2 on Wed.
• MC2 motors updates has been started also in FXE. 
• SASE3: meeting with Lasermet for the integration of interlock signals in Karabo. Controls has handed over the scenes to the instruments that will optimize them.
• firmware of the power supply for the AGIPD has to be updated urgently (by Wed.). CY sent a mail to AM. If no reply the power supplies will be switched off for the update.

MID:

• SDL tests ongoing. Important motors have been tested. Alignment of the mechanics with the alignment laser ongoing. Will be pumped over Christmas.

 EEE:

• uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
• SASE2 EPS loop will be updated probably today.
• SPB/SFX is getting a new CPU.
• Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

ITDM

• Meeting this week ITDM/EEE/TS to discuss how to mitigate issues caused by accidents (e.g. cooling water interruption caused by power cut). 

Controls:

• CY sent a mail several weeks ago to XO and the DH. The topic is emergency procedures during Christmas. No reply. CY announces that in the period 23.12-07.01 the Controls OCDs would not be available.

AoB

• No dispatch meeting on 06.01. Next meeting will be the Technical Integration one on 08.01.

SPB:

• Coordinatd motion for MC2 motors is going to be rolled out by EEE soon.

FXE:

• No issues to report.

MID:

• SDL tests ongoing. 
• All SDL cables have been installed and tested apart from 6 cables (still waiting for pins to be delivered). 
• Various loops update and hexapod integration ongoing.
• CRLs - EPS loop will be update probably this week. (involves position sensors in the y direction for CRLs in SASE2 and 1. To be done in the framework of the full beam operation project)

 EEE:

• SA1 VAC loop update. Issues with the eplan export. Discussion ongoing with VAC and XPD.
• SA2 MOV loop update for imagers MC2 motors. This week a meeting with all the stakeholder will be held (MID, HED, EEE, Controls) and a suitable date (during next week) for the update will be found. PMO will ask MDommach to provide some feedback on the Tunnel closing schedule. As soon as the feedback is available, it will be distrubuted through the readiness mailing list.