General

• Still recovering from the power loss occured on Saturday (The primary line was affected). In the hall the power was down for about 30 minutes, while in the tunnels it seems it was only a glitch.
• It is clear that it was a major system failure. Optimization of procedures and responsibility will be discussed within the DOD.

FXE

• Issues with the acquisition of images fromt he Jungfrau detector with the last version of Karabo. DET to follow it up offline.
• There are concerns about the replacement of the simple motor with the MC2 library, in particular about whether all functionalities will still be available (e.g. step-up/down). According to EEE this should be okay. CAS has already release the new library for testing. ==> will be done in thw WS for FXE, later for th tunnels and the other instruments.
• Remote control of the radiation shutter dows not work. IP of the new computer is needed. Will be solved offline between FXE, CAS and ITDM.

SPB/SFX

• Hexapods are being integrated. ==> This was rescheduled due to the issues with hexapods at HED and MID. Currently, there is no dire need to integrate the SPB/SFX hexapod, it can be run from TwinCAT. ==> postponed.

SCS

• Support from EEE is needed to make sure motors needed for the user experiments in 2 weeks are actually working (e.g. Elmo).
• If issues with the pulse IDs occur, AS, PG and BF are to be contacted (part of the trigger board)

SQS

​MID

• DPS turbo pump issue. A software workaround exists. The problem is that the used controller has not been commissioned firmwarewise before. This is not a showstopper and wil be addressed later. ==> The cabling is done differently. EEE is testing the device + firmware. ==> Ongoing
• Vacuum interlock worked fine following the power failure.
• Recovering from the power failure. Definition of check list in progress.

HED

• Recovering of Loop 9 is ongoing (motors fpor Mono1 and 2). EEE is supporting.
• Jungfrau has been fully recovered. Epix in progress.

Optical lasers

VAC

XRO

• There was an issue with a drift of the HED mirror. As a consequence, VAC and XRO have switched of the mirror cooling. HED and MID should report about the change of the situation. ==> still exchange of information needed.

XPD

​DET

• In general, it seems that no major issues occured after the power loss. 
• In FXE the power supply and the chiller for the LPD are connected on different power lines and this lead to the fact that, after the power came back, the power supply was back on while the chiller was off. This should be avoided anc hecked by the instruments.

EEE

CAS

• 2.6.0. being prepared. ==> Sneak peak was deployed in SPB/SFX. Actual deployment datas can be found in the Shutdown planing, will be discussed in the chapters and with the SASE responsibles. ==> tests with ITDM being planned.

ITDM

SRP

TS

AoB

XO

• A review of the procedures in case of power failure is in progress. Thanks to everybody for the efforts during the last days.

​FXE

• Remote control of the radiation shutter dows not work. IP of the new computer is needed. Will be solved offline between FXE, CAS and ITDM. ==> Done for FXE. Small issues being fixed for MID.

SPB/SFX

SCS

• Support from EEE is needed to make sure motors needed for the user experiments in 2 weeks are actually working (e.g. Elmo). ==> fine tuning of the support plan will be made between SCS and EEE.

​SQS

​MID

• DPS turbo pump issue. A software workaround exists. The problem is that the used controller has not been commissioned firmwarewise before. This is not a showstopper and wil be addressed later. ==> The cabling is done differently. EEE is testing the device + firmware. ==> Ongoing

HED

• The smaract stages with the current firmware can be either driven fast or precise. EEE will discuss with the isntruments involved how to modify the stages implementation to be able to have both options available.

Optical lasers

VAC

XRO

• There was an issue with a drift of the HED mirror. As a consequence, VAC and XRO have switched of the mirror cooling. HED and MID should report about the change of the situation. ==> still exchange of information needed. ==> Measurements from MID have been received. HED will do the same. Investigation of the problem will happen next week (shutdown)

XPD

​DET

• Calibration database haes been migrated. If problems are experienced please send a ticket to DET.

EEE

• Loop updates in the tunnels and instruments being scheduled together with the involved groups.

CAS

• 2.6.1. being prepared.  Deployment plan is still valid.

ITDM

SRP

TS

AoB

XO

• Today a test of the fire alarm system will take place. Mail has been circulated by SRP.
• From next monday the dispatch meeting time slot will change to 8.30. Chair will be with XO.

​FXE

• Leak in the LPD cooling system has been found. DET on it.
• Laser went out of synch several time during the last weeks. Being addressed with CAS.
• Mono middle layer is not working properly. Calibration parameters are not reloaded correctly after restart. CAS on it.

SPB/SFX

• CSLITS - discrepancies between the behavior of normal motors vs MC2 motors. CAS on it.

SCS

SQS

​MID

• During last WE big issues with the AGIPD were experienced. Probably calibration pipeline related. DET on it.

HED

• Hirex 2 commissionig ongoing.
• Mono commissioning - cameras were found not to work. CAS/EEE to check.
• Access to the experiment hutches will be restricted this week due to laser shots tests.

Optical lasers

VAC

XRO

• SASE1 FEL imager was damaged following the last power cuts and EPS/MPS issues. Will be replaced by Thursday.

XPD

​DET

​EEE

​CAS

• 2.6.1. ready to be deployed. Deployment schedule has been agreed with individual groups.

ITDM

SRP

TS

AoB

XO

FXE

• Leak in the LPD cooling system has been found. DET on it. Ongoing
• Laser went out of synch several time during the last weeks. Being addressed with the synch. group.
• Mono middle layer is not working properly. Calibration parameters are not reloaded correctly after restart. CAS on it. Ongoing.
• SA1 MOV loop has been updated. Minor issues with the CRL encoders are being addressed.

SPB/SFX

• CSLITS - discrepancies between the behavior of normal motors vs MC2 motors. CAS on it. ==> will be fixed once Karabo 2.6.1. has been deployed.

SCS

SQS

​MID

• During last WE big issues with the AGIPD were experienced. Probably calibration pipeline related. DET on it.
• Laser hutch A.07 is not supposed to be accessible in the next weeks because of the laser operation. SRP points out that they are not sure whether the interlocks are properly working and active. To be clarified offline between MID, HED and SRP.

HED

• Access to the experiment hutch, Optics and Rackroom will be restricted this week due to laser shots tests.

Optical lasers

VAC

XRO

• Change of middle layer devices for the attenuator in SASE2 will be discussed offline with CAS

XPD

​DET

​EEE

• Loops are being updated in coordination with the instruments.

​CAS

• 2.6.1. ready to be deployed. Deployment schedule has been agreed with individual groups.

ITDM

• Maxwell cluster maintenance on 2-3. 09.
• Control PCs wil be deployed during the shutdown week
• Claoud service (DESY platform) will be put into operation. An official announcement will details will follow.

SRP

TS

AoB

​FXE

SPB/SFX

​SCS

SQS

​MID

HED

Pptical lasers

VAC

XRO

​XPD

​DET

​EEE

​Controls

• Many IT/controls services will be down in KW48 (shutdown)

​ITDM

• Server 13 for SCS will be down today.
• General remark from Controls and ITDM: devices and functionalities (e.g. new devices in the DAQ, data aggregators…) that have not been tested during the normal working ours should not be used during shifts in the weekend or during the night. This creates a huge and evitable overhead for the people in OCD.

SRP

TS

AoB

​FXE

• Radiation tests ongoing.
• EEE informs that the PLC drawings will be exchanged in the WS

SPB/SFX

​SCS

SQS

​MID

HED

• During the first week of shutdown (KW47) the loop1 devices will be put into closed loop (e.g. for backlash compensation).
• An electrical shutdown seems to be necessary to upgrade the protection against the power glitches, especially to protect the optical lasers. This is now planned for the 17/18.19 for SASE2 and similar dates for SASE1 and 3. AV will try to collect more information and come back to the instruments ans technical groups.

Optical lasers

VAC

XRO

​XPD

​DET

​EEE

• EEE needs the updated interlock definitions from VAC
• The list of requested services during the WS is very long. NJ is trying to plan the works inside EEE and will come back to XO/PMO in case priorities have to be assigned. Also, late requests and changes are making things more difficult.

​Controls

• The overnight and weekend OCD service will stop during the shutdown. OCD only available until 18:00
• Karabo 2.7.0 deployment date (now forseen for KW51) is being verified. More info to come.

​ITDM

SRP

TS

AoB

​FXE

• Wille was asked by D3 to pull new cables for the Pandora in FXE on 19.11.. This was not known by FXE. Will be discussed with Safety/DNölle.
• Mono was fixed and is now ready for commissioning with beam

​SPB/SFX

• uTCA update planned for tomorrow#
• Chris Blöm will leave the company next week. His tasks need to be re assigned.
• Order of loops upgrade will be discussed with M.Stupar as soon as he is back.
• The drag chain will be opened for cables re organization. HV cables will be also replaced by Wille . SPB/SFX will get the new HV cables from Sonay Sayar.

​SCS

SQS

​MID

• Loop11 (AGIPD) needed. The task was not envisaged in the shutdown task list and is required by the detector group. Will be discussed with MS.
• It has to be checked whether the delivery of the SDL is confirmed and the patch panels infrastructure is ready for cables installation.

HED

Optical lasers

VAC

XRO

​XPD

​DET

​EEE

• Power supplies - firmware update will be performed in the next days. Mail will be sent around.
• Tming system update. No triggers will be available. Will be announced by mail.
• uTCA CPUs migration. The CPUs stored valued will be lost. Karabo devices will need to be re installed. Karabo stored values will not be lost. Will be announced by mail.

​Controls

• The overnight and weekend OCD service will stop during the shutdown. OCD only available in the time 6:00 - 18:00
• Karabo 2.7.0 deployment date for the tunnels is the 13.12.- Instruments to follow.

​ITDM

SRP

TS

• The power cut works will start at 6:00 in the morning. The electronics should ne put in a safe state the day before. 
• TS can provide support for temporary light and powerr during the power cut periods.

AoB

FXE: no issues

SPB: no issues

MID: Split & Delay line arriving on Friday, MID aims for a quick integration into instrument / beamline
         EEE informed to switch Hexapod motors from loop3 to loop 7

HED: changing loop1 (JJ X-ray motors) from open to closed loop operation

SQS: KB works, see sometimes spontaneous switching of end switch logics

EEE: MID, SPB, VAC loop updates running this week;
         no triggers for experiments available until end of power off work
         Jörn: cabling for MID is in full swing, for future installations first check phase 1 cabling, for non blocking
                                   HED: cabling
                                   SCS company Witte 30 ccables
                                   FXE: two CRDs not approved yet, no cabling during this shutdown

Controls:
 

FXE

• No major issues - PLCs update has been started by EEE (BB)

SCS

• Hrixs and related components will be delivered on 13 and 14.01.2020. SCS is coordinating the works with TS in order to crane the chambers into the hall (collision with the scaffolding required for HED dipole laser)
• Puls picker update in XTD10. Requires update of loops. EEE is not informed. JTD follows the issue up.

EEE

• General work is ongoing.
• Next week the EPS loops will be updated in the framework of the full beam operation project.
• Issue with a PLC in SPB crashing after update to Win10. Beckhoff is working to a solution. If this does no work the PLC will be replaced.

Controls

• A mail will be sent out early next week to announce th deployment schedule of the new Karabo release.

FXE:

• No major issues.
• Karabo will be updated to the version 2.7.0. on Friday.
• uTCA update will be done on Thursday.

SCS:

• it was forgotten to involve EEE in the PLC update of the train picker in the tunnel. Now the issue is solved.

MID:

• SDL cabling ongoing.
• SDL preliminary tests ongoing
• Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 

 HED:

• IC1: cheking of the encoder is going on with the help of EEE (Hazem Ali)
• New interlock definitions for Loop 6 are giong to be delivered with some delay. NJ is informed.
• Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 
• XPD installed the diamond screens in the pop in monitors in the tunnel.

EEE:

• uTCA puippet migration  will start today in the optical laser systems. Next weeks it is planned to do it in the tunnel (XPD is informed). Info mail will follow.
• 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. Info mail will follow.
• SCS: new set of cables (microscope and others) will be ready for installation tomorrow.

Controls:

• 13.12. Karabo 2.7.0 deployment will begin.
• Normal motors will be upgraded to MC2. First tests are foreseen in MID tunnels. XPD and XRO are affected. XPD has been informed in this dispatch meeting. KS will inform MV (XRO). MS (EEE).

FXE:

• New optical table will be installed today. Access to the hutch could be limited.
• The new Airbox installed by D3 is colliding with the sliding door. It might be problematic to bring the new laser table in. SRP informed. It is being followed up. 
• uTCA update will be done on Thursday.

SCS

• The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP. It is not so clear who are the main players so far.

MID:

• SDL will be completed at the beginning of next week (inc. cables measurement).
• SDL tests ongoing
• Hexapod controllers will be moved to the rackroom. KS is waiting for the controller allocation in the racks in relation to the cables length from MFobian. 

 EEE:

• uTCA puppet migration: in MID it has been done yesterday. Today SQS and SCS will undergo the update. Tomorrow HED and FXE. Next week tunnels.
• 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. ITDM has to come back to BF to define a date.
• Update of loops in the tunnels are scheduled for next week

FXE:

• uTCA update has been completed by EEE on Friday evening. Information will be passed to controls for the finalization.

SCS

• The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP.
• MT will talk to VAC to clarify how to go on with the train picker integration in the tunnel.

SQS

• uTCA has been updated.
• A channel in a digitizer is bisbehaving. The issue is being fixed.
• Steinmeyer stages will be commissioned today with MS.

MID:

• SDL tests ongoing
• Hexapod controllers will be moved to the rackroom. Cables have to be pulled and only after the position of the controllers can be chosen.
• Loop12 will be updated today by EEE

 EEE:

• uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
• SASE1 and 3 EPS loop will be updated probably today.
• PLC in SPB with issues is being worked on.
• Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

 XPD:

• Some issue with the font size in the new Karabo GUI. Controls will have a look.

 Controls:

• Karabo updates ongoing. SASE3 today until 13:00. SA1, 2 on Wed.
• MC2 motors updates has been started also in FXE. 
• SASE3: meeting with Lasermet for the integration of interlock signals in Karabo. Controls has handed over the scenes to the instruments that will optimize them.
• firmware of the power supply for the AGIPD has to be updated urgently (by Wed.). CY sent a mail to AM. If no reply the power supplies will be switched off for the update.

MID:

• SDL tests ongoing. Important motors have been tested. Alignment of the mechanics with the alignment laser ongoing. Will be pumped over Christmas.

 EEE:

• uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
• SASE2 EPS loop will be updated probably today.
• SPB/SFX is getting a new CPU.
• Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

ITDM

• Meeting this week ITDM/EEE/TS to discuss how to mitigate issues caused by accidents (e.g. cooling water interruption caused by power cut). 

Controls:

• CY sent a mail several weeks ago to XO and the DH. The topic is emergency procedures during Christmas. No reply. CY announces that in the period 23.12-07.01 the Controls OCDs would not be available.

AoB

• No dispatch meeting on 06.01. Next meeting will be the Technical Integration one on 08.01.

SPB:

• Coordinatd motion for MC2 motors is going to be rolled out by EEE soon.

FXE:

• No issues to report.

MID:

• SDL tests ongoing. 
• All SDL cables have been installed and tested apart from 6 cables (still waiting for pins to be delivered). 
• Various loops update and hexapod integration ongoing.
• CRLs - EPS loop will be update probably this week. (involves position sensors in the y direction for CRLs in SASE2 and 1. To be done in the framework of the full beam operation project)

 EEE:

• SA1 VAC loop update. Issues with the eplan export. Discussion ongoing with VAC and XPD.
• SA2 MOV loop update for imagers MC2 motors. This week a meeting with all the stakeholder will be held (MID, HED, EEE, Controls) and a suitable date (during next week) for the update will be found. PMO will ask MDommach to provide some feedback on the Tunnel closing schedule. As soon as the feedback is available, it will be distrubuted through the readiness mailing list.

SPB:

• Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
• optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished

FXE:

• XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE

MID:

• SDL should enter the hutch by the end of January

SCS:

• pulse (train) picker to be installed in the tunnel by the end of this week

 EEE:

• FXE had been migrated to MC2 lib
  • motors need to be recalibrated
• XTD6 motion part is to be migrated to MC2 lib
• SQS gets interlock updates

CTRL:

• busy with tunnel, additional support for XPD (imagers motion components)

SPB:

• nothing to report

FXE:

• MC2 library motor commissioning on-going

MID:

• SDL motor parameters defined, however one crate is blocked by Wille and 6 motor connectors (Harting material) are still missing
  once fixed, motor paramaters can be uploaded
• WCB (wall connection boxes) in otics hutch are partially finished

HED:

• Wille is working on some crates and the respective Beckhoff loops are down for the moment
• loop updates from Tuesday on

 EEE:

• additional limit switched for XTD6 CRLs (MID) are being installed
• HED is asking for limit switches on their XTD6 CRLs as well --> EEE and HED will investigate when and how
• cable change in FXE starts this week with exchaging wrongly poled SHV cables

CTRL:

• nothing to report

VAC:

• nothing to report

XRO:

• nothing to report

SPB:	Nothing to report;

FXE: 	Everything ongoing; nothing special to be reported;

MID:    Finalize motors of split and delay line; MID has to test interlock for vacuum loop update.

HED:	Commissioning of laser heat table. It may need assistance from EEE and CTRL during the week.

SCS:	Nothing to report - Last week's mounted setup will be disassembled in order to mount the standard setup for users;

EEE: 	OCD was called from XRO during beamline alignment of SA2: 1 BMPI motor had a problem and is under investigation - EEE reminds on the new homing position for 
negative direction motion.

CTRL: 	Following reset of train-ID and timing system for today.

XPD: 	Shifted Beam Pointing Studies from A. Koch to Tuesday. 

DET: 	Nothing to report

*machine is currently not running due to filter clean in the cold compressor. Shall be re-started this afternoon and by tomorrow must be lasing.
*Master time reset and train-ID reset this morning. Then EEE (check systems), followed by CTRL (karabo servers re-start) and ITDM (check DAQ operation). LAS and DET are 
informed.
*Wednesday: Network stress test with Cameras by CTRL on Wednesday 12:00-14:30.

FXE:
Nothing to report: no issues;

MID:
Delay line pumping down; Connecting all devices in preparation to be ready for Friday;

EEE:
Finishing SQS interlock scenes and SPB loops;
ELMO parameters lost themselves (in connection to XRO/Maurizio request). Parameters to be checked this week and next shutdown;
XPD ticket to be checked;
Cabling shall be started for SQS/SCS: waiting for open hutch;

CTRL:
Ticket from XPD (shared EEE/CTRL): dry run did not work at BKR: missing pieces. 
Please instruments check the systems during the day prior Friday when the Test Run starts;
Request from CTRL/Chris: Define guidelines for the users/support on the ticketing and e-log (who has responsibilities to check devices?).

ITDM: 
Ticked from XPD/Liuba: it is a ticket with link (?) to E-log.


* Mc2 motors: XRO needs a list of all motors in order to check configurations. Example: SA3 MONO pre-mirrors with torque set to zero; SA3 exit 
slits with wrong settings.

* CTRL reminds the instruments to check devices during working hours prior Friday when the Test Run starts with beam delivery.

SPB:

• nothing to report

FXE:

• not present due to group retreat

MID:

• Mono done, no severe issues from commissioning weekend

SCS:

• not present

 EEE:

• SQS pnCCD cables have been installed, waiting for crate to connect

CTRL:

• two minor OCD call and one regarding AGIPD
  • SPB not aware that no support was scheduled for AGIPD

 XRO:

• piezo karabo device and firmware in HED mono need to be checked for compatibility

ITDM:

• 2 calls
  • derver down and restarted
  • data aggregator down and restarted
• new RunConfig delpoyed at FXE

DA:

• jupyterhub @SCS showing performance issues --> investigated by DA

SPB:

• AGIPD improvement campaign great success
• EEE --> 2nd C&C worked just fine

FXE:

• BTM tests successful
• Johann MDL motors should be in closed loop
• JF CTRL device crashed

MID:

• AGIPD Flat Fields done
• Data Correction pipeline was down --> 1 day lost
• PPLaser comm this week
• extrafoam issues

SCS:

• nothing to report

 EEE:

• updates for SDL (MID)
• fastADC issue --> don't take more than 40k samples; bugfix in progress

CTRL:

• 8 OCD events
  • mostly deployment isues
  • FXE DA3 overload
  • old version of scantool @ SCS

 XPD:

• unusual behaviour of beam pos measured with XGM
  • cross check done yesterday

ITDM:

• DA crashes --> being investigated

HED:

• tecnosoft terminal issue
  • motor ran into Ty limit (CCW) and got stuck due to tecnosoft logic