Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 WP74 FXE SCS SPB MID HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 20 of 31  Not logged in ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
Entry   Notes XO Integration Meeting 10.05.2021, posted by Frederik Wolff-Fabris on 10 May 2021, 10:05 
General Announcement:
- Two smoke sensors affected in SA3 triggering fire alarm: The first one could be reset but for the second one the reset did not work and fire brigade was in. 
Beam operation was recovered only after the External company came to reset it. The downtime of about 4 hours for SA1/SA3 operation happened.

CTRL:
- Nothing to report.

DA:
- On behalf of calibration team: this morning a major upgrade will be done. No access to calibration services during this time.

EEE:
- One loop in FXE was not working during the weekend. Issues with movs/motor loops and the cause has to be investigated (old scripts were in use). No devices were 
damaged.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Deployment for HED calibration taking place shall fix last week's issue.
- A fix for DAQ on AGIPD to be done.
- FXE issue with LPD calibration: still open. This is not related to the DA deployment today. Jolanta S-D. will follow it inside DET.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- Nothing to report.

UNSYS:
- Fire alarms in SA3 / cell#22-23 on Saturday evening: BKR informed that could be related to synchrotron radiation on the detector signal. Fire brigade was on-site but no 
fire was found.

SPB:
- Nothing to report.

FXE:
- Thanks to EEE help with Gotthard.
- M2 Mirror pitch issues seems to be back to normal as before. Antje T. (XRO) made ZZ to correct loose gear box.
- Motor loops issues found and addressed by EEE.

MID:
- Nothing to report.

HED:
- Motor stop working and ticket has been written.
- Preparing for experiment this week.

SCS:
- Zoom problems last week has been taken care; Thanks to ITDM.

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 05.05.2021, posted by Frederik Wolff-Fabris on 05 May 2021, 10:20 
General Announcement:
-	 

CTRL:
- The device clone deployment went smoothly yesterday.

DA:
- Nothing to report.

EEE:
- Call from DOC referring to motor mirrors in SA1: XRO is investigating together with FXE. The issue is that pitch accuracy can't be reproduced as before.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Issue with offline calibration mention last Monday: the problem has been identified and a software fix is been worked (for HED detector). The calibration team is 
working on it.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report besides the SA1 mirror motors.

LAS:
- Problem with Optics in SA3 PP-laser (either coating degradation or lack of clean) was found. The mirror will be rotated.
- DOC reported a synchronization issue in SA2. It is not a LAS issue, has to be fixed by DOC.

UNSYS:
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Chris Milne mentions the steering issues with M2 and M3 mirrors and XRO is looking into it. 

MID:
- Synchronization issue with Laser is fixed.
- DAQ issue also fixed during the night.

HED:
- Things reported on Monday have been looked up.
- Train-ID issue: there is a walk around for it and test/fix to be done next week.

SCS:
- SCS asks for a solution to share screens with users in the control room (general solution also applicable to other instruments?). 
K. Wrona says ITDM has solutions that were tested last week at FXE and was working properly. 
An offline discussion between ITDM and SCS will take place to set the solution.  

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 03.05.2021, posted by Frederik Wolff-Fabris on 03 May 2021, 10:06 
General Announcement:
- The deadline for submitting the summer maintenance tasks is ended. In case you have not done so, please submit it as soon as possible.
- Monica T. as DRC invites people to attend the DRC jour-fixe meeting.

CTRL:
- Bug fix for the device clone is prepared and will be deployed today. The clone devices will go down for a couple of minutes only.
- ADC 412 bug: there is a fix and instruments shall get in touch with their contacts for the deployment.

DA:
- One OCD call related to Karabo bridge and usage of online monitoring: the problem was not related to Karabo but with an environment conflict.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- All PP-lasers are up and running.

UNSYS:
- Waiting for the ZZ in SA1 and SA3.

SPB:
- User experiments last week went well again.

FXE:
- Nothing to report.

MID:
- All fine at MID.

HED:
- HED run last week went well - quite successful with tests with Laser.
- Missing Train-ID issue was reported to expert groups; All the devices are not correctly seen (arriving some trains later); 
Steffen Hauf says CTRL are working and will be ready to test. There is a chance for tests in the coming week or still this week. 
An offline discussion will take place between CTRL and HED.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 28.04.2021, posted by Frederik Wolff-Fabris on 28 Apr 2021, 10:04 
General Announcement:
- A. Galler points to the deadline for submitting summer maintenance task list on April 30th.
- XO would like to remind instruments that in case of operation failure related to Interlocks issue, please also include the PRCs (prc@xfel.eu) on the communications. 
The Photon Run Coordinators are the first contact point that can help in such situations.
- Tomasz J. asks on the reason for the water-cooling problems with SA1 PP-laser last Monday. A. Galler says the origin was a power glitch that caused the interruption of 
water systems. 
Raul V. says that it's also seen in some tunnel devices but because of the changes in interlock logics this worked nicely.
- Zunaira A. asks why many power glitches are happening. Andreas G. says there were this year 3 events with different causes. 
Last Monday's one was a real power glitch from distribution line; the one on Easter Sunday is under investigation via an 8D report and the one last week was caused by 
human error.

CTRL:
- A bug was found in the devices for HED (clown system). A test in HED will be done prior a full deployment.

DA:
- Not present.

EEE:
- Status of ADQs in SQS: Bruno F. says that it worked yesterday and suspicion is that data were saved locally. DOC and CTRL are following with this issue.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- All fine for Detectors.

ITDM: 
- New feature for data migration is set – main ADC.

XRO:
- Nothing to report.

LAS:
- All lasers are UP and running.

UNSYS:
- Issue with phase shifter in SA2/cell26 this week was solved (Doocs problem).
- Waiting for the ZZ access in SA1/SA3. A. Galler says that when time window opens this will be informed.

SPB:
- Experiment is going smoothly this week.

FXE:
- Last week HED had a few issues with brig brother server when scanning the undulators. The Big Brother was setting bunches back to zero. Issue will be followed up with 
machine.

MID:
- All fine at MID.

HED:
- Nothing to add; thanks to CTRL for looking after the scan tool.
- Test with Relax synchronization will be done tonight.

SCS:
- Started using the PES and recorded runs - about 35% of missing train-IDs. A. Galler says that one point of contact is via DOC. 
Steffen Hauf askes how many boards were using and Laurent M. informs 4 that run in the SCS topic.

SQS:
- Apart of the shift team in quarantine, everything seems to be going fine.
Entry   Notes XO Integration Meeting 26.04.2021, posted by Frederik Wolff-Fabris on 26 Apr 2021, 11:08 
General Announcement:
- The deadline to submit the maintenance task list is on 30.04. We thank all the groups that have already submitted. In case of any question and comments, please contact 
us at xo-planning@xfel.eu
- Patrick Gessler reminds on the DRC meeting on Mondays 13:00. This meeting is done together with instruments to exchange information for the weekly plan. The e-mail for 
contact is drc-jour-fixe@xfel.eu 
- Andreas G. says that with the new 7-11 operation mode, XO is interested in knowing how longtime instruments need to be ready to take the beam in the early morning.
Please inform us for that XO checks the efficiency of this model.

CTRL:
- Remind on Beckhoff training next Wednesday (following e-mail to readiness).
- Upgrades of two Karaboo Beckhoff servers today (Karabo devices go down for 5-10 minutes, proposed to be done today during lunch time): SA2 master tunnel loop; and the 
HED loop.
- Bug on ADQ for twelve devices, affecting SQS and others: EEE has discovered it and CTRL is analyzing to come up with a fix.

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- PES in SA3 was provided and operated last week for instrument's usage.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Maurizio V. requests the SA2 experiments to provide information to XRO on issues related with mirror alignments or transport beamline.

LAS:
- Received info from Desy on a power glitch today at around 6AM. At the same time SA1 Amphos switched OFF due to not enough water. 
It’s under investigation with TS. Jan G. points that also there was a beam loss in delivery around the same time.
- SA2 and SA3 PP-lasers are still in operation.

UNSYS:
- Issue with undulator SA1 /cell33 and cell1/SA3 needs ZZ. Currently the impact in operation is minimal and ZZ is postponed to the next available date.

SPB:
- Nothing to report.

FXE:
- Run with beam delivery went smooth.

MID:
- Quite successful beam week with small hick-ups to be addressed with experts.

HED:
- Commissioning beamtime this week; Data analysis will be made this week.
- Test of the PLC synchronization with Relax laser to be done this week.

SCS:
- Not present.

SQS:
- uTCA in ADQ had been mentioned already by CTRL. It has been tackled.
Entry   Notes XO Integration Meeting 21.04.2021, posted by Frederik Wolff-Fabris on 21 Apr 2021, 10:50 
General Announcement:
-  XO reminds on the deadline of 30.04 for submitting the Summer maintenance task lists. For this, please do so sending the list to xo-planning@xfel.eu. 

CTRL:
- Update on undulator Doocs side: CTRL needs to test for two or multi-colors schemes and asks instruments to CTRL if there are plans to use it.
- CTRL will schedule a short practical training on Beckhoff devices for next Wednesday. An announcement will be made through readiness mailing list.

DA:
- Nothing to report.

EEE:
- From the PLC side, EEE is following up the MID/CRLs interlock case.
- If groups submit a change request needed to be implemented during summer maintenance time, please contact and inform EEE beforehand.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- The network cable exchange mentioned last Monday was done.

XRO:
- Not present.

LAS:
- All 3 PP-lasers are in operation.

UNSYS:
- After power glitch on Monday in SA1, about half the undulator cells were half power cycled. Most could be recovered expect SA1/cell33 and SA3/cell1. This will require 
ZZ accesses. 
XO states the power failure was man-made caused.
- Quad mover sudden move: Lars F. from MXL does not have records on the command given for the sudden move and therefore the root cause can't be traced.

SPB:
- Nothing to report.

FXE:
- Beam since yesterday; one interlock issues. 
Nerea J. says a ticket was received and the interlock issue was related to a Imager that always come up activated. This will be removed from the EPS.

MID:
- All fine at MID.

HED:
- Nothing to report.

SCS:
- Undulator photon energy couldn't be changed via Karabo. Jan-Torben D. will talk to Steffen H; David Hickin adds that the location was changed and the Karabo side is 
done/corrected. 
One needs the BKR connection/authorization and this will be organized by CTRL/Steffen H.

SQS:
- Issue with digitizer on work. It is needed for next week.
Entry   Notes XO Integration Meeting 19.04.2021, posted by Frederik Wolff-Fabris on 19 Apr 2021, 11:39 
General Announcement:
-  Remind on the Summer maintenance period: deadline to submit task list is April 30th. XO invite all groups to submit by this date.

CTRL:
- Remind on the update of some specific Karabo installations to fix bugs (Data logger installations) to start today (e-mail was sent to the readiness).
- Bunch pattern statistics devices need to be fixed today at the same time as the data loggers (relates to lose of time stamps in lasers).
- Upgrade on instruments are coordinated by ICIs; Updates of Mc2 motors today.
- Z. Ansari asks if updates of MID will affect interlocks as well; Valerii B. answers this is a Karabo update and not PLCs, so no effect is expected.
- On recent days CTRL finds Karabo broker pileup and this was successfully understood; DOC was preventively monitoring it and fixed it beforehand.

DA:
- No OCD calls for DA; 
- Collaboration and preparation for supporting SPB/SFX last week and for beamtime continuation in two weeks.

EEE:
- Today 08:30 it was found SA1 tunnel racks lost power. The uTCAs were also down but not the one in room 1.14. It affected SA1 tunnels loops such as Vacuum and motion.
The guess is on a power glitch. XO will bring up to the machine side and eventually correlated with the Easter Sunday's event.

VAC:
- Recovering the SA1 tunnel loops from 08:30 downtime. All seems to be fine right now.
- VAC asks if FXE is running cryo coolers. FXE tells they were running and will check the signals for possible issues.

XPD:
- HIREX motors in SA1 were OFF this morning.

DET:
- One OCD call was received and solved; DET is not aware of issues with Detectors. 

ITDM: 
- On the Wednesday problem with network (in SA1, the online calibration was down): Today a cable will be exchanged. 
- ITDM was involved with experiments together with DA on offline calibrations.
- As DRC, K. Wrona reminds today that 1PM there is the DRC meeting and the participation of instruments is strongly encouraged.

XRO:
- Nothing to report.

LAS:
- SA1 PP-laser is fine.
- SA2 PP-lase with small issue with regulations. Tickets sent to CTRL.
- SA3 PP-laser with several small problems: damage to optics (need to exchange one AMPHOS stage); interlock tripped due to door problems and LAS went OFF. 
LAS hopes to bring back laser and make available for instruments by tomorrow.

UNSYS:
- SA1/cell12 Quad mover sudden move last week: it was not related to a software issue and is under further investigation on Doocs side. A Doocs command was sent.

SPB:
- Experiment last week: quite successful and not many issues.

FXE:
- Nothing to add.

MID:
- All fine.

HED:
- A Motor on the CRL HED chamber was engaged and not moving (hardware issue). It may need a ZZ. It is not related to the CRL's arms.
- SEPS interlock tripped machine on Friday almost midnight (colleagues close shutter before moving CRLs). The beam permission was restored after lens removed from beam.
- ADC is offline. HED will contact Bruno F. offline.

SCS:
- Not present.

SQS:
- Commissioning for preparing next experiment.
- Issue with digitizers that can't be used: Bruno F. adds that action is scheduled for 14:00 as discussed with SQS's members.
Entry   Notes XO Integration Meeting 14.04.2021, posted by Frederik Wolff-Fabris on 14 Apr 2021, 10:50 
General Announcement:
- We would like to remind groups that the deadline for submitting to XO the feedback on the planned activities for the Summer/21 maintenance time is on April 30rd. 
In case of questions please contact XO via xo-planning@xfel.eu

CTRL:
- Updates announced on last XO Int. meeting will take place next Monday (unless if a group protests). This means a short downtime on data loggers in tunnels. An e-mail 
with a reminder will be sent.
- Upgrade for the bunch pattern statistics devices to take place (also will be offline for few minutes).
- Issues with MC2 motors: the workaround is to shutdown device and click/instantiate device. A fix will be deployed.

DA:
- DA involved and continuing with "Moonshot" coordination meeting.

EEE:
- Nothing to add.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- Announcement with respect to Maxwell cluster: ITDM will change partition schemes in the next Summer maintenance. 
Three new partitions will be introduced, for calibrations and a special one for reservations for beamtime (UPEX related); 
A new feature is that if Maxwell is full of jobs, one can pre-empty it by killing jobs and putting back in the queue.

XRO:
- Yesterday beam going to HED: some problems were found regarding parameter configurations. 
XRO reminds SA2 instruments to be careful and check devices positions and parameters are they may be changed in relation to April 1st due to the issue with PLCs during 
Easter Sunday. 
- Zunaira A. says there were problems with CRLs arms configurations. Some devices had to be restored but worked normally.

LAS:
- All lasers are fine. 
- Shutters in SA1 PP-Laser were in maintenance mode - without defined reason.
- SA2 PP-laser is fine. 
- SA3 PP-laser has degradation due to the cleaning. XO will address the issue with LAS.

UNSYS:
- All fine.

SPB:
- Running user experiment. Low photon energy yesterday (0.8mJ) and drop intensity over the full 352 bunch train but today it got better.

FXE:
- Nothing to add.

MID:
- All fine.

HED:
- All motors lost positions, e.g. mono, screens being driven (some in and not supposed to be in). CRL1 issue the position was lost at the axis; motors not being homed; 
Zuzana K. will summarize main issues and forward to XO.
- Steffen H asks for a summary of encoders and Zunaira A. also asks on the use of interlocks and CRLs/shutter to evaluate if all is fine. Zuzana K. says HED did not work 
yet on the CRL2 but will communicate as soon as experience is get.

SCS:
- Question to CTRL on the workaround of MC2 motors. Steffen H. says to shutdown device, check configuration and instantiate it again. An instruction e-mail will be sent 
to readiness list.

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 12.04.2021, posted by Frederik Wolff-Fabris on 12 Apr 2021, 14:40 
General Announcement:
-	 

CTRL:
- Broker pile-up on Friday: Alessandro S. says that was triggered by network configurations; it took time to recover situation. 
Started SA3 topics for deployment and for few tickets related to that, please get in touch with DRC or try re-starting devices.
- Deployments: 1. Fix for data loggers and timing servers. We propose to do next Monday; 2. Karabo 2.10.4 was released with few bug fixes. CTRL is in touch with 
instruments.

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Found on Saturday a problem with SA3 gas attenuator. It was solved with the DOC.

XPD:
- Power increase tests ongoing this morning.
- A. Koch says two servers were not running this morning and will make contact to DRC.

DET:
- Couple of OCD calls received and fixed.

ITDM: 
- Network failure: systems are very sensitive to network failures (switch failure). Relates to replacement of devices.

XRO:
- Some SA3 cameras lost configurations (maybe related to the SA3 event during the weekend): fixed by M. Vannoni. 
- SCS/SQS teams: structure for the M6 chamber has to be fixed (motors): XRO will communicate with the instruments, to keep the motors in proper conditions.

LAS:
- PP-lasers are running. Small minor issues: 
1. SA3 problem with displaying number of pulses. Solved by CTRL by installing latest Karabo server;
2. SA2 still shows small problem with regulations/settings, spectrum started drifting. Can be fixed by OCD but ticket was created.

UNSYS:
- Nothing to report.

SPB:
- Preparing for user experiment this week.

FXE:
- Nothing to report:
- Last Friday PP-laser could not config laser pattern. Tomasz J. says the OCD laser is aware and asks to please also inform DOC/DRC in such cases as LAS can't help in 
this time. 
Monica T. confirms that one has to call the DOC and invite instruments that do not get beam to also join the Monday 13:00 DOC meetings.

MID:
- Suffered with SA2 PLC downtime - mono lost positions.
- Scan energies on the undulators will be needed on next week.
- Last week operation was smooth.

HED:
- Starting operation this week.

SCS:
- Nothing to report.

SQS:
- Spectrum of detectors - two spectras were split. This was reported to BKR and DOC.
Entry   Notes XO Integration Meeting 07.04.2021, posted by Andreas Galler on 07 Apr 2021, 08:52 

General Announcement:

  • .

CTRL:

  • recurrent error reprted by LAS  --> drift compensation in SA2 --> need feedback
  • minor issues reported instruments during startup

DA:

  •  

EEE:

  • Issue on April 4th --> 4 PLCs shut down without a cause, (looks like power cut)
  • MID crate went down at 9:00
  • SA3 utca shut down (could be voltage drop, could be ehternet) (9:35)

VAC:

  • VAC-OCD recovered SA2 vacuum state (valves, and ystem) (see EEE SA2 incident above)
  • VAC obseved that the MID Mono warmed up, the cryo warmed up presumably after a PLCs update which interrupted an interloop communication

XPD:

  • nothing report

DET:

  • everything running fine

ITDM:

  • SA1 baclony room checked
    • some racks lost power --> investifation going on (possibly happened on Sat. Apr 3rd 8:00 am)

XRO:

  • incorrect motor velocity in SA3 --> fixed
  •  

LAS:

  • small issue in SA2 with slow feedback --> see CTRL report

UNSYS: 

  • problem on SA1 central control node, after a power cycle aircoils and phase shifters were unresponsive

SPB/SFX:

  • Laser Interlock issue was solved, power spike involved that reported by MEA3 ( klaus Witt).

FXE:

  • no issues

MID:

  • M1 mirror was moved completely out on 1st of Apr.
    • --> Naresh Kujala moved M1 out.

HED:

  • Laser interlock issue on Tuesday
  • with regards to the MID Mono cryo, there had been HED PLC updates
  • calibration pipeline issue on AGIPD
  • HED utca in optics hutch went off

SCS:

  • no issues

SQS:

  •  
Entry   Notes XO Integration Meeting 31.03.2021, posted by Frederik Wolff-Fabris on 31 Mar 2021, 11:01 
General Announcement:
- There is no UOM on next Friday April 02.
- The next XO Int. Meeting takes place on Wednesday 07.04.

CTRL:
- Loop updates by EEE today - CTRL will follow in case of support is needed.

DA:
- Offline calibration pipeline update was done yesterday successfully.
- Online calibration pipeline update: on Monday it was done in SA2; today it takes place in SPB with tests on the afternoon. These updates are related to the Detectors 
for the upcoming next user experiments.

EEE:
- Update in SA3 and SA2 motion loops from 10:00 to 15:00.

VAC:
- Not present.

XPD:
- Information got on the test of PKG for SA2: tests foreseen for this week was postponed.
- XPD working on the calorimeter.

DET:
- Nothing to report.

ITDM: 
- Update of the dCache storage was successfully done yesterday.

XRO:
- Nothing to report.

LAS:
- SA1 and SA2 PP-lasers are fine;
- SA3 PP-laser problem with timing system was fixed but required longer than expected. Laser was OFF until this morning. SQS shall not count with available laser (50-50% 
chances) tomorrow morning. 

UNSYS:
- PKG test was shifted.

SPB:
- Problem with laser hutch interlock when closing the door: when closing the external door this breaks the interlock (laser hutch). SPB will contact SRP.

FXE:
- Nothing to add.

MID:
- All right at MID.

HED:
- HED loop update takes place tomorrow. Nerea J. will discuss with HED; there is no restrictions from HED side.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 29.03.2021, posted by Frederik Wolff-Fabris on 29 Mar 2021, 10:45 
General Announcement:
- The UOM does not take place on next Friday 02.April.
- Pouneh S. informs the SCS panic button is not properly working (the search hutch does not perform as expected). This does not prevent operation and will be look after 
in the coming days.

CTRL:
- Following the DOC developments; nothing major to be announced.
- Karabo 2.10.4 is to be released which contains features to facilitate opening scenes and has some bug features fixed. CTRL hopes for a wider adoption. 

DA:
- Nothing major to report.

EEE:
- EEE has a time slot on Wednesday from 10:00 to 15:00 to update loops in SA3 and SA2. Updates shall be transparent and an e-mail will be sent. 
A. Galler confirms there will be no beam operation during this time.

VAC:
- Nothing to report.

XPD:
- ZZ request for Shimadzu camera in XTD9: A. Galler says it is not yet scheduled. As soon as valid information is coming, it will be communicated (maybe on Wednesday 
during the loop updates time slot).

DET:
- Small things were fixed with DOC help on available calibration constants for one SPB detector was fixed.

ITDM: 
- Reminder of tomorrow's maintenance on storage system.

XRO:
- Not present.

LAS:
- Thursday's SA2 ADC problem was fixed. Today LAS and EEE will attempt to fix the SA3 missing triggers in the ADC cards.
- All PP-lasers are running.

UNSYS:
- Information: there will be from today to Wednesday tests in SA2 performance with and without air conditioning systems.

SPB:
- Few problems with last week solved by DOC.
- SPB shift crew said machine was running at 2.2MHz. This had to be informed to BKR Operators.

FXE:
- Nothing to report.

MID:
- MID requests that planned interventions in MID hutches that are longer than few minutes and restrict access to it shall be informed in advance. This allows proper 
planning of activities. 

HED:
- Started using PP-laser and preparing for next user run (also tests with THz laser).
- Konstantin S. would like to thank EEE for the help received during last week.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
Entry   Notes XO Integration Meeting 24.03.2021, posted by Frederik Wolff-Fabris on 24 Mar 2021, 10:06 
General Announcement:
-	 

CTRL:
- In SA2/HED few interlocks are protected and not altered. CTRL needs to re-start the HEDloop1 and HEDcryo loop and SA2 EPS loop. It takes about 5-10 minutes without PLC 
restart. 
Suggestion is to make it today: XO will communicate to HED. 
- When submitting a ticket, please include the device ID on it. 

DA:
- Not present due to training. 

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Problem with SPB Jungfrau: M. Sikorski says the chiller was OFF and two modules are misbehaving. One can’t be fixed now and cards are ordered and have to be replaced.
The Chiller went OFF manually. It may become protected by EPS.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- SA1 and SA2 PP-lasers are fine;
- In SA3 still ongoing due to small problem of missing train-IDs in ADC cards. Consequently, no DAQ data can be taken for SCS and SQS. 
Nevertheless, the laser is ready for SCS use (not yet available to SQS). 

UNSYS:
- ZZ for the cell36/SA1 took place on Monday. The problem related to a coupler (spare parts are available and were replaced). It is fixed.

SPB:
- Start with data collection. Aside from detector, no other issues.

FXE:
- Nothing to report.

MID:
- All fine.

HED:
- Not present.

SCS:
- A problem was found on Monday regarding the door interlock. 
Zunaira A. comments that during a hutch search training it was found that a switch off (red button /panic switch) was pressed and did not release the magnet locks.
SRP has contacted a company and they have to come in to fix it. 
Zunaira A. sends another instruction to SCS staff in how to proceed in case of emergencies (one has to open door manually).

SQS:
- Nothing to report. Panic button at SQS works normally.
Entry   Notes XO Integration Meeting 22.03.2021, posted by Frederik Wolff-Fabris on 22 Mar 2021, 10:14 
General Announcement:
- The Covid-19 health task force is looking into testing schemes to be employed on-site. In the future there will be an opportunity to do the self-tests on-site for 
people who does critical works.
- The UOM series will restart on next Friday 26/03 at 09:00. Communication via e-mail will be send.

CTRL:
- Follow-up for delay train-ID: progressing with discussion on using longer time data.

DA:
- Thomas Michelat reports one issue with Agipd. The Maxwell server was restarted and then issue was fixed. If something similar happens again, the advice is to contact 
Maxwell team from Desy.

EEE:
- Today EEE perform the SEPS update for SA3 Interlock. For SA2 the test takes place on Thursday.

VAC:
- Not present.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- ITDM reminds on the storage/cache maintenance taking place next week (inclusion of additional storage place). Janusz S. will send an e-mail to readiness list.

XRO:
- Nothing to report.

LAS:
- SA3 PP-laser is off today due to work on interlock system. LAS hopes to get back as soon as possible (not before Thursday/Wednesday) for SCS use.

UNSYS:
- Problem in cell36/SA1 (communication error). A ZZ will be needed to recover the cell.

SPB:
- Ongoing with preparations for back-up user experiments; A problem with Jungfrau has been communicated.

FXE:
- Problem with Mono last week: positions for the crystals were shifted by 10mm. It is not known if this is a physical or only reading issue.

MID:
- Pressurized air by a compressor was moved to another location; the topic was discussed with TS and a big compressor will be installed until May 1st to solve problems. 

HED:
- Nothing to report.

SCS:
- Issued a ticket with EEE to verify possibility to averaging positions on middle layer servers.

SQS:
- Preparing for beam in April.
Entry   Notes XO Integration Meeting 17.03.2021, posted by Arvid Gebhardt on 17 Mar 2021, 09:15 

General Announcement:

- nothing.

CTRL:

- LAS reported the pulse statistics did not work properly: re-started systems due to lost connections.

- OCD call related to HED AGIPD test system: fix ongoing.

- Update on train-ID slow data on XGM: CTRL is in touch with Desy and Theo. Question to SPB and XPD: how much effort is needed for that CTRL and ITDM to place the missing information in the run. Involved parts will discuss an agreement by the end of the week. Steffen H. will prepare a summary to inform XPD.

DA:

- no important news.

EEE:

- no important news.

VAC:

- nothing to report.

XPD:

- nothing to report.

DET:

- nothing special.

ITDM:

- all fine.

XRO:

- Not present.

LAS:

Going fine: Laser pattern issue from yesterday was solved by CTRL.

UNSYS:

- Problem on Monday with quad mover cell18/SA1 took 6 hours on ZZ. It is now fixed.

- A problem with undulator segment on cell36/SA1 needs diagnostic and eventual ZZ access. RCs are contacted.

SPB:

- The samples to be used next week are challenging to prepare. 

FXE:

- Used beam yesterday. No problems to report.

MID:

- all fine.

HED:

- nothing to report.

SCS:

- nothing to report.

SQS:

- all good.

Entry   Notes XO Integration Meeting 15.03.2021, posted by Frederik Wolff-Fabris on 15 Mar 2021, 11:26 
General Announcement:
- Following Desy e-mail to all EUXFEL staff, the emergency call system of the elevators will be offline on Thursday March 18th during the morning. Please avoid using the 
elevators in this day. 
- On March 23rd is scheduled the SEPS interlock tests in SA3. On March 25rd is planned for SA2, pending a confirmation with the machine on the impact in SA1 operation.

CTRL:
- Progressing with investigating the train-ID issues, currently it relates to DOOCS/Desy.
- DOC: reports one fix today at 13:00 (following a call). Please respect a sign "we are in", "we are out" at the DOC to help staff focusing on issues. 

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- All working fine in XPD.
DET:
- Detector commissioning in SPB: AGIPD was completed as planned and Jungfrau seems to run properly (data to be analyzed.

ITDM: 
- ITDM will add more space to the storage system (data taking). Initial idea is to make it on March 30th to minimize impact. Acquiring data and data analysis are not 
impacted. 

XRO:
- Beam power increase ongoing.

LAS:
- An issue with SA1 interlock in PP-laser room: the inside doors were reported to be open on Thursday afternoon. 
As SPB was not using the PP-laser, the laser was put in safe mode and Interlock was broken to access and verify the situation. The internal doors were closed. 
Further discussions will follow. This door issue happened for the first time in SA1. 
- LAS is providing beam to SPB and HED as requested.

UNSYS:
- Couple of e-log entries on settings of air coils and phase shifter, all addressed during the weekend.

SPB:
- Detector program is completed. One measurement (which is not critical) is missing from Jungfrau.
- Interacting with DOC worked nicely.

FXE:
- Preparing for this week.

MID:
- Fully in preparation for first April beam. All fine.

HED:
- Nothing to report.

SCS:
- SCS asks for update of missing CPU from EEE. No news from EEE. Contact will be made.

SQS:
- Preparing to have beam in April.
Entry   Notes XO Integration Meeting 10.03.2021, posted by Arvid Gebhardt on 10 Mar 2021, 09:20 

General Announcement:

- nothing for now

CTRL:

- A few issues for DOC: Cameras IP addresses need to be reconfigured

- Continuing investigation of train-IDs on Doocs/XGM (they oscillate). CTRL will follow and close contact with Theo M. and Desy colleagues.

DA:

- nothing to report

EEE:

- nothing to report

VAC:

- nothing to report

XPD:

- Not present.

DET:

- nothing special – commissioning this week

ITDM:

- nothing to report

XRO:

- Maurizio V. reminds the SEPS interlock check in SA3 still has to happen as it is a condition prior beam operation. Zunaira A. says she sent an e-mail to xo-planning.

F. Wolff-Fabris asks if the SCS door is fixed or not, a pre-condition for the test. Zunaira A. replies the problem was with the power supply to the door and is fixed.

A date for the SEPS test can be planned by XO.

LAS:

- SASE2 interlock: LAS got unexpected calls when disconnecting AMPHOS cables (thus triggering SA2 Interlocks).

- SA3 AMPHOS no news since Monday. The ADC cards do not receive train-IDs.

- LAS suffer of lack of laser lab, missing PLC installation. LAS asks XO for a follow-up on prioritizations.

- Zunaira says that lasers are coupled to the interlocks (yesterday event happens as all lasers in room A23 and A07 are connected to the HED and MID interlocks). Thomas Preston says it triggers the warning lights for the laser lab. Zunaira A. says if shutters are closed it shall not be a problem if lasers reach this shutter signals. Tomasz J. says that ideally they must be decoupled.

UNSYS:

- nothing to report

SPB:

- going on with the plan; Detector commissioning from tomorrow.

FXE:

- the floor works are ongoing - looking good.

MID:

- not present

HED:

- nothing to report

SCS:

- A new experimental chamber (chemistry chamber) will be installed for beam commissioning on next month. A CPU in rack room is needed for recommissioning the motor an Martin T. waits for feedback.

SQS:

- Staff is getting the search training which was expiring.

- Noticed issues notice with DACHS system which show the access properties with delay. 

Entry   Notes XO Integration Meeting 08.03.2021, posted by Frederik Wolff-Fabris on 08 Mar 2021, 10:49 
General Announcement:
- SPB/SFX will continue receiving beam this week for commissioning.
- Issues with access to the Operations/21 e-log: SPB run coordinator wrote to wrote to the admin but no answer so far. Entries by Adrian M. while using VPN also did not 
work. 
XO will follow with IT (see below).
- Gerd W. reports that works in SCS chicane are done and the radiation interlock keys are returned.

CTRL:
- Some observations triggered by DOC regarding XGM trains coming too late (20s). An update will be made later on the day or next days.

DA:
- Not present due to DOC shift compensation day.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- A. Koch and Theo M. say no issues to XPD were reported. All operating fine.

DET:
- Commissioning beamtime with SPB detectors. The AGIPD program for last week was completed successfully.

ITDM: 
- Janusz S. says on the e-log issues that administrator of the e-log has the power to include new people. 
While using VPN access one can read the e-log and post entries using the copy function from another e-log. Nothing in the procedure has changed from past time.

XRO:
- Nothing to report.

LAS:
- SA1 PP-laser recovered power for SPB and is delivering in the nominal power level.
- SA3 AMPHOS was switching off at random during the weekend; no reliable operation in SA3. LAS is in contact with external company.

UNSYS:
- Cell4 and cell6 in SA2 required intervention from expert (Suren K.) during the weekend. 

SPB:
- Drift of horizontal beam position was observed on the weekend and reported to the DOC which provided a tool to diagnostic the drift. SPB/SFX would like to thank the DOC 
for the availability.
Beam pointing was changing every hour and was difficult to keep into M1 and M2. Theo M. says the XGM drift happened in the horizontal by many mm. DOC can not solve and 
then contacted Optics group. 
The observed drift seemed to be too big to be compensated. Richard B. says the movement was seen from XTD9 XGM and was a fast drift. 
Andreas G. will address communication channels between DOC/DRC and PRC (PRC was not informed). 

FXE:
- Floor refurbishment starts today.
- Last Wednesday it was checked the gate valves are corrected integrated in EPS and MPS. If shutter is open this signal is sent to the MPS and triggers closing V0.

MID:
- All fine.

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- All good at SQS.
Entry   Notes XO Integration Meeting 03.03.2021, posted by Arvid Gebhardt on 03 Mar 2021, 09:23 

General Announcement:

- Hick-ups in the beam operation are happening since last night. There is no obvious indications on the origin of the problem and MXL is investigating to restore a normal beam delivery condition.

- There is an issue with “Operation-2021” e-log where some SPB staff can’t login to create entries. This will be addressed by XO/PRC and ITDM (see ITDM entry).

CTRL:

- SA1 DAQ was dropping some trains (the late ones) in the XGM. CTRL and ITDM are following up and will test the XGM late this week or next week. CTRL will contact XPD for deployment first in SA2 and SA3.

DA:

- News from the DOC: deployment of major updates ongoing/to take place.

EEE:

- nothing to report

VAC:

- not present

XPD:

- Commissioning and setting up for SA3 HIREX

DET:

- preparing for commissioning Detectors

ITDM:

- Janusz S. says the configuration for the “Operation-2021” e-log uses the same configuration as previously set for 2020. F. Wolff-Fabris likens the issue with possible new users and asks SPB to provide the name of users who can’t login into the e-log.

Janusz S. says one can use the copy function from instruments e-log to place a copy into the Operations-2021 e-log. Jan G. says that login is necessary to avoid the entry being as “Anonymous”.

XO and ITDM will address it offline.

XRO:

- not present

LAS:

- In SA3 PP-laser one had exchange of power supply for AMPHOS due to communication errors.

- In SA1 PP-laser one observes a loss of 1/4 of power in AMPHOS. LAS is not sure if there is enough power for users.

- Two problems with triggers on Monday: 1. a faulty board in SA3; 2. MCS group update timing system caused trigger loss in all hutches (due to a bug in their server).

UNSYS:

- nothing to report

SPB:

- Couple of Helps asked to DOC and SPB/SFX are thankful for that.

- Login at Operation-2021 e-log does not work for 4 people.

FXE:

- Prepared for AIBS. Few PLC problems remain but not related to AIBS.

- Next week between Tuesday/Friday there are no access to the hutch due to floor work ongoing.

MID:

- all fine

HED:

- all fine

SCS:

- sliding door will probably be repaired today with an external company.

SQS:

- nothing to report

Entry   Notes XO Integration Meeting 01.03.2021, posted by Frederik Wolff-Fabris on 01 Mar 2021, 10:49 
General Announcement:
- An e-mail from Steve Aplin informs on how to contact the DATA operation center during beam delivery times. It changes contact phone numbers for OCDs. 
Old numbers are discontinued. The 98089 is the new number.
- The lead scheduling for machine/operation can always be found at the TTFINFO link (BKR E-log). Any other schedule is not official.
- Hutch search training goes invalid for many scientists in the instruments. Please check your status at the DARF-DACHS system or with RPOs.
- If there is anything unclear with masks or Corona rules, please give feedback in this meeting series or to the corona task force.
- Gerd W. says that works on SCS chicane are taking place right now. Afterwards a check will be done until Wednesday. The SCS key is still removed.

CTRL:
- Next days CTRL would like to solve DAQ situations while restarting devices. For SPB it is under discussion when to proceed.

DA:
- Not present due to planning meeting for end-to-end DA commissioning.

EEE:
- Fast electronics: a problem with SA3 PP-Laser (timing issue) was found and a board will be replaced.
- PLC side: deploying SA3 laser interlock. EEE will get in touch with XO for agreeing on a date.

VAC:
- XTD6 is still open and will be closed tomorrow (Tuesday 02.Mar) afternoon.

XPD:
- Friday at 15:00 there was a wide spread network issue and XGMs were giving warnings. The XGMs came back rather quickly but beam was lost until Saturday.

DET:
- Preparing for the SPB detectors commissioning with beam this week. One module was exchanged and will be evaluated next Monday.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- SA3 PP-laser: issue on Thursday afternoon late; Replacement boards were received. Tomasz J. informs the triggers were lost in the last 5 minutes.
- Tomasz J. question if LAS is going to receive enough support on Mondays from the DATA center as they have to prepare lasers for Tuesday. 
LAS expect same level of support on Mondays from DATA department. Steffen H. says that the OCD and Data Run coordinator will support the activities.

UNSYS:
- Nothing to report.

SPB:
- Receiving beam from tomorrow on.

FXE:
- Working with EEE to get Beckhoff loops finished for the AIBS test taking place at FXE on Thursday.

MID:
- All fine at MID.

HED:
- HED asks for the status on the ADC issue. Bruno F. says it is being fixed today (crate is up and running) and objective is to finish by today.
- Question to CTRL on cameras: S. Hauf says that CTRL will ask HED to test the fix. HED will contact Dennis G.
- Pouneh S. says that for laser safety googles it is exceptionally allowed to use FFP2 masks with filters. A. Galler points that the Corona task force strictly forbids 
this type of masks at XHEL as protection against Coronavirus. 
This subject will be brought to the Task Force for more clarifications.

SCS:
- Sliding door issue is ongoing. Eric Boyd says that Zunaira A. is in contact with the company and they will come to XFEL on Wednesday (it is caused by the controller of 
the door).

SQS:
- Nothing to report.
ELOG V3.1.4-7c3fd00