Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 FXE SPB HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 21 of 32  Not logged in ELOG logo
ID Date Author Location Severity Subject
  231   07 Jun 2021, 10:29 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 07.06.2021
General Announcement:
- Accelerator has been shutdown this morning and the Summer/21 maintenance period has started.
- A. Galler asks the instruments to check the beamlines are in safe state for that tuning and setup starts after the maintenance without potentially damaging equipment.

CTRL:
- Reminder that tomorrow CTRL starts the deployment of larger Karabo topics (details with contact persons).

DA:
- Nothing to report from OCD service.
- Detector calibration: Philipp S. reminds that the pipelines/services will be temporarily removed for updates but it will be brought back.

EEE:
- Nothing apart of maintenance tasks.

VAC:
- Nothing to report.

XPD:
- Jan G. reports as PRC many technical issues happened during the weekend: 
1. Problem with DACHS system (hutch search at HED was not able to be performed). This was a communication problem and DACHS server had to be rebooted There is no OCD over 
weekend). 
2. EPD went down due to network failure (down for all tunnels and was fixed on Sunday). Martin Dommach confirms that the entry at TXD9 was possible this morning.
- XPD reports on a comment from SPB on June 1st says a damage of 25um Si was found. Jan G. asks more details which device it is. 
M. Sikorski will check that. Naresh K. says it shall be in the SPB branch.
- XGMs are powered down for the maintenance.

DET:
- DRCs/DOC is OFF during the maintenance period. If support is requested, please contact support via ticketing system.

ITDM: 
- For the shutdown period, the emergency IT phone is available only for critical service (old number 98070).
- Storage systems will be serviced and will not be in operation all the time for the next weeks.

XRO:
- Not present.

LAS:
- SA2 and SA3 PP-lasers will be running next weeks for instruments.
- AMPHOS work taking place next week: needs cooling water and timing system until June 22nd. For short interruptions of 20 minutes the doors will be opened. 
Patrick G. says that local timing is a good solution. Tomasz J. says they have equipment to provide for 1 PP-Laser.

UNSYS:
- Quad mover issue in cell18/SA1 will be looked on today or tomorrow.

SPB:
- Nothing to report.

FXE:
- Nothing to report.
- A. Galler asks if cross talk was observed during the weekend. Jan G. says he talked to SQS this morning and they made checks and on Saturday they did not find any shift 
or cross-talking. 

MID:
- All fine at MID.
- Accesses in the optics hutches may be restricted. Also the experimental hutch has restricted access. If access is needed, please contact MID staff beforehand.

HED:
- DACHS system: one has to investigate further in the cache if this will happen again. 
Jan G. says that there is an entry by D. Kruse and the hutch search is another issue. Maybe the cache works only for one and not both issues.
- For the ADC T. Preston will investigate why the bunch pattern was not read correctly when changing the settings.

SCS:
- Nothing to report.

SQS:
- Not present.
  229   02 Jun 2021, 10:24 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 02.06.2021
General Announcement:
- XO reminds staff to check the safety trainings and the DACHS access rights/permissions for the tunnels when planning tunnel accesses during the maintenance period. 
- A. Galler asks all instruments who work with X-ray beam that in case it is found any damaged component, please take a snapshot and post in the operation e-log. 
This allows to get known prior/during the maintenance period.
- Last UOM prior maintenance period takes place next Friday. 

CTRL:
- For the maintenance period: CTRL will send around the Karabo schedule for updates such as MC2 motors. CTRL will get also in touch with affected people.
- Macro update announcement on Monday will not take place anymore.

DA:
- Nothing to report.

EEE:
- EEE is gathering and scheduling all shutdown tasks and will start next week with tunnel works.
- SASE1 VAC, EPS and MOV, and SASE2 MOV2 loops will be updated next week.
- EEE will also assist, if needed, on the testing of the Mc2 interface migration that Steffen H. / CTRL has informed.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- This is the last week of DOC operation. The DOC will re-start on cw28.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Man Jiang reports all 3 PP-Lasers are running.
- SCS can't open the shutter (interlock issue) but this is not related to LAS.

UNSYS:
- Nothing to report.

SPB:
- DOC has contacted SPB saying Maxwell calibrations failed. This will be checked.

FXE:
- Nothing to add.

MID:
- All fine at MID.

HED:
- Issue with camera at 10:30PM. It Needed to be power cycled. Steffen H. says that the camera worked this morning. Maybe case is already solved. One will keep an eye if 
it happens again.

SCS:
- Issue with Interlock mentioned by LAS. Pouneh S. is working on it.

SQS:
- User operation this week including optical laser. 
- Thomas B. says SQS struggles with temporal overlap but spatial overlap is fine. They have first to find clear signal to investigate more on the temporal overlap.
  228   31 May 2021, 10:18 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 31.05.2021
General Announcement:
- Next Friday is the last User & Operation meeting for this run. We will have presentations of 3 user groups.
- Today 31.05 is a normal delivery beam day.

CTRL:
- Information for maintenance period: CTRL will update macros for compatibility at SPB and SCS instruments. CTRL will contact the instruments for scheduling.
- Undulator energies were not corrected updated in Karabo. Re-starting the servers may help. A mismatch of values with Doocs were found.

DA:
- Nothing to report - quiet OCD week from last Wednesday.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Couple of connection issues due to power glitch this morning at 4:56AM. All affected systems were recovered.
- The DOC is not operational today. Tomorrow at 7AM the DOC is back to operation. Calls goes today directly to DRC.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- PP-Laser in SA1 is down due to power glitch. Cooling water (8 or 11 C) for AMPHOS was affected and caused this downtime. Water has come back few minutes ago. 
- SA2 and SA3 lasers were not affected. There are different pumps for each SASEs. Lorenz K. is investigating (e-mails from GLT and no warning from MKK). 
- SA1 PP-laser will be stable by the afternoon as requires time for stabilization.

UNSYS:
- Nothing to report.

SPB:
- Laser interlock on Wednesday to Thursday night was triggered: panic button switch was falsely indicated that was pressed. It seems to be a failure in the system. 
Desy / BKR reset it and operation could be restored. 
Zunaira A. says that if panic switch is pressed, then the beam shutter will not be able to be moved (SPB will not be able to receive beam). 
There is no relation with the power glitch.

FXE:
- Nothing to report.

MID:
- Looks fine at MID. Joerg H. asks for an e-mail informing on situations like the power glitch for that groups be informed and check if devices are affected. 
A. Galler says currently the reason for the power glitch is under investigation and will be communicated. 

HED:
- This week will be a busy week due to energy changes already at 7:00AM.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  227   26 May 2021, 10:23 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 26.05.2021
General Announcement:
- XO reminds the procedure for ZZ access is organized through the PRC. 
If a ZZ is needed, please get in touch with the PRC for advice and fill it up the Redmine access request (link located at the TTfinfo e-log). 
The PRC will organize and communicate the access with machine RCs, specially in cases of urgent matters. 
It is not beneficial to do not inform the PRC as components in tunnels have to be checked and/or grounded prior the ZZ access takes place.

CTRL:
- Announcement for the summer maintenance: when updating to Karabo 2.11 some instrument macros have to be changed. CTRL will care that and will get in touch with 
instruments to properly organize it.

DA:
- Couple of E-log entries referring to DA but nothing major to report.

EEE:
- Reminder to FXE, MID, SPB, SCS: please check processes for eligible documentation. 
Bruno F. And Joern R. inform that in Alfresco there are various folders with pending documentations (automatic e-mail was sent to referents). Please contact Jorn 
Reifschlager in case of doubts.
- PES system in SA3: digitizers were stopping after acquiring data for a random amount of time. This will be addressed during the maintenance period, as the fix needs 
several hours of ZZ access.

VAC:
- Not present.

XPD:
- Nothing to report.

DET:
- Couple of issues with FXE Jungfrau detector this morning: solved or under fix.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Everything is fine with Lasers.

UNSYS:
- All good with Undulators.

SPB:
- Progressing well with beam activities during the nights.

FXE:
- Nothing to report.

MID:
- All fine at MID.

HED:
- Nothing to report. HED does not receive beam this week.

SCS:
- Nothing to report.

SQS:
- Nothing to report. PES digitizer issue affects SQS but one can continue with it until the maintenance period.
  226   19 May 2021, 10:13 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 19.05.2021
General Announcement:
- A. Galler says the Interlock Tests schedule is placed at the EUXFEL Infrastructure Calendar. 
Concerning the TUV tests at Schenefeld side, this happens between June 30th and July 2nd (the 2nd is a reserve date). When a detailed schedule is made, this will be 
communicated. 
Tomasz J. asks on the effects for the PP-lasers when, e.g. triggering NOT-AUS. XO will enquire with Zunaira A.

CTRL:
- The ADQ digitizer fix for SQS is working well. If other instruments would like it deployed, please contact CTRL.
- Bassler camers upgrade worked for SPB. One can also contact CTRL for the deployment.
- ITDM will need to exchange memory module in SA2 - Karabo will go down. This has to be scheduled between tomorrow and Friday. It will be coordinated by the PRC and CTRL.

DA:
- Nothing to report.

EEE:
- Yesterday incident in SQS: faulty PLC CPU triggered the OFF for 5 CPUs. Four of them have recovered and the fifth one will be replaced or fixed today.

VAC:
- Not present.

XPD:
- FD studies are ongoing this week. Nothing else to report.

DET:
- Nothing to report.

ITDM: 
- Today the update on the Maxwell cluster. Action take place between 10:00 and 12:00 and running jobs are not affected. Submission of new jobs will not be possible during 
this time.
- Issues with Network yesterday: Janusz S. says they do not have yet the final e-mail from Desy-IT. 

XRO:
- A. Galler says that on Monday the beam power reached 32 W at 2125 bunches. Mirrors were working properly during the whole test period.

LAS:
- All fine with PP-lasers.
- Tomasz J. mentions LAS will ask CTRL for update on the functionality of sync system properties for the PP-laser. In case instruments have also interest on it please 
contact LAS.
	
UNSYS:
- Issue with cell#18/SA1 quad mover: it was agreed to run in this condition until the shutdown.

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- All fine at MID.

HED:
- All fine at HED.

SCS:
- Nothing to report.

SQS:
- PLC issue yesterday afternoon: vacuum was lost from ^-11 to ^-6. This was a near-miss and almost made SQS to cancel next experiment. It seems to be under fix now.
  225   17 May 2021, 10:25 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 17.05.2021
General Announcement:
- Adrian Mancuso tells on the workaround for the crosstalk between SA1/SA3 documented by Harald Sinn. The idea was to switch off the feedback prior the bunch changes. 
Harald S. says that it is not sure if this has worked. Adrian M. says that fir the future this has to be addressed as both SA1 and SA3 would like to change bunch number.

CTRL:
- Follow up from the DOC last week: some cameras in HED started in unexpected states. This has been understood and when the update is ready the deployment has to be 
coordinated with HED.

DA:
- From DOC OCDs side only minor issues, that were solved by restarting devices.

EEE:
- Due to the special FD week starting today, there is no DOC this week. 
The DRCs (Patrick G. until Thursday and Monica T. from Thursday) will receive the calls: for normal issues, please send a ticket; for urgent cases call the DRCs. 
One can also contact directly the individual groups.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Incident at SPB with AGIPD detector: damage to several pixels. The cause for it is not yet known. 
The pixel modules have the replaced and the cause relates to intense radiation. It's important to follow up and understand how it happens. SPB will investigate and 
communicate findings.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Everything good with Lasers.

UNSYS:
- Issue with Quad mover in cell18/SA1 - Suren A. could address it without need of access but further investigations have to be done on-site.

SPB:
- Small injection incident over the weekend.
- Problem with one DAQ: number of pulses was OK but not the values. This was solved by the DOC and some data were wrongly saved for the users (attenuation values).

FXE:
- Timing tool tests at FXE during the night: crosstalk between SA1/SA3. When number of SA1 bunches were changed, the beam pointing was changing in SA3. 
So FXE was requested to do not change bunch numbers what hindered the tests.
- LPD issue: progress has been made last week and is being accelerated.

MID:
- Quick warning: until Wednesday some Seeding tests will be ongoing at MID.

HED:
- Successfully experiment last week: HED thank all involved people.
- Addressing CTRL deployment: HED will contact CTRL after this meeting to discuss some details before the deployment action goes on.
- On last Friday at 13:00 one camera corrected configurated did not acquire data and DOC was contacted.
- On Saturday the timing drifted linearly for 0.5ps. HED contacted PRC and DRC and is not yet completely understood.

SCS:
- Nothing to report.

SQS:
- Confirm what FXE has said on the crosstalk with SA1/SA3.
- Sometimes MCP detector was inserted and trigger the EPS to beam stop. Jan G. says that there was a mistake in the interlock configurations for going above 30 bunches.
When SQS wanted to go to 67 bunches it triggered the beam OFF.
  224   12 May 2021, 09:21 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 12.05.2021

General Announcement:

- User Operations Meeting takes place on next Friday 09:00.

CTRL:

- CTRL would like to excuse itself to people receiving an automatic SMS tonight for notification. This was a test message and now it’s turned OFF (Good side: test for lasers went well).

- ADQ digitizers: two fixes are driven and based for SQS commissioning. CTRL encourage instruments to get in touch with contacts for making the update for the last version.

DA:

- Follow up from Monday / calibration update (on behalf of cal team): it worked as planned. AGIPD seems to run faster now.

- For SQS Remi detectors/digitizers can also be used.

EEE:

- nothing to report

VAC:

- nothing to report

XPD:

- are preparing for FD studies next week

DET:

- Follow-up on LPD online calibration pipeline pointed last Monday’s meeting: work is on progress and is expected to be ready for next week.

- DAQ processing for AGIPD: a minor bug in validation is reported and under investigation. But it is currently working.

- Issue reported for SPB for data aggregators: follow up with CTRL.

ITDM:

- nothing to report

XRO:

- nothing to report

LAS:

- nothing to report

UNSYS:

- nothing to report

SPB:

- not present

FXE:

- Thank to DET and DA for pipeline calibration. Initially it looks promising.

- Last week: digitizer went to wrong configuration. To be fixed with new updated.

- FXE will use the beam during the nights. D. Khakhulin will inform PRC and XO via e-mail.

MID:

- all fine

HED:

- not present

SCS:

- nothing to report

SQS:

- nothing to add

  223   10 May 2021, 10:05 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 10.05.2021
General Announcement:
- Two smoke sensors affected in SA3 triggering fire alarm: The first one could be reset but for the second one the reset did not work and fire brigade was in. 
Beam operation was recovered only after the External company came to reset it. The downtime of about 4 hours for SA1/SA3 operation happened.

CTRL:
- Nothing to report.

DA:
- On behalf of calibration team: this morning a major upgrade will be done. No access to calibration services during this time.

EEE:
- One loop in FXE was not working during the weekend. Issues with movs/motor loops and the cause has to be investigated (old scripts were in use). No devices were 
damaged.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Deployment for HED calibration taking place shall fix last week's issue.
- A fix for DAQ on AGIPD to be done.
- FXE issue with LPD calibration: still open. This is not related to the DA deployment today. Jolanta S-D. will follow it inside DET.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- Nothing to report.

UNSYS:
- Fire alarms in SA3 / cell#22-23 on Saturday evening: BKR informed that could be related to synchrotron radiation on the detector signal. Fire brigade was on-site but no 
fire was found.

SPB:
- Nothing to report.

FXE:
- Thanks to EEE help with Gotthard.
- M2 Mirror pitch issues seems to be back to normal as before. Antje T. (XRO) made ZZ to correct loose gear box.
- Motor loops issues found and addressed by EEE.

MID:
- Nothing to report.

HED:
- Motor stop working and ticket has been written.
- Preparing for experiment this week.

SCS:
- Zoom problems last week has been taken care; Thanks to ITDM.

SQS:
- Nothing to report.
  222   05 May 2021, 10:20 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 05.05.2021
General Announcement:
-	 

CTRL:
- The device clone deployment went smoothly yesterday.

DA:
- Nothing to report.

EEE:
- Call from DOC referring to motor mirrors in SA1: XRO is investigating together with FXE. The issue is that pitch accuracy can't be reproduced as before.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Issue with offline calibration mention last Monday: the problem has been identified and a software fix is been worked (for HED detector). The calibration team is 
working on it.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report besides the SA1 mirror motors.

LAS:
- Problem with Optics in SA3 PP-laser (either coating degradation or lack of clean) was found. The mirror will be rotated.
- DOC reported a synchronization issue in SA2. It is not a LAS issue, has to be fixed by DOC.

UNSYS:
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Chris Milne mentions the steering issues with M2 and M3 mirrors and XRO is looking into it. 

MID:
- Synchronization issue with Laser is fixed.
- DAQ issue also fixed during the night.

HED:
- Things reported on Monday have been looked up.
- Train-ID issue: there is a walk around for it and test/fix to be done next week.

SCS:
- SCS asks for a solution to share screens with users in the control room (general solution also applicable to other instruments?). 
K. Wrona says ITDM has solutions that were tested last week at FXE and was working properly. 
An offline discussion between ITDM and SCS will take place to set the solution.  

SQS:
- Nothing to report.
  221   03 May 2021, 10:06 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 03.05.2021
General Announcement:
- The deadline for submitting the summer maintenance tasks is ended. In case you have not done so, please submit it as soon as possible.
- Monica T. as DRC invites people to attend the DRC jour-fixe meeting.

CTRL:
- Bug fix for the device clone is prepared and will be deployed today. The clone devices will go down for a couple of minutes only.
- ADC 412 bug: there is a fix and instruments shall get in touch with their contacts for the deployment.

DA:
- One OCD call related to Karabo bridge and usage of online monitoring: the problem was not related to Karabo but with an environment conflict.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- All PP-lasers are up and running.

UNSYS:
- Waiting for the ZZ in SA1 and SA3.

SPB:
- User experiments last week went well again.

FXE:
- Nothing to report.

MID:
- All fine at MID.

HED:
- HED run last week went well - quite successful with tests with Laser.
- Missing Train-ID issue was reported to expert groups; All the devices are not correctly seen (arriving some trains later); 
Steffen Hauf says CTRL are working and will be ready to test. There is a chance for tests in the coming week or still this week. 
An offline discussion will take place between CTRL and HED.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  220   28 Apr 2021, 10:04 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 28.04.2021
General Announcement:
- A. Galler points to the deadline for submitting summer maintenance task list on April 30th.
- XO would like to remind instruments that in case of operation failure related to Interlocks issue, please also include the PRCs (prc@xfel.eu) on the communications. 
The Photon Run Coordinators are the first contact point that can help in such situations.
- Tomasz J. asks on the reason for the water-cooling problems with SA1 PP-laser last Monday. A. Galler says the origin was a power glitch that caused the interruption of 
water systems. 
Raul V. says that it's also seen in some tunnel devices but because of the changes in interlock logics this worked nicely.
- Zunaira A. asks why many power glitches are happening. Andreas G. says there were this year 3 events with different causes. 
Last Monday's one was a real power glitch from distribution line; the one on Easter Sunday is under investigation via an 8D report and the one last week was caused by 
human error.

CTRL:
- A bug was found in the devices for HED (clown system). A test in HED will be done prior a full deployment.

DA:
- Not present.

EEE:
- Status of ADQs in SQS: Bruno F. says that it worked yesterday and suspicion is that data were saved locally. DOC and CTRL are following with this issue.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- All fine for Detectors.

ITDM: 
- New feature for data migration is set – main ADC.

XRO:
- Nothing to report.

LAS:
- All lasers are UP and running.

UNSYS:
- Issue with phase shifter in SA2/cell26 this week was solved (Doocs problem).
- Waiting for the ZZ access in SA1/SA3. A. Galler says that when time window opens this will be informed.

SPB:
- Experiment is going smoothly this week.

FXE:
- Last week HED had a few issues with brig brother server when scanning the undulators. The Big Brother was setting bunches back to zero. Issue will be followed up with 
machine.

MID:
- All fine at MID.

HED:
- Nothing to add; thanks to CTRL for looking after the scan tool.
- Test with Relax synchronization will be done tonight.

SCS:
- Started using the PES and recorded runs - about 35% of missing train-IDs. A. Galler says that one point of contact is via DOC. 
Steffen Hauf askes how many boards were using and Laurent M. informs 4 that run in the SCS topic.

SQS:
- Apart of the shift team in quarantine, everything seems to be going fine.
  219   26 Apr 2021, 11:08 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 26.04.2021
General Announcement:
- The deadline to submit the maintenance task list is on 30.04. We thank all the groups that have already submitted. In case of any question and comments, please contact 
us at xo-planning@xfel.eu
- Patrick Gessler reminds on the DRC meeting on Mondays 13:00. This meeting is done together with instruments to exchange information for the weekly plan. The e-mail for 
contact is drc-jour-fixe@xfel.eu 
- Andreas G. says that with the new 7-11 operation mode, XO is interested in knowing how longtime instruments need to be ready to take the beam in the early morning.
Please inform us for that XO checks the efficiency of this model.

CTRL:
- Remind on Beckhoff training next Wednesday (following e-mail to readiness).
- Upgrades of two Karaboo Beckhoff servers today (Karabo devices go down for 5-10 minutes, proposed to be done today during lunch time): SA2 master tunnel loop; and the 
HED loop.
- Bug on ADQ for twelve devices, affecting SQS and others: EEE has discovered it and CTRL is analyzing to come up with a fix.

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- PES in SA3 was provided and operated last week for instrument's usage.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Maurizio V. requests the SA2 experiments to provide information to XRO on issues related with mirror alignments or transport beamline.

LAS:
- Received info from Desy on a power glitch today at around 6AM. At the same time SA1 Amphos switched OFF due to not enough water. 
It’s under investigation with TS. Jan G. points that also there was a beam loss in delivery around the same time.
- SA2 and SA3 PP-lasers are still in operation.

UNSYS:
- Issue with undulator SA1 /cell33 and cell1/SA3 needs ZZ. Currently the impact in operation is minimal and ZZ is postponed to the next available date.

SPB:
- Nothing to report.

FXE:
- Run with beam delivery went smooth.

MID:
- Quite successful beam week with small hick-ups to be addressed with experts.

HED:
- Commissioning beamtime this week; Data analysis will be made this week.
- Test of the PLC synchronization with Relax laser to be done this week.

SCS:
- Not present.

SQS:
- uTCA in ADQ had been mentioned already by CTRL. It has been tackled.
  218   21 Apr 2021, 10:50 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 21.04.2021
General Announcement:
-  XO reminds on the deadline of 30.04 for submitting the Summer maintenance task lists. For this, please do so sending the list to xo-planning@xfel.eu. 

CTRL:
- Update on undulator Doocs side: CTRL needs to test for two or multi-colors schemes and asks instruments to CTRL if there are plans to use it.
- CTRL will schedule a short practical training on Beckhoff devices for next Wednesday. An announcement will be made through readiness mailing list.

DA:
- Nothing to report.

EEE:
- From the PLC side, EEE is following up the MID/CRLs interlock case.
- If groups submit a change request needed to be implemented during summer maintenance time, please contact and inform EEE beforehand.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- The network cable exchange mentioned last Monday was done.

XRO:
- Not present.

LAS:
- All 3 PP-lasers are in operation.

UNSYS:
- After power glitch on Monday in SA1, about half the undulator cells were half power cycled. Most could be recovered expect SA1/cell33 and SA3/cell1. This will require 
ZZ accesses. 
XO states the power failure was man-made caused.
- Quad mover sudden move: Lars F. from MXL does not have records on the command given for the sudden move and therefore the root cause can't be traced.

SPB:
- Nothing to report.

FXE:
- Beam since yesterday; one interlock issues. 
Nerea J. says a ticket was received and the interlock issue was related to a Imager that always come up activated. This will be removed from the EPS.

MID:
- All fine at MID.

HED:
- Nothing to report.

SCS:
- Undulator photon energy couldn't be changed via Karabo. Jan-Torben D. will talk to Steffen H; David Hickin adds that the location was changed and the Karabo side is 
done/corrected. 
One needs the BKR connection/authorization and this will be organized by CTRL/Steffen H.

SQS:
- Issue with digitizer on work. It is needed for next week.
  216   19 Apr 2021, 11:39 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 19.04.2021
General Announcement:
-  Remind on the Summer maintenance period: deadline to submit task list is April 30th. XO invite all groups to submit by this date.

CTRL:
- Remind on the update of some specific Karabo installations to fix bugs (Data logger installations) to start today (e-mail was sent to the readiness).
- Bunch pattern statistics devices need to be fixed today at the same time as the data loggers (relates to lose of time stamps in lasers).
- Upgrade on instruments are coordinated by ICIs; Updates of Mc2 motors today.
- Z. Ansari asks if updates of MID will affect interlocks as well; Valerii B. answers this is a Karabo update and not PLCs, so no effect is expected.
- On recent days CTRL finds Karabo broker pileup and this was successfully understood; DOC was preventively monitoring it and fixed it beforehand.

DA:
- No OCD calls for DA; 
- Collaboration and preparation for supporting SPB/SFX last week and for beamtime continuation in two weeks.

EEE:
- Today 08:30 it was found SA1 tunnel racks lost power. The uTCAs were also down but not the one in room 1.14. It affected SA1 tunnels loops such as Vacuum and motion.
The guess is on a power glitch. XO will bring up to the machine side and eventually correlated with the Easter Sunday's event.

VAC:
- Recovering the SA1 tunnel loops from 08:30 downtime. All seems to be fine right now.
- VAC asks if FXE is running cryo coolers. FXE tells they were running and will check the signals for possible issues.

XPD:
- HIREX motors in SA1 were OFF this morning.

DET:
- One OCD call was received and solved; DET is not aware of issues with Detectors. 

ITDM: 
- On the Wednesday problem with network (in SA1, the online calibration was down): Today a cable will be exchanged. 
- ITDM was involved with experiments together with DA on offline calibrations.
- As DRC, K. Wrona reminds today that 1PM there is the DRC meeting and the participation of instruments is strongly encouraged.

XRO:
- Nothing to report.

LAS:
- SA1 PP-laser is fine.
- SA2 PP-lase with small issue with regulations. Tickets sent to CTRL.
- SA3 PP-laser with several small problems: damage to optics (need to exchange one AMPHOS stage); interlock tripped due to door problems and LAS went OFF. 
LAS hopes to bring back laser and make available for instruments by tomorrow.

UNSYS:
- SA1/cell12 Quad mover sudden move last week: it was not related to a software issue and is under further investigation on Doocs side. A Doocs command was sent.

SPB:
- Experiment last week: quite successful and not many issues.

FXE:
- Nothing to add.

MID:
- All fine.

HED:
- A Motor on the CRL HED chamber was engaged and not moving (hardware issue). It may need a ZZ. It is not related to the CRL's arms.
- SEPS interlock tripped machine on Friday almost midnight (colleagues close shutter before moving CRLs). The beam permission was restored after lens removed from beam.
- ADC is offline. HED will contact Bruno F. offline.

SCS:
- Not present.

SQS:
- Commissioning for preparing next experiment.
- Issue with digitizers that can't be used: Bruno F. adds that action is scheduled for 14:00 as discussed with SQS's members.
  215   14 Apr 2021, 10:50 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 14.04.2021
General Announcement:
- We would like to remind groups that the deadline for submitting to XO the feedback on the planned activities for the Summer/21 maintenance time is on April 30rd. 
In case of questions please contact XO via xo-planning@xfel.eu

CTRL:
- Updates announced on last XO Int. meeting will take place next Monday (unless if a group protests). This means a short downtime on data loggers in tunnels. An e-mail 
with a reminder will be sent.
- Upgrade for the bunch pattern statistics devices to take place (also will be offline for few minutes).
- Issues with MC2 motors: the workaround is to shutdown device and click/instantiate device. A fix will be deployed.

DA:
- DA involved and continuing with "Moonshot" coordination meeting.

EEE:
- Nothing to add.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- Announcement with respect to Maxwell cluster: ITDM will change partition schemes in the next Summer maintenance. 
Three new partitions will be introduced, for calibrations and a special one for reservations for beamtime (UPEX related); 
A new feature is that if Maxwell is full of jobs, one can pre-empty it by killing jobs and putting back in the queue.

XRO:
- Yesterday beam going to HED: some problems were found regarding parameter configurations. 
XRO reminds SA2 instruments to be careful and check devices positions and parameters are they may be changed in relation to April 1st due to the issue with PLCs during 
Easter Sunday. 
- Zunaira A. says there were problems with CRLs arms configurations. Some devices had to be restored but worked normally.

LAS:
- All lasers are fine. 
- Shutters in SA1 PP-Laser were in maintenance mode - without defined reason.
- SA2 PP-laser is fine. 
- SA3 PP-laser has degradation due to the cleaning. XO will address the issue with LAS.

UNSYS:
- All fine.

SPB:
- Running user experiment. Low photon energy yesterday (0.8mJ) and drop intensity over the full 352 bunch train but today it got better.

FXE:
- Nothing to add.

MID:
- All fine.

HED:
- All motors lost positions, e.g. mono, screens being driven (some in and not supposed to be in). CRL1 issue the position was lost at the axis; motors not being homed; 
Zuzana K. will summarize main issues and forward to XO.
- Steffen H asks for a summary of encoders and Zunaira A. also asks on the use of interlocks and CRLs/shutter to evaluate if all is fine. Zuzana K. says HED did not work 
yet on the CRL2 but will communicate as soon as experience is get.

SCS:
- Question to CTRL on the workaround of MC2 motors. Steffen H. says to shutdown device, check configuration and instantiate it again. An instruction e-mail will be sent 
to readiness list.

SQS:
- Nothing to report.
  214   12 Apr 2021, 14:40 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 12.04.2021
General Announcement:
-	 

CTRL:
- Broker pile-up on Friday: Alessandro S. says that was triggered by network configurations; it took time to recover situation. 
Started SA3 topics for deployment and for few tickets related to that, please get in touch with DRC or try re-starting devices.
- Deployments: 1. Fix for data loggers and timing servers. We propose to do next Monday; 2. Karabo 2.10.4 was released with few bug fixes. CTRL is in touch with 
instruments.

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Found on Saturday a problem with SA3 gas attenuator. It was solved with the DOC.

XPD:
- Power increase tests ongoing this morning.
- A. Koch says two servers were not running this morning and will make contact to DRC.

DET:
- Couple of OCD calls received and fixed.

ITDM: 
- Network failure: systems are very sensitive to network failures (switch failure). Relates to replacement of devices.

XRO:
- Some SA3 cameras lost configurations (maybe related to the SA3 event during the weekend): fixed by M. Vannoni. 
- SCS/SQS teams: structure for the M6 chamber has to be fixed (motors): XRO will communicate with the instruments, to keep the motors in proper conditions.

LAS:
- PP-lasers are running. Small minor issues: 
1. SA3 problem with displaying number of pulses. Solved by CTRL by installing latest Karabo server;
2. SA2 still shows small problem with regulations/settings, spectrum started drifting. Can be fixed by OCD but ticket was created.

UNSYS:
- Nothing to report.

SPB:
- Preparing for user experiment this week.

FXE:
- Nothing to report:
- Last Friday PP-laser could not config laser pattern. Tomasz J. says the OCD laser is aware and asks to please also inform DOC/DRC in such cases as LAS can't help in 
this time. 
Monica T. confirms that one has to call the DOC and invite instruments that do not get beam to also join the Monday 13:00 DOC meetings.

MID:
- Suffered with SA2 PLC downtime - mono lost positions.
- Scan energies on the undulators will be needed on next week.
- Last week operation was smooth.

HED:
- Starting operation this week.

SCS:
- Nothing to report.

SQS:
- Spectrum of detectors - two spectras were split. This was reported to BKR and DOC.
  213   07 Apr 2021, 08:52 Andreas GallerGeneralInformationNotes XO Integration Meeting 07.04.2021

General Announcement:

  • .

CTRL:

  • recurrent error reprted by LAS  --> drift compensation in SA2 --> need feedback
  • minor issues reported instruments during startup

DA:

  •  

EEE:

  • Issue on April 4th --> 4 PLCs shut down without a cause, (looks like power cut)
  • MID crate went down at 9:00
  • SA3 utca shut down (could be voltage drop, could be ehternet) (9:35)

VAC:

  • VAC-OCD recovered SA2 vacuum state (valves, and ystem) (see EEE SA2 incident above)
  • VAC obseved that the MID Mono warmed up, the cryo warmed up presumably after a PLCs update which interrupted an interloop communication

XPD:

  • nothing report

DET:

  • everything running fine

ITDM:

  • SA1 baclony room checked
    • some racks lost power --> investifation going on (possibly happened on Sat. Apr 3rd 8:00 am)

XRO:

  • incorrect motor velocity in SA3 --> fixed
  •  

LAS:

  • small issue in SA2 with slow feedback --> see CTRL report

UNSYS: 

  • problem on SA1 central control node, after a power cycle aircoils and phase shifters were unresponsive

SPB/SFX:

  • Laser Interlock issue was solved, power spike involved that reported by MEA3 ( klaus Witt).

FXE:

  • no issues

MID:

  • M1 mirror was moved completely out on 1st of Apr.
    • --> Naresh Kujala moved M1 out.

HED:

  • Laser interlock issue on Tuesday
  • with regards to the MID Mono cryo, there had been HED PLC updates
  • calibration pipeline issue on AGIPD
  • HED utca in optics hutch went off

SCS:

  • no issues

SQS:

  •  
  212   31 Mar 2021, 11:01 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 31.03.2021
General Announcement:
- There is no UOM on next Friday April 02.
- The next XO Int. Meeting takes place on Wednesday 07.04.

CTRL:
- Loop updates by EEE today - CTRL will follow in case of support is needed.

DA:
- Offline calibration pipeline update was done yesterday successfully.
- Online calibration pipeline update: on Monday it was done in SA2; today it takes place in SPB with tests on the afternoon. These updates are related to the Detectors 
for the upcoming next user experiments.

EEE:
- Update in SA3 and SA2 motion loops from 10:00 to 15:00.

VAC:
- Not present.

XPD:
- Information got on the test of PKG for SA2: tests foreseen for this week was postponed.
- XPD working on the calorimeter.

DET:
- Nothing to report.

ITDM: 
- Update of the dCache storage was successfully done yesterday.

XRO:
- Nothing to report.

LAS:
- SA1 and SA2 PP-lasers are fine;
- SA3 PP-laser problem with timing system was fixed but required longer than expected. Laser was OFF until this morning. SQS shall not count with available laser (50-50% 
chances) tomorrow morning. 

UNSYS:
- PKG test was shifted.

SPB:
- Problem with laser hutch interlock when closing the door: when closing the external door this breaks the interlock (laser hutch). SPB will contact SRP.

FXE:
- Nothing to add.

MID:
- All right at MID.

HED:
- HED loop update takes place tomorrow. Nerea J. will discuss with HED; there is no restrictions from HED side.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  211   29 Mar 2021, 10:45 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 29.03.2021
General Announcement:
- The UOM does not take place on next Friday 02.April.
- Pouneh S. informs the SCS panic button is not properly working (the search hutch does not perform as expected). This does not prevent operation and will be look after 
in the coming days.

CTRL:
- Following the DOC developments; nothing major to be announced.
- Karabo 2.10.4 is to be released which contains features to facilitate opening scenes and has some bug features fixed. CTRL hopes for a wider adoption. 

DA:
- Nothing major to report.

EEE:
- EEE has a time slot on Wednesday from 10:00 to 15:00 to update loops in SA3 and SA2. Updates shall be transparent and an e-mail will be sent. 
A. Galler confirms there will be no beam operation during this time.

VAC:
- Nothing to report.

XPD:
- ZZ request for Shimadzu camera in XTD9: A. Galler says it is not yet scheduled. As soon as valid information is coming, it will be communicated (maybe on Wednesday 
during the loop updates time slot).

DET:
- Small things were fixed with DOC help on available calibration constants for one SPB detector was fixed.

ITDM: 
- Reminder of tomorrow's maintenance on storage system.

XRO:
- Not present.

LAS:
- Thursday's SA2 ADC problem was fixed. Today LAS and EEE will attempt to fix the SA3 missing triggers in the ADC cards.
- All PP-lasers are running.

UNSYS:
- Information: there will be from today to Wednesday tests in SA2 performance with and without air conditioning systems.

SPB:
- Few problems with last week solved by DOC.
- SPB shift crew said machine was running at 2.2MHz. This had to be informed to BKR Operators.

FXE:
- Nothing to report.

MID:
- MID requests that planned interventions in MID hutches that are longer than few minutes and restrict access to it shall be informed in advance. This allows proper 
planning of activities. 

HED:
- Started using PP-laser and preparing for next user run (also tests with THz laser).
- Konstantin S. would like to thank EEE for the help received during last week.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  210   24 Mar 2021, 10:06 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 24.03.2021
General Announcement:
-	 

CTRL:
- In SA2/HED few interlocks are protected and not altered. CTRL needs to re-start the HEDloop1 and HEDcryo loop and SA2 EPS loop. It takes about 5-10 minutes without PLC 
restart. 
Suggestion is to make it today: XO will communicate to HED. 
- When submitting a ticket, please include the device ID on it. 

DA:
- Not present due to training. 

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Problem with SPB Jungfrau: M. Sikorski says the chiller was OFF and two modules are misbehaving. One can’t be fixed now and cards are ordered and have to be replaced.
The Chiller went OFF manually. It may become protected by EPS.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- SA1 and SA2 PP-lasers are fine;
- In SA3 still ongoing due to small problem of missing train-IDs in ADC cards. Consequently, no DAQ data can be taken for SCS and SQS. 
Nevertheless, the laser is ready for SCS use (not yet available to SQS). 

UNSYS:
- ZZ for the cell36/SA1 took place on Monday. The problem related to a coupler (spare parts are available and were replaced). It is fixed.

SPB:
- Start with data collection. Aside from detector, no other issues.

FXE:
- Nothing to report.

MID:
- All fine.

HED:
- Not present.

SCS:
- A problem was found on Monday regarding the door interlock. 
Zunaira A. comments that during a hutch search training it was found that a switch off (red button /panic switch) was pressed and did not release the magnet locks.
SRP has contacted a company and they have to come in to fix it. 
Zunaira A. sends another instruction to SCS staff in how to proceed in case of emergencies (one has to open door manually).

SQS:
- Nothing to report. Panic button at SQS works normally.
ELOG V3.1.4-7c3fd00