What has happened:
During euler motion calibration of M1, we were driving the vertical motors by a few mm back and forth. At some point, the Rz motor went into error, as we attempted to go further in negative direction. Resetting the axis several times did not solve the problem. The vertical motors were then moved to 0 position. DOC was called and helped to overwrite some parameters that triggered the error.
When resuming the calibration routine, the beam suddenly got lost and the vacuum interlock triggered V0 to close. Vacuum in M1 recovered quickly, but on M2 popIn, only the direct beam could be seen. At this point, the conclusion was that one of the motor connections must have come loose, which caused the vacuum event and removed M1 from the beam.

It was then decided, that we break the interlock, vent the chamber and re-connect the motor if possible.

The situation we found was the following:
-Rz connection lead-screw to bellow was detached
-mirror sitting on the mirror holder
-significant InGa spillage on chamber bottom, window flanges opposite to the mirror AND a lot of drops on the mirror surface

Pictures are attached.

Intervention:
The lead-screw and nut were exchanged. There was significant wear on the bearing plate, connecting the lead-screw to the bellow.
When trying to install the new lead-screw with a spare bearing plate, it was found that apparently the thread on the bellow side was deformed as well, and the spare part did not fit to the thread. Since the old bearing plate was the only possible way to connect the lead-screw to the bellow, it was cleaned, lubricated and re-used. The position was manually adjusted around 0 position (66,7mm protrusion) and the motor connected.

Recommendations:
- since the damaged bearing plate was re-installed, it should be avoided to move Rz. The motor should be permanently interlocked and only operated with absolute care for initial alignment.
- the mirror surface is severely contaminated with InGa, it must be checked if a position with acceptable beam can be found (only to be done by XRO as this needs Rz to be moved)
- the affected bellow has to be exchanged to facilitate replacement of the damaged bearing plate. As this requires the mirror chamber to be opened, this only makes sense in connection with replacement of the mirror + cleaning of the chamber.

We fine tuned geometry parameters for linear effects on X motion (parallel offset) for all three mirrors.

dTz was corrected for M2 and M3 to -6.5mm (same geometry)

dTz was corrected for M1 to +6mm

for M2 we were able to do a parabola scan Y motion to check for lrTz

lrTz was then corrected to -433mm

correction in the same direction for M3 did not yield better results

Final mirror positions at HED - see picture

RC / BKR informs : Beam is down (since 12h10), all beamlines affected

The reason is failure of a quadrupole (power supply) downstream of the dogleg.
The MPS crew is working on recovery, access to XTL ongoing.
RC will inform when new estimates come in on when we can restart beam operation.

Currently affected on Photon side:
- SA1: Naresh (XPD) and Marco (DET) working on Gotthard-II commissioning. Naresh is present at BKR.
- SA2/SA3: Antje + Mikako (XRO) working on SA2 mirror flipping tests and SA3 exit slit check. Both are located at SXP control hutch.
Other XRO members will come in for the afternoon/evening shift work. Antje will inform them via chat.

At 21:29:20 today, the number of bunches in SASE1 was increased from 1 to 352 without intervention of the SPB/SFX shift crew. 30 seconds later, the number of bunches was reduced to 1, again without intervention from the SPB/SFX shift crew (see attachments 1 and 2).

We were in the middle of a measurement at that time and the sudden increase from 10 mW to 3.5 W average power resulted in the destruction of the sample we were using.

AGIPD and JUNGFRAU are safe and were not at risk at the time of the incident, but this is merely by luck.

The origin of the change in number of bunches is unclear to us and needs to be investigated. It goes without saying that this is extremely dangerous for equipment and the damages could have been even worse.

Like after the previous winter maintenance period, the shutter of FXE appears as opened in DOOCS panels even though it is closed (https://in.xfel.eu/elog/OPERATION-2023/58).

The cross on the FEL imager, which we use as a reference for BKR crew, is not centred on the GATT aperture. when was this changed, and why?

att. 1 and 2 show the current position (the saturated image aids the positioning of the GATT aperture).

att. 3 is from two weeks ago. then, the cross was centred on the GATT aperture.

The status of the XTD2 attenuators has been checked during the experiment last week and many were found to cause disturbances in the wavefront ranging from mild to severe.

Attachment 1: No attenuator
Attachment 2: 75 um CVD - Fringes visible in beam
Attachment 3: 150 um CVD - Strong fringes in beam
Attachment 4: 300 um CVD - Looks more or less ok
Attachment 5: 600 um CVD - Weak distortion in the beam profile
Attachment 6: 1200 um CVD - Ok
Attachment 7: 2400 um CVD - Ok
Attachment 8: 500 um Si - Not ok

Not sure if this is the cause of the fringes reported before, but at 5.6 keV there are clear cracks visible on the XTD2 Attenuators 75 umm and 150um CVD, see attached. There is also a little bit of damage on the 600 um CVD, but not as strong as the others.

At 5.6 keV we will need to use the thin attenuators. I will mention again in case this causes problems downstream.

Edit 8:50h: We do not observe additional interference effects in the downstream beam profile (see attachement 4)

The number of bunches during the last night shift kept being changed down to 24 by the device SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020 acting on SA1_XTD2_BUNCHPATTERN/MDL/CONFIGURATOR. This was not the case during the previous shifts, even though the pulse energy was comparable. Looking at the history of SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020, it looks like this device was started at 16:26:25 on Friday, 1st March.

SASE1 bunch number was reduced to 10 pulses for no obvious reason. 

Traced it back to a malfunctioning of the watchdog for the solid attenuator (see attached error message)

Deacttivated the watchdog for SASE1 solid attenuator. Re-activated it again.

Issue: SASE1 (SPB?) was requesting 200 pulses at 6 keV with full power with all solid attenuators in the beam. This was a potentailly dangerous situation for the solid attnenuator (!)

After that the team left to go home. Nobody is available at FXE.  Set pulse number to 1.

Checked status of solid attenuator. Seems that plate 1&2 have some vertical cracks (see attached picture)

There are now two watchdogs for the XTD2 attenuator running at the same time

• SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020
• SA1_XTD2_OPTICS_PROTECTION/MDL/ATT1

that are configured differently, which results in different pulse limits (see attachment).

11:15 SASE1/SPB called that they observed and issue with the SASE1 XGM/XTD2: about every 35 seconds the power reading jumps from 4 Watts to zero and back. It seems to be related to some automated re-scaling of the Keithlyes, which coincides. The fast signal and downstream XGMs do not show this behavior. The concern is that this may affect the the data collection later today. 

12:15 The XGM expert was informed via email, however, there is no regular on-call service today. If this cannot be fixed in the next few hours, the PRC advises to use in addition the reading of the downstream XGM for data normalisation. 

13:15 Theo fixed the problem by disabling auto-range. Needs follow-up on Monday, but for this weekend should be ok. 

BKR called PRC around 06:10AM to inform on vacuum issues at SA1 beamline leading to a lost beam permission for the North Branch.

VAC-OCD found an ion pump in the PBLM / XTD9 XGM presented a spike in pressure that triggered valves in the section to close.

After reopening valves the beam permission was restored; further investigations and monitoring in coming days will continue. Currently all systems work normally.

https://in.xfel.eu/elog/Vacuum-OCD/252

Pulse energy fluctuation observed after full train operation in SA2 starts.

DOOCS panels are again incorrectly reporting the FXE shutter as opened when it is in fact closed. This had been fixed after it was last reported in February (elog:24), but reappeared now.

Continuous pointing drift in horizontal direction.

It may trigger a vacuum spike at SPB OH (MKB_PSLIT section) at 21:07 pm.

840 pulses not possible at SXP due to big brother settings for the EPS Power Limit rule. The minimum pulse energy was assumed to be 6 mJ while the actual pulse XGM reading is 550 uJ.
Solution: The minimum assuemd pulse energy had been lowered from 6 mJ to 2 mJ. This shall be reverted on Tuesday next week.

Dear all,

Due to a probable issue at the SA1 Transmissive imager, the "OUT" and "YAG" position indication signals seems to be not responsive.

After the FXE team noticed that MODE S was interlocked, the root cause has been narrowed down to this component. Later, further investigation with A. Koch & J. Gruenert (XPD) has confirmed this fact. Apparently to recabling works performed during yesterday's ZZ may have something to do with this issue.

Once that the complete retraction of all the screen has been jointly confirmed via DOOCS and Karabo, the affected interlock condition (see attached picture) has been disabled for the rest of the week to allow normal operation of SA1 (release of MODE S and MODE M).

Further investigation and the eventual issue resolution is expected to happen on Tuesday next week via ZZ-access.

With best regards,

Raul [PRC@CW19]

our pp laser experienced timing jumps by 18 ns today. this happened for the first time at ~7:30 in the morning without us noticing and a couple of more times during the morning. we realized it at 15:30 due to the fact that the signal on our pulse arrival monitor (X-ray/laser cross correlator downstream of the experiment) disappeared. we spent some time to investigate it, until we also see this jump in two different photodiodes different triggers (oscilloscope and digitizer) so it was clear that it is indeed the laser timing jumping. we also see the laser spectrum and intensity change for the trains that are offset in timing.

we called laser OCD at ~16:30. they have been trying to fix the problem by readjusting the instability zone multiple times and by adjusting the pockels cell, but this did not change the timing jumps. at 21:30 they concluded that there was nothing more they could do tonight and a meeting should happen in the morning to discuss how one can proceed.

effectively, this means that we lost the entire late and night shifts today for the pump-probe beamtime and have some data of questionable quality from the early shift as well.

EPS issue for Transmissive imager SA1 (OTRC.2615.T9) w.r.t Operation eLog #96

The motor screen positions w.r.t. the EPS switches had been realigned (last Wednesday 8th May), tested today (14th May) and the EPS is again enabled for the different beam modes. All is back to normal.