- Due to the failure of the cryo cold compressor. The machine is down.

- Next update 9 pm today

Please do watch public display for status info.

Info from Machine RC (Lars):

----------------------------------------------------------------------------------------------------------------------------------------

Author: Froehlich

One of the cold compressors has stopped working around 2:00 on Saturday morning due to a problem with the new motor (with magnet bearing). Since no beam operation is possible anyhow, the cryo team and machine coordination have decided to start necessary work on cryo plant stages 3&4 already on the weekend, in parallel to troubleshooting the motor problem.

All XFEL shifts up to Monday 7:00 are canceled. The morning shift on Monday will supervise the already scheduled access for the repair of A16 and, if possible, the restart of the linac.

This elogbook entry was sent to following experts:
Photon RC mailing list RC mailing list xfel-operation-board@desy.de

------------------------------------------------------------------------

Direct link to e-logbook entry:
https://ttfinfo.desy.de/XFELelog-sec/show.jsp?dir=/2023/36/09.09_M&pos=2023-09-09T14:51:07

------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------

SASE3 gas supply alarm

SCS called the PRC at 17h04 due to a loud audible alarm in XHEXP1 and contacted also TS-OCD.
It turned out to be the gas supply system for SASE3, an electrical cabinet on the balcony level in the SE-corner of XHEXP1.

Actions taken by PRC:
a) contacted TS-OCD (Christian Holz, Alexander Kondraschew)
b) terminated the audible alarm
c) source for the alarm was: "Stroemungssensor -BF15.3+GSS3"
d) from this point on, TS is taking over

1) The instruments need to receive a number to call to reach the hall crew.
2) The alarm was likely triggered by the SCS team since there was a problem with the Krypton gas supply to their XGM around the same time.

Update of power outage aftermath

Achieved by now / status

• SASE1
  • ​beam delivery and FXE operating fine (beam back since about 11h30)
• SASE3
  • SA3 getting beam since about 11h30, again difficulties / no beam between 12h30 - 13h34, currently ok.
  • SASE3 Optical PP laser issues since the morning. DOC support was required by LAS-OCD (task completed by DOC at 14h30)
  • LAS-OCD working on SA3 PP laser and estimate to have it operational around 16h.
• SASE2
  • the inoperational MTCA crate for MPS was successfully restarted and AIBS alarms cancelled in PLC
  • beam permission was recovered and XFEL is lasing in XTD1 until shutter
  • karabo of SA2 is DOWN, DOC working on it (see below)

Main remaining issues:

1. Complete restart of karabo for SASE2 is required. This is a long operation, duration several hours. DOC working on it.
2. SASE2 Online-GPFS is down, probably due to lack of cooling. Possible hardware damage, ITDM assessing and working on it.
3. SASE3 optical PP laser to be brought back

8h05 Beam is switched back on in SASE2. HED ready for taking beam.

8h45 BKR is having problems to turn on SA2 beam. Not due to any 

Beam actually went down in all beamlines today at 7:02 (for the records).

9h00: Electricity comes back to SASE3.
We can see SCS-XGM crate coming back on (that's how we can tell that power is back).

9h15 BKR struggles with SA2

9h20: Info from TS-OCD: Problem with USV at SASE3, all needs to be switched 

9h30 BKR is working on making beam in SA1/3. SA1 beam on IMGFEL, switching to FXE conditions (5.6 keV).

9h08 FXE info:
D3 has reset the alarms for SA1 beamstop monitor 9.1 (burnthrough monitor) and SA3. FXE is ready to receive beam.

9h04 EEE-OCD informs:
Power has been restored to the LA3 PPL PLC

9h08 DOC info:
restoring settings for LA3 PPL as of before today's power cut.

Machine went down overnight (4AM and 6AM) and could not be properly restarted by BKR.

Ongoing tuning is proceeding with RC onsite since 6AM; Instruments are informed and in direct contact to BKR.

Status 09:45AM: SA1 ~3mJ; SA2: ~500uJ; SA3 being restarted

Status 11:10AM: Beam restored in all SASEs; SA1 delivery from 10AM; SA2 delivery from 11:10; SA3 delivery from 10:45

At the end of the MON late shift, the tuning status and obtained lasing pulse energies (ATT#1) are as follows:

• SA1 at 19 keV : 660 uJ +/- 23 uJ
• SA2 at 18 keV : 1073 uJ +/- 55 uJ
• SA3 at 1.2 keV : 4330 uJ +/- 122 uJ

SD for a stable 2 min interval, fast XGM train average (INTENSITY.RAW.TRAIN)

More tuning will happen overnight, and for SA3 the photon energies 2.5keV and 3.0 keV will also be prepared.

Typical delivery status during daytime during the week:

• SA1 at 19 keV : 841 uJ +/- 41 uJ
• SA2 at 18 keV : 990 uJ +/- 80 uJ
• SA3 at 3.0 keV : 2900 uJ +/- 138 uJ

SD for a stable 2 hours long interval, fast XGM train average (INTENSITY.RAW.TRAIN)

Finally, we're completely back in operation,
and the performance is recovered in all beamlines.

See also ttfinfo, e.g. https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2023/44/05.11_a&pos=2023-11-05T16:01:25

Ramp down activities completed by XPD. References:

• SASE1
  • https://in.xfel.eu/elog/SASE1+technical+commissioning/2204
  • https://in.xfel.eu/elog/SASE1+technical+commissioning/2205
  • https://in.xfel.eu/elog/SASE1+technical+commissioning/2206
• SASE2
  • https://in.xfel.eu/elog/SASE2+technical+commissioning/1206
  • ...until...
  • https://in.xfel.eu/elog/SASE2+technical+commissioning/1209
• SASE3
  • https://in.xfel.eu/elog/SASE3+technical+commissioning/2220
  • https://in.xfel.eu/elog/SASE3+technical+commissioning/2221

The XFEL photon beam was switched off as planned at 19h00 sharp.

Machine is preparing for opening the photon tunnels tomorrow morning and to start the winter maintenance period.

Instability observed today at 14:05, reported to PRC and BKR, including RC: We saw RMS jitter amplitude of up to 4 mm at FXE imager. The jitter was also seen by the upstream XGM with 100 um RMS amplitude. So it is something from the source.

This instability is suddenly appearind/disappearing and was absent over night and started this morning at 7h to occasionally turn on. It appear approx. everz ~30 mins for a few minutes.

HED would like to report that since 2 days the SASE2 beam is instable.
The attached movie shows, on the left (90° rotated, so up-down is left-right)
the beam at the end of the HED hutch, and on the right its the same beam
at the end of the XTD6 tunnel (HED-POPIN diamond window).

It jitters horizantally by more than 1.5 diameters.

We are taking data for users, but due to the jitter, only every 4th image is usable,
which mean also 3 out of 4 targets are lost.

We are haveing near collimated beam with CRL1 (XTD1) 1,6 and CRL2 (XTD6) 2,3)
to generate a large field of view on the target.

Ulf

the pulse energy in XTD10 is lowered by ~30% when SASE1 changes photon energy, compare att. 1 and 2.

In addition, the intra-train pulse energy stability gets much lower. Potentially, there is an impact on the pointing as well, which is critical for the pump-probe experiment.

changing SA3 mono from grating to blank while having the und to mono synchronization running apparently caused the und photon energy to change to 680eV. the synchronization MDL went into pink-disable when changing mirror settings, so this change in photon energy should not have occurred.

After going back to 940eV, the SASE level did not come back. BKR tried to tackle this for some while until they asked for intervention of the MRC.

On 27th of July, it was reported that arm 5 of the solid attenuator did not stay in 'out' position when retracted, but moved in immediately again.

Reason was, that the limit switch for the 'out'-position did not trigger, and thus the actuator was in error state. Default for error state is 'in'. In order to keep the arm removed, the default was changed on PLC level for the time being.

Today, 30th of July, we did ZZ to investigate the issue. It was found that the bar that accommodates the upper and lower limit switches had become loose and the limit switches were pressed out of the way by the motion, rather than being triggered.

The reason for the bar becoming loose, probably is a not ideal alignment of pneumatic actuator, guiding rod and limit switch bar, which resulted in a small relative motion between the three. This was fixed and limit switches were again aligned and trigger in both 'in' and 'out' position.

We believe, this will remain stable until WMP, but should be investigated more thoroughly, as we could not reach very well all connections in situ.

Summary of today's activity:

• Beam handover ~ 3pm.
• SA2 mirror set tp B4C.
• SA2 Rough alignment to MID & HED @ ~ 2-2.5 mrad ==> img 1, 4, 5 & 6
• SA2 HED-Preabsorber position check with x-ray (working) ==> img 2 &3
• SA3 M6 mirror Switch check (seems to be working)
• SA1 rough alignment to SPB (M1,M2) at 2 mrad ==>img 7&8

Activities:

• Found coarse and fine timing between FEL and PPL at 266nm
• Set up for tr-RIXS during the night

Problems:

• 2 fastADC channels appear to be dead.
• Optical locking of the PPL syncronisation system took a very long time (> 30 minutes)

Achievements:
- Optimized hRIXS settings for OK edge at 145 deg scattering angle, resolution of about 50 meV
- Transferred the user sample
- Prepared for delay scans on fluorescence signal from YBCO

Problems:
- There was no internet connection at XFEL and DESY for some time
- hRIXS camera goes frequently into error
- XRD motors (STY) go frequently into error

Next steps:
- Find time zero on fluoresce signal from YBCO, test the same for the samples
- Test PP laser damage threshold

Mirror pitch motors Ry were tested after scaling was changed to um.

Tx and Ry motor positions were calibrated to their measured physical positions.

ALL PREVIOUSLY SAVED SETTINGS FOR SASE2 MIRRORS ARE OBSOLETE!

New mirror positions for 9.3keV ; 2.2mrad ; B4C stripe are stored in MID-setpoints and HED setpoints

Please check the respective ELOG entries from July 20th

Beam aligned to SPB Pop-In https://in.xfel.eu/elog/XRO+Beam+Commissioning/185

Beam aligned to FXE Pop-In https://in.xfel.eu/elog/XRO+Beam+Commissioning/187

Vibration tests: https://in.xfel.eu/elog/XRO+Beam+Commissioning/188

Camera of M2 may not be working. Please check.