- We get stable x-ray delivery with 130µJ.

- We could move the detector succesfully in and out of the LFoV configuration!

- We added a hole in in the black Kapton foil and increased the holes in the Kapton shielding of the jet. These were not positioned well, due to a misalignment of the ALAS. After this intervention we did not observe any scattering rings on the detector.

- We aligned the liquid jet and get a nice diffraction rings from water. 

We had stable x-ray delivery with 120µJ and up to 300 pulses.

Successful user data acquisition of water and water+THF samples at various supercooled states

Stable beam delivery with 23keV and 140µJ at 2.2, 1.1 and 0.55MHz

User data acquisition during the day interrupted by storage and detector DAQ problems:

- problem with detector DAQ, which is correlated to the full SSD cache.

- we run out of storage, this problem was solved by the DAQ problems. We lost 6h of measurement time today, due to storage limitations. 

- 2 detector modules stopped sending data

Problem: Beam operation in SASE2 is prohibited by SEPS / MPS
Origin of problem: HED-CRL3
(this was already an issue during the past weekend)

HED (Karen) today realigned CRL3 and brought the lenses into the safe OUT state, see entry https://in.xfel.eu/elog/OPERATION-2023/2
In karabo then everything seemed find, SEPS switches all green / OUT, see ATT#1.
Nevertheless, in DOOCS the signal safety_HED_CRL_OPT was still red, see ATT#2.

The MPS safety_HED_CRL_OPT signal from the Beckhoff HED PLC was missing.
Onsite investigation by EEE (Leandro and Bernard) revealed that the Beckhoff crate BKF000476 was offline (switched off).
Switching the crate ON fixed the missing signal problem.

Presumably crate BKF00476 was not powered back on since after the planned power-cut in December 2022,
and because it is not part of PLC EtherCAT loop, the Nagios system didn't inform EEE about the crate status.

Result: Problem solved, beam operation in SASE2 can resume normally.

Goal: Analyze and quantify the mispositioning of the new 4mm-aperture behind the undulator

Positions in tunnel:

• ABSP4.2422.SA2 = ABSP4.CELL37.SA2 L_z position 2422.558m
• Last closed undulator for this test is cell16 at 2294m (at end of segment) = SR radiation source point
• IMGTR = 2576m (BT=181.1m, PD=582.0m).
• IMGSR = 2586m (BT=191.6m, PD=591.5m).
• IMGFEL = 2637m (BT=242.0m, PD=642,8m).
• Sequence: end-of-cell16 -- ABSP4 -

Measurements:

• IMGTR : dX=15.15-8.25=6.9mm. dY=17-9.8=7.2mm. Aperture shadow mean diameter = 7.1mm
• IMGSR : Aperture shadow mean diameter = ca. 8.5 mm
• IMGFEL : Aperture shadow mean diameter = 9.0 mm.
  The handover cross is at dy = +1.0mm (up) and dx = -2.6mm (left) relative to the apparent center of the aperture shadow.

Comments to the screenshots:

• The handover cross on the IMGFEL cannot be seen simultaneously with the weak SR background because it has the color scaled to high intensity beam.
• The FEL beam spot is almost perfectly centered on the handover cross.

Calculations:

Conclusions:

The 4mm aperture is misplaced by 0.97mm (x) and 0.37mm (y).
It should be moved dx = -0.97mm and dy = 0.37mm, or the FEL has to be made on a different trajectory.

SASE1 blocked to 1 bunch/train by EPS interlock (sic, not 2 but 1) --- PARTIALLY RESOLVED

Several issues:

1) MODEM device is broken, in state UNKNOWN, see ATT#1 and ATT#2. Fixed with deinstatiating and then using BECKHOFF_ASSISTANT to re-instantiate. --> ATT#3

2) Several devices in XTD9 are causing that MODES is interlocked: I restart broken devices of the FXE-PBLM and SPB-PBLM.
Then there is a broken MDL of SPB_XTD9_SCR called SPB_XTD9_SCR/MDL/BIU. This MDL is not in the SA1-topic system topology but shown with (X) on the SA1_MAIN scene, and this screen is blocking the interlock (see ATT#4)

SPB_XTD9_SCR/VSWITCH/HOLSTER_X_LIMITL is triggering the interlock SA1_SYS_EPS2MPS/DCTRL/MODES and causing that the machine cannot go to more than 2 bunches per train.
Even when going to SPB-topic this issue cannot be solved - all devices in SPB-topic related to this SCR seem fine.

3) To allow SASE1 tuning by the machine, BKR is closing shutter XS3. This avoids any beam into XTD9, resolved the interlock problem and allows the machine to continue SASE tuning.

4) We don't know what had blocked the beam not only to 2 but actually to 1 bunch / train (no traces of problems in Big Brother).

Status of machine SASE tuning (just before switching off for monday maintenance):

REPAIRED / FIXED

Problem: The imager SPB_XTD9_IMGPII45 insertion motor lost steps and kept the machine to maximum 2 bunches per train.
Mitigation by DOC/EEE/PRC on 8 April 2023: made sure that screen seems actually out of beam and took imager out of interlock condition.
"Repair" by XPD on 10 April 2023 02h15:
a) recalibrated OUT position.
b) verified that EPS-OUT switch shows green when moving to OUT position in MDL.
c) put back the imager into the interlock conditions.
d) communication with BKR before and after the intervention (SPB and FXE didn't use the beam).
Repair instruction see: https://in.xfel.eu/elog/SASE1+technical+commissioning/2065

REPAIRED / FIXED

Problem: The imager SPB_XTD9_IMGPII45 insertion motor lost steps and kept the machine to maximum 2 bunches per train.
Mitigation by DOC/EEE/PRC on 8 April 2023: made sure that screen seems actually out of beam and took imager out of interlock condition.
Repair by XPD on 10 April 2023 02h15: 
a) recalibrated OUT position.
b) verified that EPS-OUT switch shows green when moving to OUT position in MDL.
c) put back the imager into the interlock conditions.
d) communication with BKR before and after the intervention (SPB and FXE didn't use the beam).
Repair instruction see: https://in.xfel.eu/elog/SASE1+technical+commissioning/2065

End of Diamond Sensor Study

We leave the SASE2 beamline in the following safe state:

• Pre-Abs inserted
• Solid ATT with T=0.001
• All CRLs OUT
• All imagers OUT

The machine tuning over the night had not yet accomplished fully satisfactory beam conditions mainly at the high photon energies for SA1 and SA2,
therefore tuning continued during WED morning.
Beam was handed over to SCS (SASE3) at 9h with about 2.4mJ at 2.5 keV.
Beam was handed over to FXE (SASE1) at 13h with about 530uJ at 19.5 keV.
Tuning for MID (SASE2) at 23 keV is agreed to continue until 14h, currently at a level of 120 uJ. 

SASE1 status:

First shift switchover this week at 23h from FXE (19.5 keV) to SPB/SFX (12.4 keV) with issues. 
No immediate lasing at 12.4 keV with standard procedures / files, then tuneup using FEL imager (CBor) in multibunch. Wrong pointing (far off up in vertical).
Recovered lasing by 23h55 with 1 mJ at 12.4 keV at 10 bunches. Handover to SPB by midnight.

Diagnostics:
XTD2-XGM stayed with HAMPv during the photon energy change, which turned out fine, but the signal would be non-linear in multibunch
operation with many bunches, thus switch to XGMDh (HAMPv power remains turned on).
Procedure practiced with BKR-crew. BKR will switch back to HAMPv when switching back to FXE / 20 keV.

MID reports an observation that is particular to the operation at the currently high photon energy of 23 keV

MID inserted the XTD6-MID-preabsorber and assumed that it completely stops the beam.
They acquired AGIPD detector images as dark frames but were surprised to still find some weak background signal (not dark).
The explanation by PRC and XRO is that the purpose of the pre-absorber is to protect the shutters against potential
damage and not to entirely stop the beam like a radiation protection shutter itself.
The pre-absorbers contain diamond and B4C but no tungsten. The calculated transmission of the pre-absorber at 25 keV is 4%.
3rd harmonic of the FEL (here at 69 keV) is on the order of 1%, and MID uses the mirror coating
stripes for high photon energies, thus transmitting well these high photon energies.

As another result, the apparent beam size observed on the downstream imagers can be more complex to understand:
MID actually also reports that the beam size appeared larger in the afternoon than in the morning,
depending on the insertion status of the pre-absorber and the lenses.
The scintillation efficiency of the YAG screens is known to be high also for hard x-rays beyond 30 keV.
Additionally, the CRLs focus the various harmonics differently. When CRL1 is set to focus the beam,
it will likely defocus higher harmonics yielding a larger spot size on a downstream imager.

PRC informed XRO to follow up together.

Info from XRO (Maurizio Vannoni) to PRC concerning tunnel access request #152936 (Inspection and repair of SQS ESLIT)

- The ZZ-access will be done by Marc Planas, Massimiliano Di Felice, and Idoia Freijo.

- They will start their access between 9h00 and 9h30 and end their access by 12h30.

https://redmine.xfel.eu/issues/152936

UPDATE 11h45: ZZ-access is completed, beam is back in operation.

SASE2 vacuum incident

7h10 : MID informs PRC about a vacuum incident in the SASE2 tunnel which tripped the beam. Instructed to contact directly VAC-OCD

8h26 : VAC-OCD informs that investigation is ongoing, but beam can be delivered. This kind of trip usually occured rarely
           and usually can be reset by VAC. Apparently the source of the issue is at the XTD1 solid attenuators.

8h36 : MID confirms to PRC that already 3 trips occcured but the experiment so far can continue. Beam delivery continues.

10h11 : VAC-OCD informs that now there is a bigger more wide-spread issue with the XTD1 vacuum system
           and beam operation is stopped for at least 1 hour. VAC group and DOC are working on the issue.

11h : It turns out that the vacuum bellows of the 2.4mm diamond attenuator in SA2_XTD1_ATT has a leak, which is worse and
         not acceptable in terms of vaccuum in the extracted state (bellows elongated) but less severe in the inserted state (bellows compressed).

12h33 : info by VAC: issue is temporarily solved by blocking the 2.4mm diamond arm in the IN position. Must clarify today how to proceed further.
          The vacuum issue thus is temporarily solved, beam permission is back,

12h38 : Beam coming back to MID. MID can work today with this ATT always inserted, but not for the weekend.

SA2_XTD1_ATT solid attenuator vacuum issue

Info from VAC (Raul Villanueva) 12h30:

It seems to be a leak in the arm nr. 6 (2.4mm CVD Diamond). This has escalated to trip several times the vacuum sector interlock.
On top of that, we also discover an issue with the configuration of a ion pump controller that have required the intervention of DOC / EEE. 

If the arm 6 is kept closed (2.4 mm CVD  is inserted), the leak rate seems to be small enough to allow operation (average sector pressure in the low 10-7 mbar). Otherwise it will blow consistently the interlock. (I repeat here: DO NOT MOVE THE 2.4 MM CVD!!!) See attachment #1. Most probably the leak would be in the custom-made bellow of that particular actuator. XRO just confirmed that there is no spares available.

Info by MID (12h30):
For today, MID can operate with the 2.4mm diamond inserted without impact to their experiment. However, tomorrow morning there will be a new user and experimental setup, which will require full transmission throughout the weekend, therefore the 2.4mm diamond attenuator must be removed friday. The change in experimental setup means that MID will work in the hutch at 10am for several hours and won't take beam anyhow.

Discussion of PRC+RC+XO+VAC+MID (15h30), with agreement to the following actions:

a) VAC prepares an intervention to XTD1 with broken interlock tomorrow morning startin 9am to remove arm 6 permanently and putting a blind flange instead. This intervention is estimated to start 9am and last for 4 to 5 hours. VAC will inform BKR if the work can be completed quicker.

b) Next week (last operation week in this run) therefore the Solid attenuator arm 6 won't be available. HED confirmed to PRC that it will be acceptable for their experiment since they plan to use full transmission in XTD2 and use their own attenuators in the HED hutch to attenuate the beam. XPD confirms that the XTD2 attenuators are not required for HIREX operation since HIREX contains its own attenuators between the bent crystal and the detector.

c) BKR/RC will organize the required tunnel search for the friday afternoon, and subsequently the HXRSS team will start optimization.

d) X-Ray Optics has already initiated the contact with the ATT device supplier to get replacement parts that will allow to recover full functionality during the upcoming (short) summer maintenance week.

Repair of SA2_XTD1_ATT

Status 12h30:
The current MID users took beam until 9am, only then the machine could start preparing for the tunnel access e.g. by decoupling the South branch, powering down magnets and taking XTD1 out of the interlock. The VAC team was ready to go into the tunnel for the repair in XTD1 already since 8am.
Unfortunately, the tunnel access of the VAC team got delayed beyond 9am by one more hour to 10am because the DACHS terminal at ground level of XS2 was found defective, and releasing the tunnel interlock took substantial time.
DESY experts went onsite and informed that a proper repair of the DACHS terminal can only happen next monday.
Only with a workaround, finally the VAC team was able to use the elevator to bring the required large venting and pumping equipment down to the tunnel level.
The vacuum works are ongoing with highest priority.

Status 14h20 :
VAC has finished the intervention, informed BKR and PRC, and is leaving the tunnel. Tunnel search team can start.

Status 15h00:
Tunnel search almost finished (also found some technical obstacles), Seeding experts are informed and ready to start their work.

18h05 : RC informs:

Machine is down, because Cryo-OK was lost.
Apparently it is a turbine but not the cold compressors.

It is thus not yet known how long the downtime will last.
More info will come within the next hour.

MID and FXE were informed by PRC, SCS could not be reached yet.

The Cryo team solved the problems, beam is back.
Downtime from 17h59 until 21h03.