SASE1 blocked to 1 bunch/train by EPS interlock (sic, not 2 but 1) --- PARTIALLY RESOLVED

Several issues:

1) MODEM device is broken, in state UNKNOWN, see ATT#1 and ATT#2. Fixed with deinstatiating and then using BECKHOFF_ASSISTANT to re-instantiate. --> ATT#3

2) Several devices in XTD9 are causing that MODES is interlocked: I restart broken devices of the FXE-PBLM and SPB-PBLM.
Then there is a broken MDL of SPB_XTD9_SCR called SPB_XTD9_SCR/MDL/BIU. This MDL is not in the SA1-topic system topology but shown with (X) on the SA1_MAIN scene, and this screen is blocking the interlock (see ATT#4)

SPB_XTD9_SCR/VSWITCH/HOLSTER_X_LIMITL is triggering the interlock SA1_SYS_EPS2MPS/DCTRL/MODES and causing that the machine cannot go to more than 2 bunches per train.
Even when going to SPB-topic this issue cannot be solved - all devices in SPB-topic related to this SCR seem fine.

3) To allow SASE1 tuning by the machine, BKR is closing shutter XS3. This avoids any beam into XTD9, resolved the interlock problem and allows the machine to continue SASE tuning.

4) We don't know what had blocked the beam not only to 2 but actually to 1 bunch / train (no traces of problems in Big Brother).

Goal: Analyze and quantify the mispositioning of the new 4mm-aperture behind the undulator

Positions in tunnel:

• ABSP4.2422.SA2 = ABSP4.CELL37.SA2 L_z position 2422.558m
• Last closed undulator for this test is cell16 at 2294m (at end of segment) = SR radiation source point
• IMGTR = 2576m (BT=181.1m, PD=582.0m).
• IMGSR = 2586m (BT=191.6m, PD=591.5m).
• IMGFEL = 2637m (BT=242.0m, PD=642,8m).
• Sequence: end-of-cell16 -- ABSP4 -

Measurements:

• IMGTR : dX=15.15-8.25=6.9mm. dY=17-9.8=7.2mm. Aperture shadow mean diameter = 7.1mm
• IMGSR : Aperture shadow mean diameter = ca. 8.5 mm
• IMGFEL : Aperture shadow mean diameter = 9.0 mm.
  The handover cross is at dy = +1.0mm (up) and dx = -2.6mm (left) relative to the apparent center of the aperture shadow.

Comments to the screenshots:

• The handover cross on the IMGFEL cannot be seen simultaneously with the weak SR background because it has the color scaled to high intensity beam.
• The FEL beam spot is almost perfectly centered on the handover cross.

Calculations:

Conclusions:

The 4mm aperture is misplaced by 0.97mm (x) and 0.37mm (y).
It should be moved dx = -0.97mm and dy = 0.37mm, or the FEL has to be made on a different trajectory.

Problem: Beam operation in SASE2 is prohibited by SEPS / MPS
Origin of problem: HED-CRL3
(this was already an issue during the past weekend)

HED (Karen) today realigned CRL3 and brought the lenses into the safe OUT state, see entry https://in.xfel.eu/elog/OPERATION-2023/2
In karabo then everything seemed find, SEPS switches all green / OUT, see ATT#1.
Nevertheless, in DOOCS the signal safety_HED_CRL_OPT was still red, see ATT#2.

The MPS safety_HED_CRL_OPT signal from the Beckhoff HED PLC was missing.
Onsite investigation by EEE (Leandro and Bernard) revealed that the Beckhoff crate BKF000476 was offline (switched off).
Switching the crate ON fixed the missing signal problem.

Presumably crate BKF00476 was not powered back on since after the planned power-cut in December 2022,
and because it is not part of PLC EtherCAT loop, the Nagios system didn't inform EEE about the crate status.

Result: Problem solved, beam operation in SASE2 can resume normally.

CRL3 arm 2, 5 and 9 were mecanically religned to the V groove and retracted to some y position.

The y motor position was recalibrated by moving the motor to the lower limit switch, moving it up by 0.5 mm and then recalibrating the motor and encoder position to the former software limit position.

In that down position, the x range was tested by moving into the limit switches:

arm 2 x -1.213 to 0.824

arm 5 x -1.22 to 1.73

arm 9 x -0.32 to 2.256

The x limit switches were not changed and also the motor x axis were not recalibrated.

After that the x and y motors were moved to the x and y out position as stored in the CRL3 MDL. In that position, it was confirmed that all SEPS interlocks are triggered.

CRL3 is thus set-up for the SEPS testing, that will take place on the 24th of January.

I started the upstream XGMs in XTD2, XTD1, and XTD10.

beam: 30 keV / 30 uJ
reduce to single bunch for insertion
put FEL imager in
put DD into beam at X=178 mm
we found the signal
centering the DD by minimising the diffrence signal

Bias voltage = 40 V

when moving Y there seems to be some mechanical bending of the detector holder. The position reached the negative end of the range (approx. -9 mm).
When moving out and in again we could find the beam at:
X= 178.629 (centred)
Y= -3.48 (close to center)
see screenshot

run 29 with DD and IMGFEL
 

The aperture was opened by moving the slits from 2x2 to 4x4 mm

switch to 2 bunches

screenshot

try to center Y position

stop run 29

run 30: approx. 5 min at this position

change FEL-imager ND filter to 0.9 to avoid saturation
run31

take IMGFEL out and switch to 50 bunches
actual X center = 178.44

move left until we loose the signal and scan x from 176.5 to 179.9
y signal changes -> mechanical coupling
run 32

in order to center the Y position we had to move 2 mm down to -5.53

run 33:
move left until we loose the signal and scan x from 176.5 to 179.9
y signal changes

Now the center of y seems to be at -2.88 mm

Image was observed with FEL YAG imager, x movement does not result in a horizontal motion but it is some sort of a diagonal motion

After some movements the y center is now at -5.23 mm

After a new try the y center is at -3.37 mm

Due to the mechanical problem with the Y-position we do the scan with the K-mono chamber X-motor

-----
see elogs from Jan and Tuba

-----

Sascha made an orbit kick with 150 bunches
due to some numerical problem the kick is disturbed (see screenshot)
DD can reproduce the kick trace
take run 40

Sascha altered the orbit again 150 bunches
https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2022/47/26.11_a&pos=2022-11-26T17:20:00
(see attached graph)
take run 41

set bias from 40 V to 100 V

run 42

Bias set to 40 V again

beam changed to 27 keV

x-calibration scan from 8.3 to 11.7 (K-mono X)
run 43

Bias -> 100 V
run 44 (as before)

Bias -> 20 V
run 45

Suspicion: beam is touching K-mono setup
move K-mono chamber to X=4.6 mm and move DDM X to 173.55
Y again has to be adjusted, is now -3.18

bias set to 40 V

scan K-mono X from 3.1 to 6.1
run 46

bias runs: see Tuba's elog