The intra-bunchtrain feedback (IBFB) was not active until today afternoon.
Although not strictly required or requested for the current week experiments,
it was decided to turn on this feedback, which should generally be active to
stabilize and improve the arrival time of the photon pulses along pulse trains.

In order to avoid complications during long data acquisitions, the PRC checked back with all ongoing experiments,
identified a suitable time slot, and the IBFB was activated without issues at aound 16h45.

20.4.2024 / 22h51:
DOC informed PRC that the SASE2 topic broker had to be restarted.
The EPS power limit was reset by PRC to the previous value of 40 W.

DOC and DRC are working on the restart. HED is in hardware setup configuration change (not currently taking beam, thus no harm to experiment),
will start taking beam with new users sunday 7am, but would need before this some test runs for the detectors. 

Currently the SA2 topic watchdogs (for ATT, CRL, imagers) cannot be restarted - they crash quickly after restarting.
DOC/DRC are investigating this issue. Until the watchdogs are successfully restarted, extra care must be taken when using ATT, CRL, imagers in SA2.

Update 21.4.2024 / 1h43 by DRC:
In a similar fashion to the incident of last 5.04, the Broker process for the SA2 and LA2 topic has restarted.
This was noticed by the DOC crew and by the HED crew in parallel.
With the help of the CTRL OCD (Micheal Smith) we restarted both topics. As a lesson learned from the last interruption, the POWER_LIMIT
middlelayer device has been set up by the PRC, so that when the DOOCS bridges will be restarted by the DESY-MCS personnel,
the maximum power will not be set to 0.
Please check in due time if all systems are working as intended.

V0 valve for SA2 closed as the pump P22090I at HED branch enters in wrong state. This closed nearby vacuum valves for HED branch and consequently V0. Vacuum levels at the section remained in 10^-9 all time.

Thanks to VAC-OCD Denis Finze we removed this pump signal from interlock to allow operation in SA2 at MID branch. Further investigation will follow on Friday.

840 pulses not possible at SXP due to big brother settings for the EPS Power Limit rule. The minimum pulse energy was assumed to be 6 mJ while the actual pulse XGM reading is 550 uJ.
Solution: The minimum assuemd pulse energy had been lowered from 6 mJ to 2 mJ. This shall be reverted on Tuesday next week.

Dear all,

Due to a probable issue at the SA1 Transmissive imager, the "OUT" and "YAG" position indication signals seems to be not responsive.

After the FXE team noticed that MODE S was interlocked, the root cause has been narrowed down to this component. Later, further investigation with A. Koch & J. Gruenert (XPD) has confirmed this fact. Apparently to recabling works performed during yesterday's ZZ may have something to do with this issue.

Once that the complete retraction of all the screen has been jointly confirmed via DOOCS and Karabo, the affected interlock condition (see attached picture) has been disabled for the rest of the week to allow normal operation of SA1 (release of MODE S and MODE M).

Further investigation and the eventual issue resolution is expected to happen on Tuesday next week via ZZ-access.

With best regards,

Raul [PRC@CW19]

In SASE2 undualtor cell 10 has issues. Suren K. will do a ZZ at 11:30 to fix it (if possible). Estimated duration: 1-3 hours. 

Update 12:30: The problem is more severe than anticipated. Suren needs to get a replacement driver from the lab and then go back into the tunnel at about 14:00 

In principle, the beam could be use dby HED until 14:00

Update 13:30: Beam was not put back, because energizing and de-energizing the magnets would take too much time. Whenever Suren is ready, he will go back into XTD1 and try to fix the problem. 

Udate 17:30: Suren and Mikhail could do the repair and finish now th zz. A hardware piece (motor driver) had to be exchanged and it turned out that the Beckhoff software on it was outdated. To get the required feedback from Beckhoff took considerable time, but now everything looks good.  

The control rack of the soft mono at SASE3 has some failure. Leandro from EEE will enter the XTD10 at 12:00. 

Beam will be interupted for SAS1 and SASE3.

Estimate to fix it (if we are lucky) is one hour. 

Udate 12:30: Problem is solved (fuse was broken, replaced, motors tested)

Update 13:30: Beam is still not back, because BKR has problems with their sequencer

SASE1: Solid attenuator att5 is in error. When it is tried to move it out, the arm lifts, but does not reach the final position before the software time-out. We  increased the software time-out from 4 seconds to 7 seconds, which didn't help. We suspect that there is a mechanical problem that prevents the att5 actuator to reach the out position. 

SPB states that it is ok to leave the attenuator arm in over the weekend, because they run mostly detector tests and don't require full intensity.  

EPS Interlock change for Mode30:
the 3 HIREX Interlock conditions had been disabled

Issue with SA3 Gas attenuator (SA3_XTD10_GATT)

9h06: SQS informs PRC that the Argon line in GATT is not reacting on pressure change commands, no GATT operation, which is required to reduce pulse energies for alignment. VAC-OCD is informed.

ZZ access is required to check hardware in the tunnel XTD10.
SA1 / SPB is working on other problems in their hutch (water leaks) and don't take beam.
SA3 / SQS will work with the optical laser.
SA2 / MID will not be affected: there is also an issue in the hutch and MID currently doesn't take beam into the hutch, and would anyhow only very shortly be affected during the moment of decoupling the beamline branches.

9h45: BKR decouples North Branch (SA1/SA3) to enable ZZ accesss for the VAC colleagues.

ZZ access of VAC colleagues.

10h12: The problem is solved. A manual intervention in the tunnel was indeed necessary. VAC colleagues are leaving the tunnel.

• Beam down SASE1: 9h43 until 10h45 (1 hour no beam), tuning done at 11h45, then 1.7mJ at 6keV single bunch.
• Beam down SASE3: 9h43 until 10h56 (1.25 hours no beam). Before intervention 4.2mJ average with 200 bunches, afterwards 2.5mJ at 400 eV / 3nm single bunch.

Bunch pattern SA1 / SA3

SQS requests to change the bunch pattern.
Currently SA1/SA3 run interleaved as 131313...
The first bunch in the train (for SA1, non-lasing in SA3) nevertheless is visible as a small pre-peak in the diode signal which SQS uses to check the timing.
Therefore SQS requests to change to 31313131....

It was checked with SA1 (SPB/SFX) and the accelerator experts that this should not cause any issues (AGIPD detector is running on dynamic trigger),
therefore it was agreed that SPB/SFX informs BKR when there is a good moment without data acquisition, and BKR will make this change after informing SQS.
This change will happen within the next hour (as of 15h20).

A2 failure - downtime

The machine / RC informs (11h22) that there is an issue with the accelerator module A2 (first module in L1, right after the injector).
This causes a downtime of the accelerator of 1 hour (from 10h51 to 11h51), affecting all photon beamlines.

Beam is back. Current status:

SASE1 (SPB/SFX)

• 2.72 mJ, 352 bunches/train, 6 keV

SASE2 (MID)

• 880 uJ, 1 bunch/train, 8 keV

SASE3 (SQS)

• 4.56 mJ, 1 bunch/train, 400 eV (3.1nm)

14:40hr

FXE and HED reported the laser synchronization issue. Informed to Machine RC. 

LbSync team couldn't fixed this issue remotely and on there way to Schenefeld to fix this issue.

Naresh 

PRC wk 41

Desy Laser group has fixed the issue about the laser Synchronization.

https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2024/41/07.10_M&pos=2024-10-07T14:51:12

Naresh

PRC

wk41

- While FEL IMG with YAG screen was inserted in SA1, EPS/MPS was not limited to the expected 2 bunches, but to 30 bunches; 
- It was found the condition #3 ("SA1_XTD2_VAC/Switch/SA1_PNACT_Close") at the EPS Mode-S was disabled which should be always enabled; 
- Condition #3 has been enabled again and now SA1 is limited to 2 bunches, as expected when YAG screen is inserted. 

Further investigations will be done to find why and when it was disabled. 

SA2 cell#5 undulator segment moved at ~09:45AM Fri to 25mm without aparent reason and undulator experts need a ZZ to exchange power supply; FEL instensity dropped from 230uJ to 60uJ s consequence.

RCs are preparing south branch for the iminent ZZ. HED is informed.

**Update: ZZ finished by 12:10; recovered performance and re-started delivery by 13:30.

20h16: beam down in all beamlines due to a gun problem
20h50: x-ray beams are back
Shortly later additional 5 min down (20h55 - 21h00) for all beamlines, then the gun issue is solved.

After that, SA1 and SA3 are about back to same performance, but SA2 HXRSS is down from 130 uJ before to 100 uJ after the downtime.
Additionally, MID informs at 21h07 that no X-rays at all arrive to their hutch anymore. Investigation with PRC.

At 21h50, all issues are mostly settled, MID has beam, normal operation continues.
Further HXRSS tuning and preparation for limited range photon energy scanning will happen in the night.

Broken Bridges

Around 21h20 the XPD group notices that several karabo-to-DOOCS bridges are broken.
The immediate effect is that in DOOCS all XGMs in all beamlines show warnings
(because they don't receive important status informations from karabo anymore, for instance the insertion status of the graphite filters nrequired at this photon energy).

PRC contacts Tim Wilksen / DESY to repair these bridges after PRC check with DOC, who confirmed that on karabo side there were no issues.
Tim finds that ALL bridges are broken down. The root cause is a failing computer that hosts all these bridges, see his logbook entry;
https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2024/47/22.11_a&pos=2024-11-22T21:47:00
That also explains that HIREX data was no longer available in DOOCS for the accelerator team tuning SA2-HXRSS.

The issue was resolved by Tim and his colleagues at 21h56 and had persisted since 20h27 when that computer crashed (thus 1.5h downtime of the bridges).
Tim will organize with BKR to put a status indicator for this important computer hosting all bridges on a BKR diagnostic panel.

Broken Radiation Interlock

Beam was down for all beamlines due to a broken radiation interlock in the SPB/SFX experimental hutch.
The interlock was brocken accidently with the shutter open, see https://in.xfel.eu/elog/OPERATION-2024/203
Beam was down for 42 min (from 3h00 to 3h42) but the machine came back with same perfomrmance.