SA2_XTD1_ATT solid attenuator vacuum issue

Info from VAC (Raul Villanueva) 12h30:

It seems to be a leak in the arm nr. 6 (2.4mm CVD Diamond). This has escalated to trip several times the vacuum sector interlock.
On top of that, we also discover an issue with the configuration of a ion pump controller that have required the intervention of DOC / EEE. 

If the arm 6 is kept closed (2.4 mm CVD  is inserted), the leak rate seems to be small enough to allow operation (average sector pressure in the low 10-7 mbar). Otherwise it will blow consistently the interlock. (I repeat here: DO NOT MOVE THE 2.4 MM CVD!!!) See attachment #1. Most probably the leak would be in the custom-made bellow of that particular actuator. XRO just confirmed that there is no spares available.

Info by MID (12h30):
For today, MID can operate with the 2.4mm diamond inserted without impact to their experiment. However, tomorrow morning there will be a new user and experimental setup, which will require full transmission throughout the weekend, therefore the 2.4mm diamond attenuator must be removed friday. The change in experimental setup means that MID will work in the hutch at 10am for several hours and won't take beam anyhow.

Discussion of PRC+RC+XO+VAC+MID (15h30), with agreement to the following actions:

a) VAC prepares an intervention to XTD1 with broken interlock tomorrow morning startin 9am to remove arm 6 permanently and putting a blind flange instead. This intervention is estimated to start 9am and last for 4 to 5 hours. VAC will inform BKR if the work can be completed quicker.

b) Next week (last operation week in this run) therefore the Solid attenuator arm 6 won't be available. HED confirmed to PRC that it will be acceptable for their experiment since they plan to use full transmission in XTD2 and use their own attenuators in the HED hutch to attenuate the beam. XPD confirms that the XTD2 attenuators are not required for HIREX operation since HIREX contains its own attenuators between the bent crystal and the detector.

c) BKR/RC will organize the required tunnel search for the friday afternoon, and subsequently the HXRSS team will start optimization.

d) X-Ray Optics has already initiated the contact with the ATT device supplier to get replacement parts that will allow to recover full functionality during the upcoming (short) summer maintenance week.

Repair of SA2_XTD1_ATT

Status 12h30:
The current MID users took beam until 9am, only then the machine could start preparing for the tunnel access e.g. by decoupling the South branch, powering down magnets and taking XTD1 out of the interlock. The VAC team was ready to go into the tunnel for the repair in XTD1 already since 8am.
Unfortunately, the tunnel access of the VAC team got delayed beyond 9am by one more hour to 10am because the DACHS terminal at ground level of XS2 was found defective, and releasing the tunnel interlock took substantial time.
DESY experts went onsite and informed that a proper repair of the DACHS terminal can only happen next monday.
Only with a workaround, finally the VAC team was able to use the elevator to bring the required large venting and pumping equipment down to the tunnel level.
The vacuum works are ongoing with highest priority.

Status 14h20 :
VAC has finished the intervention, informed BKR and PRC, and is leaving the tunnel. Tunnel search team can start.

Status 15h00:
Tunnel search almost finished (also found some technical obstacles), Seeding experts are informed and ready to start their work.

18h05 : RC informs:

Machine is down, because Cryo-OK was lost.
Apparently it is a turbine but not the cold compressors.

It is thus not yet known how long the downtime will last.
More info will come within the next hour.

MID and FXE were informed by PRC, SCS could not be reached yet.

The Cryo team solved the problems, beam is back.
Downtime from 17h59 until 21h03.

staus of JUN16, 11:30

SA1 and SA3 have been adjusted to acceptable conditions to be compatible with simultaneus operation at SA2

tuning on SA2 is ongoing

---------------------------------------------------------------------------------------------------------------------------------------------------------

status of JUN16, 8:40

Update: problem is solved. Rack fuse in electrontunnel was blown. Operators are bringing the machine back up at the moment.

Afterwards (around 9pm) photons are all needed at SASE2 for tuning to full train mode.

Photons will be given back to SA1 and SA3 as soon as possible. Please encounter good beam conditions for measurements not before 10 am.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Dear all,

please note that we have a downtime since 4am which is expected to last until 8am or longer. There are issues with the following components: magnets, BLMs, the XS1 dump-PLC etc.

The source is yet unknown and will be investigated in an access to XS1.

I will keep you informed in this channel.

Regards

Michaela

Photon Activities by XPD colleagues in SA3 are concluded on Sunday 23.07 at 17:00. The photon commissioning activities are concluded for this week.

The PRC agrees with RCs to handle beam back to BKR for further tuning activities while touching common nobs.

After the 12 hour cryo failure last night, it was decided to exchange a faulty window with some reduced transmission at the injector laser today (see email by Andreas to xo-info yesterday). 

This was sucessful and the injector tunnel was closed today 16:40 and restart of the gun is ongoing. Tomorrow ramp up and SASE tuning.

The hope is that the large jitter and the poor SASE performance of last week can be now improved.  Beam delivery to instrument is expected on Thursday. 

As discussed, there will be a beam interuption form 14:45 until probably 15:45 related to jitter studies at A1 that the BKR would like to try out. 

Harald

Good news: The 'jitter'-tests was sucessful and BKR could identify the jitter source related to a wrong software setting around the AH1 module of injector.

This was now fixed and the jitter has disappeared  . It is likely that the tuning was compromised by this setting from the beginning of this run (July11).

On Monday morning, a fresh tuning round will be started with hopefully improved SASE performance. Until then photon delivery with what we have.  

The hutch search of SASE3, SXP became defective around 5am. 

DESY MPS group replaced a defective warning light unit and now it seems ok.

Here is a capture of the observed issue with the SPB shutter the past night.

Observe the two opening/closing incidents: the first kept the shutter "OPEN" signal during 4 minutes aprox., and the second did the same for about 30 seconds.

Not clear the origin of these actions/changes on the status of the signals.

SA1 photon beam line status

- CRL1 lenses are in the beam. 

- The shutter between XTD2 and XTD9 is closed (The reason is not known)

I will retract the CRL1 lenses. Asked the machine to open the XTD2 and XTD9 shutter.

SA1 photon beam line status

- CRL1 lenses are in the beam. 

- The shutter between XTD2 and XTD9 is closed (The reason is not known)

I will retract the CRL1 lenses. Asked the machine to open the XTD2 and XTD9 shutter.

Photon beam delivery status:

- Machine informed that beam will be handover to instruments SA1 (SPB/SFX) and SA3 (SXP) at 18.30hr

- Machine needs more time for SA2 and the next update will be at 20hr.

This message has been communicated to instruments.

- Machine updates;

Saturday 1:30 hr Cold compressor failed. The initial estimation of downtime was 20 hr, later in today morning crew gave a new estimation of time of 30 hr. So the machine will be switched on and beam delivery to instruments will be tomorrow morning. Today at 3 pm, I will get more information and a clear picture of the best estimation time for beam delivery.

Naresh

Dear All,

Two important machine updates.

1) Still, the forecasted time for the start-up of the machine will be tomorrow morning.

2) After discussion with machine RC, it is confirmed that beam delivery is extended by 24 hr, so the beam is available till 7 am Tuesday morning for instruments.

Just want to point out, that this is a very exceptional case that it is extended by 24 hours as there will be a blue week(in wk 35).

The below programs have been canceled keeping in view of the user program and beam delivery is important for instruments. Sorry for this.

153339 - Euler MDL calibration for SASE2 mirrors M1, M2, M3: Antje 

158376 - Beamtime for Safety tests (HED Preabsorber): Eric

156699 - SASE3_XTD10 Virtual Spectrometer: Joakim

157789 - Chirp-dispersion short pulses study (further decoupling from SASE1): Svitozar

Stay tuned to the public display tomorrow morning for the machine's status.

thanks

Naresh

PRC wk 34

Info from MRC (Lars)

-----------------------------------------------------------------------------

The cryo crew reports that the cold compressor that failed is, alas!,
the new one with the magnetic bearing. They have to wait for a colleague
who is not on call to help them with the problem. We expect an update on
their further plans around noon and our status panels have been updated
to reflect that.

Cheers,
Lars

-----------------------------------------------------------------------------

- Due to the failure of the cryo cold compressor. The machine is down.

- Next update 9 pm today

Please do watch public display for status info.

Info from Machine RC (Lars):

----------------------------------------------------------------------------------------------------------------------------------------

Author: Froehlich

One of the cold compressors has stopped working around 2:00 on Saturday morning due to a problem with the new motor (with magnet bearing). Since no beam operation is possible anyhow, the cryo team and machine coordination have decided to start necessary work on cryo plant stages 3&4 already on the weekend, in parallel to troubleshooting the motor problem.

All XFEL shifts up to Monday 7:00 are canceled. The morning shift on Monday will supervise the already scheduled access for the repair of A16 and, if possible, the restart of the linac.

This elogbook entry was sent to following experts:
Photon RC mailing list RC mailing list xfel-operation-board@desy.de

------------------------------------------------------------------------

Direct link to e-logbook entry:
https://ttfinfo.desy.de/XFELelog-sec/show.jsp?dir=/2023/36/09.09_M&pos=2023-09-09T14:51:07

------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------

Concerning SASE1:

Please note that for this week the "remote power limit" was set to 5 Watt in order to protect CRL1.
The photon energies this week are 5.5 keV (FXE, day) and up to 7keV (SPB/SFX, night).
Daytime SASE1 operation by FXE will not come close to any damage limits (16 pulses at low rep rate).
However, SPB/SFX will operate at 1.1 MHz and up to 356 pulses/train. If there would be a sequence
procedural error during the shift handover switching of parameters, delivery of 5.5 keV at this rate could damage the CRLs.

If this limit would at any point limit the experiment, please get in contact with the PRC.