REPAIRED / FIXED

Problem: The imager SPB_XTD9_IMGPII45 insertion motor lost steps and kept the machine to maximum 2 bunches per train.
Mitigation by DOC/EEE/PRC on 8 April 2023: made sure that screen seems actually out of beam and took imager out of interlock condition.
Repair by XPD on 10 April 2023 02h15: 
a) recalibrated OUT position.
b) verified that EPS-OUT switch shows green when moving to OUT position in MDL.
c) put back the imager into the interlock conditions.
d) communication with BKR before and after the intervention (SPB and FXE didn't use the beam).
Repair instruction see: https://in.xfel.eu/elog/SASE1+technical+commissioning/2065

Trigger issue with SXP_XTD10_IMGES - Exit Slit imager

The external trigger of the Navitar camera was not set, fixed now.
The single pulse trigger of the imager: MCP + camera is tested and correctly working.
The pre-settings are stored in Karabo and should come up again, also after Karabo updates.

We report on an issue regarding the bunch pattern: The bunch pattern read by SCS_RR_UTC/MDL/BUNCH_DECODER shows 48 pulses, but we instead use 50 pulses.

Currently, we have a separate RF window and our 1st pulse Id is 2604. We operate at 2.2 MHz, and as required, 50 pulses. See attachment 1.

This means that the last two pulse Ids are 2700 and 2702. However, these values exceed the number of pulses allowed, and the way the bunch pattern is read (SCS_RR_UTC/TSYS/TIMESERVER:outputBunchPattern,
and the middle layer SCS_RR_UTC/MDL/BUNCH_DECODER) cast the array to 2700 values, so that the number of pulses read is 48 instead of actual 50.

We confirm that we indeed have 50 pulses, as seen by the XGM and by photodiode traces showing 50 peaks.

Since our data analysis strongly relies on the bunchpattern table, this is a critical issue.

For now, the users agreed to use 48 pulses instead of 50 pulses to facilitate the data analysis, but this issue should be investigated.

DOC was called, ITDM is aware and helped finding the reason.

edit 20 Apr 2023, 16:26 : redmine ticket https://redmine.xfel.eu/issues/151949

Issue with SA1_XTD9_PBLM and it is stuck.

Hi All,

After discussion with SPB, they will slightly move the PBLM screen where they don't see any distortion in the beam and are willing to continue to work till Monday. 

The conclusion from EEE (Leandro) and DOC is that from PLC everything looks fine and it might be with mechanics for this we need to physically look at the device. 

As SPB can work like this till Monday. I proposed on Tuesday with someone who has knowledge of PBLM device can organize the ZZ

Next week Jan is PRC

Dear FXE colleagues,

There is an issue with SA1_XTD9_PBLM and it is stuck. Please when you use the beam in the night, coordinate with SPB first. or call PRC no. 9-4800

- Received a call from machine RC (Sergey Tomin)

- There is an issue with DOOCS to Karabo bridge, Tim and DESY experts are looking into this and came to the conclusion that it can't be resolved now and need more time something like 1 week. (probably next week). Due to this PySpectrometer for SA3 is not available, as SQS would like to set up a spectrometer to measure pulse duration. However, it can work in Karabo and take data.

Laser synchronization is gone, unlocked from 04.05.2023, 22.47. E-mail was sent to lbsync-expert@desy.de and c.c. to las-op@xfel.eu.

SA3 GATT has trouble in setting the pressure at ~0.5mbar. setting any value between 0.5 and 0.6 will have it settle around 0.48.

It turns out the GATT cannot tune the pressure between 0.48 and 0.62mbar. tuning pressure continuously to keep transmission constant gets us in the need of that range eventually. the problem is that at 0.48mbar the valve V30740S is at 100%. the minimum opening of the next valve V307030S gets already 0.62mbar.

[edit simon, after talking Michaela]

we had the idea that one could use two of the MFCs (M.R.+L.R.) at the same time to extend the range of the low end. Indeed this does work but one has to make sure that Set Control Mode is 1! (att 1)

This, at least gives us a upper limit of the low range of 0.52 mBar (mind you, this is only for manual settings, parallel operation of MFCs is not forseen so far by the software)

Set Control Mode to 1 on U.R. MFC gives us a low end of 0.56 mBar on the upper range

HED reported that they are unable to go 352 pulses and while changing/dailing the no. of pulses in the bunch Pattern it is showing the EPS is limited to 300 pulses. It seems on the SA2_MAIN scene the link for bunch Pattern apparently using the macros scene, which is preventing it from going to 352 pulses. Now I have updated the link with a new scene of MDL for the bunch pattern. In the attachment, you will find both scenes. One toward the left is MDL bunchPattern configurator and one towards the right is Macros bunchPattern. 

Now the issue is solved. 

thanks

Naresh

PRC wk 22

around 6:15, the lights as the one shown in the attachment turned on by themselves in the control hutch. after some minutes, they switched off.

from the exp hutch camera, we realized that this happened in there as well, which impacted negatively on the running measurement. The pnCCD is sensitive to ambient light and we have to keep the hutch dark while measuring.

Why did these lights turn on?

SASE2 vacuum incident

7h10 : MID informs PRC about a vacuum incident in the SASE2 tunnel which tripped the beam. Instructed to contact directly VAC-OCD

8h26 : VAC-OCD informs that investigation is ongoing, but beam can be delivered. This kind of trip usually occured rarely
           and usually can be reset by VAC. Apparently the source of the issue is at the XTD1 solid attenuators.

8h36 : MID confirms to PRC that already 3 trips occcured but the experiment so far can continue. Beam delivery continues.

10h11 : VAC-OCD informs that now there is a bigger more wide-spread issue with the XTD1 vacuum system
           and beam operation is stopped for at least 1 hour. VAC group and DOC are working on the issue.

11h : It turns out that the vacuum bellows of the 2.4mm diamond attenuator in SA2_XTD1_ATT has a leak, which is worse and
         not acceptable in terms of vaccuum in the extracted state (bellows elongated) but less severe in the inserted state (bellows compressed).

12h33 : info by VAC: issue is temporarily solved by blocking the 2.4mm diamond arm in the IN position. Must clarify today how to proceed further.
          The vacuum issue thus is temporarily solved, beam permission is back,

12h38 : Beam coming back to MID. MID can work today with this ATT always inserted, but not for the weekend.

Addition to previous issue:

Last incident (of overall 4), we had 0 bunches, moved in XTD1 ATT 0.6 and 1.2 mm CVD diamond in and 2.4mm CVD out. However, vacuum started to become bad already two days ago.

Update from vacuum OCD: Seems to be related to one leaky below of 2.4 mm CVD arm. Operational consequences are discussed right now.

SA2_XTD1_ATT solid attenuator vacuum issue

Info from VAC (Raul Villanueva) 12h30:

It seems to be a leak in the arm nr. 6 (2.4mm CVD Diamond). This has escalated to trip several times the vacuum sector interlock.
On top of that, we also discover an issue with the configuration of a ion pump controller that have required the intervention of DOC / EEE. 

If the arm 6 is kept closed (2.4 mm CVD  is inserted), the leak rate seems to be small enough to allow operation (average sector pressure in the low 10-7 mbar). Otherwise it will blow consistently the interlock. (I repeat here: DO NOT MOVE THE 2.4 MM CVD!!!) See attachment #1. Most probably the leak would be in the custom-made bellow of that particular actuator. XRO just confirmed that there is no spares available.

Info by MID (12h30):
For today, MID can operate with the 2.4mm diamond inserted without impact to their experiment. However, tomorrow morning there will be a new user and experimental setup, which will require full transmission throughout the weekend, therefore the 2.4mm diamond attenuator must be removed friday. The change in experimental setup means that MID will work in the hutch at 10am for several hours and won't take beam anyhow.

Discussion of PRC+RC+XO+VAC+MID (15h30), with agreement to the following actions:

a) VAC prepares an intervention to XTD1 with broken interlock tomorrow morning startin 9am to remove arm 6 permanently and putting a blind flange instead. This intervention is estimated to start 9am and last for 4 to 5 hours. VAC will inform BKR if the work can be completed quicker.

b) Next week (last operation week in this run) therefore the Solid attenuator arm 6 won't be available. HED confirmed to PRC that it will be acceptable for their experiment since they plan to use full transmission in XTD2 and use their own attenuators in the HED hutch to attenuate the beam. XPD confirms that the XTD2 attenuators are not required for HIREX operation since HIREX contains its own attenuators between the bent crystal and the detector.

c) BKR/RC will organize the required tunnel search for the friday afternoon, and subsequently the HXRSS team will start optimization.

d) X-Ray Optics has already initiated the contact with the ATT device supplier to get replacement parts that will allow to recover full functionality during the upcoming (short) summer maintenance week.

Repair of SA2_XTD1_ATT

Status 12h30:
The current MID users took beam until 9am, only then the machine could start preparing for the tunnel access e.g. by decoupling the South branch, powering down magnets and taking XTD1 out of the interlock. The VAC team was ready to go into the tunnel for the repair in XTD1 already since 8am.
Unfortunately, the tunnel access of the VAC team got delayed beyond 9am by one more hour to 10am because the DACHS terminal at ground level of XS2 was found defective, and releasing the tunnel interlock took substantial time.
DESY experts went onsite and informed that a proper repair of the DACHS terminal can only happen next monday.
Only with a workaround, finally the VAC team was able to use the elevator to bring the required large venting and pumping equipment down to the tunnel level.
The vacuum works are ongoing with highest priority.

Status 14h20 :
VAC has finished the intervention, informed BKR and PRC, and is leaving the tunnel. Tunnel search team can start.

Status 15h00:
Tunnel search almost finished (also found some technical obstacles), Seeding experts are informed and ready to start their work.

18h05 : RC informs:

Machine is down, because Cryo-OK was lost.
Apparently it is a turbine but not the cold compressors.

It is thus not yet known how long the downtime will last.
More info will come within the next hour.

MID and FXE were informed by PRC, SCS could not be reached yet.

staus of JUN16, 11:30

SA1 and SA3 have been adjusted to acceptable conditions to be compatible with simultaneus operation at SA2

tuning on SA2 is ongoing

---------------------------------------------------------------------------------------------------------------------------------------------------------

status of JUN16, 8:40

Update: problem is solved. Rack fuse in electrontunnel was blown. Operators are bringing the machine back up at the moment.

Afterwards (around 9pm) photons are all needed at SASE2 for tuning to full train mode.

Photons will be given back to SA1 and SA3 as soon as possible. Please encounter good beam conditions for measurements not before 10 am.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Dear all,

please note that we have a downtime since 4am which is expected to last until 8am or longer. There are issues with the following components: magnets, BLMs, the XS1 dump-PLC etc.

The source is yet unknown and will be investigated in an access to XS1.

I will keep you informed in this channel.

Regards

Michaela

The hutch search of SASE3, SXP became defective around 5am. 

DESY MPS group replaced a defective warning light unit and now it seems ok.

Here is a capture of the observed issue with the SPB shutter the past night.

Observe the two opening/closing incidents: the first kept the shutter "OPEN" signal during 4 minutes aprox., and the second did the same for about 30 seconds.

Not clear the origin of these actions/changes on the status of the signals.

Potentially damaged 1.2 mm CVD in XTD2 attenuator. This feature was not present earlier in the week.

Vacuum issue at XTD2 tunnel which might be triggered by SA1_XTD2_ATT Arm5 (1.2mm thick CVD diamond)

It happened once again at around 3pm.

• We found that the x-ray pulse energy dropped a lot when we changed the undulator from 5.5 keV to 5.45 keV. We called BKR to optimise it (attachment 1)
  => There is a problem with the phase shifter of the undulator. It stacks sometime. If the x-ray intensity drops, we need to ask BKR to optimise it.
  => We were suggested not to change the undulator value by 100 eV. 5 - 10 eV step should be fine.

On the other hand, we had a user experiment 3435 Feb 2023, and we could perform photon energy scan from 5.7 keV to 6.1 keV without problem. (attachment 2)