Issue with SA3 Gas attenuator (SA3_XTD10_GATT) - SOLVED

Dear all,

after the reported issue by SQS colleagues regarding the impossibility to inject Argon, and after a first remote evaluation, we (VAC) have entered XTD10 to assess the status and verify the most probable working hypothesis: a manual valve in the supply line was closed (otherwise as expected). Joshua and Benoit confirmed it and the showstopper is now gone. The system is back to normal and standard operation can be resumed in any moment. 

A small remark: as Argon (or any other gas different from Nitrogen) is not the default operation mode, it is important that if needed, there should be an explicit request during the preparation of the operation tables with XO. This would help to avoid this kind of situations. Unfortunately sending a short notice e-mail just before the next run is clearly not recommended.

With best regards.

SASE3
Open Issues:
- currently it is not possible to change the number of pulses because of Instruments beam permission interlock
Done:
- VSLIT was calibrated with diffraction fringes (it was off by 41 microns, the bottomo blade encoder was recalibrated to compensate this error)
https://in.xfel.eu/elog/XRO+Beam+Commissioning/319
- various setting points were checked at 20 mrad using mirror switch, no particular problems found
https://in.xfel.eu/elog/XRO+Beam+Commissioning/310
https://in.xfel.eu/elog/XRO+Beam+Commissioning/306
- ESLITS width was measured using diffracted beam. No Calibration was applied, we will discuss with instruments if this is needed/wished
https://in.xfel.eu/elog/XRO+Beam+Commissioning/323
https://in.xfel.eu/elog/XRO+Beam+Commissioning/324
https://in.xfel.eu/elog/XRO+Beam+Commissioning/325
- ATTENTION: SQS ans SXP ESLITS motors are now moving in the opposite direction as before (please look at the updated picture on the scene). Software limit switches are real limits: do not deactivate them
SCS ESLIT was already in this way: now all the ESLITs are configured similarly (no Elmo controllers anymore)
- SRA calibration was recalibrated
https://in.xfel.eu/elog/XRO+Beam+Commissioning/313
- FEL imager cross adjusted to GATT
https://in.xfel.eu/elog/XRO+Beam+Commissioning/307

SASE2
Open Issues
- HED Mono1 still not going through until HED popin. Paralleism of crystals achieved. Transmission is ok until Beam Monitor but not further.
Beam height was checked but the problem remained
Done:
- M3 mirror realigned, Euler motion roughly calibrated. 
ATTENTION: Old saved values are no longer usable. "XRO setpoints" must be used as a starting point for further alignment
- Positions C and E were saved
- The fast flipping positions were almost found, see here: https://in.xfel.eu/elog/XRO+Beam+Commissioning/322
Unfortunately, beam was suddenly off and the procedure was not complete
- No particular problems found

SASE1
- FXE and SPB trajectory rechecked, no particular problems found

ATTENTION:
- M1 & M2 motors were found with wrong scaling factors (since some times). They are corrected while maintaining the mirror positions. (Most affected was M1 vertical motors which is rarely used).  
- saved positions are using the "fast flipping option" meaning that you can switch between the two only using M2 RY
https://in.xfel.eu/elog/XRO+Beam+Commissioning/283
https://in.xfel.eu/elog/XRO+Beam+Commissioning/282
- CRL1 motor calibrations were lost, and restored from 6th Dec. In case if there are further issues, contact Control. 

Starting shift with mirror motor calibration:

- beam is centered on FEL imager

- pblm in XTD9 is inserted

We removed both silicon solid attenuator and pblm

Starting shift with mirror motor calibration:

- found solid attenuator 0.5mm silicon inserted

- beam is centered on FEL imager

- pblm in XTD9 is inserted

We removed both silicon solid attenuator and pblm

- M3 TX moved to 0 in steps of 1 mm and beam on FXE Pop-In corrected with M3 RY (+) and M2 RY

- Started calibration with M3 Rz in steps of 0.5 (+) and corrected with M2 RY and RZ to find a position with a minimum of parasitic motion

- Iterations with M3 TY

- CRL 3 and 4 were inserted (using find target)

- The backlash correction was disabled

- Curve for M3 Rz position with Ry correction done. M2 RY/RZ corrections to move beam up and down to have it on the screen

- Curve for M3 Ty position with Ry correction done.

- Curve for M3 Rx position with Ry correction done

M3 vertical motors calibrated to 0

Dear all,

we calibrated the mirror vertical motors to their geometrically neutral.

Please be aware that saved values for beamline settings may no longer be valid. This mainly applies to the vertical motor positions, but will have an impact on Tx and Ry values as well.

As starting point, use XRO Optics Setpoints 'C' - this setting was last used to bring beam to SPB

We fine tuned geometry parameters for linear effects on X motion (parallel offset) for all three mirrors.

dTz was corrected for M2 and M3 to -6.5mm (same geometry)

dTz was corrected for M1 to +6mm

for M2 we were able to do a parabola scan Y motion to check for lrTz

lrTz was then corrected to -433mm

correction in the same direction for M3 did not yield better results

Final mirror positions at HED - see picture

We plan to put the piezos of M2 off, in the (unprobable) case that this is the reason of the "jumps" in SASE3 trajectory.

Therefore, I measured the change on RY and RZ motors, to correct all the positions

Fig 1: initial position, Motors Ry=10.5425 and RZ=-14.7200  Piezo Ry =5 and Rz=5

Fig 2: Ry is compensated, Motors Ry=10.4990 and RZ=-14.72  Piezo Ry =0 and Rz=5

Fig 3: Rz is compensated, Motors Ry=10.4990 and RZ=-14.7574  Piezo Ry =0 and Rz=0

Differential values: RY needs to be adjusted by -0.0435 and Rz by -0.0374

Beam is now aligned back to HED for hand-over to Naresh (see picture)

points achieved:

-vibometer measurements on M2 at high energy, 2,1 and 3 mrad setting

-tried again to find beam through pblm grating, but no success

this morning, ZZ was done together with Sylvia Huynh (EEE) to solve softmono pre-mirror motor issue https://redmine.xfel.eu/issues/177392

The issue was solved and also mechanical functionality of the stage was checked.

After ZZ, the beamline remained interlocked. Hutches will be searched and closed tonight to allow final checks on Tuesday

What has happened:
During euler motion calibration of M1, we were driving the vertical motors by a few mm back and forth. At some point, the Rz motor went into error, as we attempted to go further in negative direction. Resetting the axis several times did not solve the problem. The vertical motors were then moved to 0 position. DOC was called and helped to overwrite some parameters that triggered the error.
When resuming the calibration routine, the beam suddenly got lost and the vacuum interlock triggered V0 to close. Vacuum in M1 recovered quickly, but on M2 popIn, only the direct beam could be seen. At this point, the conclusion was that one of the motor connections must have come loose, which caused the vacuum event and removed M1 from the beam.

It was then decided, that we break the interlock, vent the chamber and re-connect the motor if possible.

The situation we found was the following:
-Rz connection lead-screw to bellow was detached
-mirror sitting on the mirror holder
-significant InGa spillage on chamber bottom, window flanges opposite to the mirror AND a lot of drops on the mirror surface

Pictures are attached.

Intervention:
The lead-screw and nut were exchanged. There was significant wear on the bearing plate, connecting the lead-screw to the bellow.
When trying to install the new lead-screw with a spare bearing plate, it was found that apparently the thread on the bellow side was deformed as well, and the spare part did not fit to the thread. Since the old bearing plate was the only possible way to connect the lead-screw to the bellow, it was cleaned, lubricated and re-used. The position was manually adjusted around 0 position (66,7mm protrusion) and the motor connected.

Recommendations:
- since the damaged bearing plate was re-installed, it should be avoided to move Rz. The motor should be permanently interlocked and only operated with absolute care for initial alignment.
- the mirror surface is severely contaminated with InGa, it must be checked if a position with acceptable beam can be found (only to be done by XRO as this needs Rz to be moved)
- the affected bellow has to be exchanged to facilitate replacement of the damaged bearing plate. As this requires the mirror chamber to be opened, this only makes sense in connection with replacement of the mirror + cleaning of the chamber.

X-ray delivery

• 0.25nC, 14 GeV, rep rate 2.25 MHz, 9300 eV, 1 to 100 pulses, ~2000 uJ
• M1 and M2 mirrors are aligned and beam trajectory to SPB.

Goal

• Commissioning of the Gotthard-II 25um MHz detector on SA1-HIREX.

Achievements

• Aligned the HIREX using bent diamond crystal C110 and no gratings used.
• 2D camera(Bragg/photonic science) aligned and spectral data is collected.
• Trigger delay scan of Gotthard-II detector. The best trigger delay setting was found at 1952637.
• Calibration Pipeline is established and can load/inject the constants
• Online preview of Gotthard-II corrected data is successfully tested and commissioned.
• successfully able to load calibration constants and dark runs in myMDC.
• Successfully able to start the calibrated data after completion of runs in myMDC.
• Calibration/corrected and dark data reports are successfully generated in the proposals
• New filter stage commissioned and tested. Updated the positions in the MDL.
• Intensity scan output for strips 2150 and 2151, obtained varying the SA1 diamond attenuators. I_0 is calculated integrating the output from strip 2000 to strip 2300.
• Different gain setting data was collected.
• Gotthard-II tested and commissioned with different rep. rate: 4.5 MHz, 2.25 MHz, 1.125 MHz, 0.5 MHz. The test was sucessfull and we can record spectrum for different rep. rate.
• There are no visible aritfacts around the gain switching region, and the correction seems to work well.
• spectral data is collected with 2.25 MHz with pulses 2, 30 and 100.
• Attachment 1: Fisrt results from Gotthard-II with spectral data in MHz rep rate.

Issues:

• Calibration pipeline of pulses/train (ASSEMBLE_STREAK) couldn't able to instance. DRC and OCD DA group helped to fix (Thank you)
• Adhoc data sources were added in SA1 COMM with support from DRC and CTRL (Thank you)
• beam lost at ~12:30hr today due magnets issues and till now no beam. Unable to continue the rest of the program planned for commissioned of the Gotthard-II.

Thanks to Marco, Jiaghuo, DRC(Monica), CTRL OCD (Andrea), DA OCD (Karim) for there support and help.

EPS issue for Transmissive imager SA1 (OTRC.2615.T9) w.r.t Operation eLog #96

The motor screen positions w.r.t. the EPS switches had been realigned (last Wednesday 8th May), tested today (14th May) and the EPS is again enabled for the different beam modes. All is back to normal.

Concerning damage of the SA1 FEL imager, Operation-2024 eLog #206

The screen position of the YAG screen - when pressing the screen selection button - has been moved by 1 mm so that the damaged area is out of the field of view, see attached screenshots (damage visible under visible light exposure).
The screenshots show the cross-hair, i.e. its nominal beam position and the circular damaged area at its original position of the screen and the new position.

More information on the damage can be found in the damage report, entry #35:
https://docs.xfel.eu/share/page/site/xfelwp74/document-details?nodeRef=workspace://SpacesStore/fd862f30-c6a3-4a5e-9a65-817cbcebb2d9
Also, there is a comment in ttfinfo: https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2024/47&pos=2024-11-19T09:39:31

Dear MID colleagues, Svitozar (PRC), Bolko and Shan (RCs),
This morning we faced the situation where MID was limited to 400 pulses (mode M) via the EPS signals. The current RF window set at the machine allows to the requested 450 pulses.
The limiting conditions were the inserted graphite filter (#2) and the diamon screen at the MID Pop-in (#17). After a discussion with experts (and thanking Naresh, Wolfgang and Andreas), both conditions in EPS mode F (see attached) are now disabled, as the current SA2 beam offer safe condition for both devices.
This special situation with disabled EPS signals in mode F is in place until Tuesday 05.Nov as MID will stay at ~12.28keV and with HXRSS (500-600uJ). Please note that in case of either a change in photon energy, substantial increase of photon flux, longer SA RF window or the wish to insert YAG screen at the SA2 Pop-in, these two mentioned conditions have to be enabled again at the SA2 EPS Mode F prior these actions take place.
After the beamtime ends on Tuesday 7AM, either me, Naresh or the PRC will make sure these two EPS conditions are re-enabled.
In case of questions please do not hesitate in contacting me.
Best wishes,
Frederik

**Update Mon 15:00: The beamtime delivery to MID is extended up to Tuesday 9AM. At this moment the EPS signals will be enabled again by Frederik W.-F.

The cross on the FEL imager, which we use as a reference for BKR crew, is not centred on the GATT aperture. when was this changed, and why?

att. 1 and 2 show the current position (the saturated image aids the positioning of the GATT aperture).

att. 3 is from two weeks ago. then, the cross was centred on the GATT aperture.

it is not clear why, but the beam position on BIU2 has changed over time, even more since we checked last. it is off in both directions

we see the GATT aperture projection drifting, so it is a change in the beam transport, not in the source.

see the beam, cut by the DIAG1 slit, compared to the cross it should be centred to.

the motor encoders in the beamline have not changed since the beginning, when the beam was pointing correctly.

we needed SASE1 to be set to 0 pulses briefly to check alignment in our experiment.

The FXE control hutch was empty. we could not reach either Chris nor Mykola by phone. The XFEL_StatusPublic shows that the FXE shutter is closed (we can also see the exp hutch door open from outside the control hutch).

Neither PRC nor BKR had any information about the FXE status.

We discussed with PRC and concluded that they are likely not using the beam, so in agreement with PRC we went ahead and got BKR setting SASE1 to 0 pulses at ~10:20. After few seconds, we got BKR setting SASE1 back to 30 pulses.

edit: PRC got in contact with FXE. Turns out they were (are) working in their ILH. we have a contact procedure for the next occasions now.

our pp laser experienced timing jumps by 18 ns today. this happened for the first time at ~7:30 in the morning without us noticing and a couple of more times during the morning. we realized it at 15:30 due to the fact that the signal on our pulse arrival monitor (X-ray/laser cross correlator downstream of the experiment) disappeared. we spent some time to investigate it, until we also see this jump in two different photodiodes different triggers (oscilloscope and digitizer) so it was clear that it is indeed the laser timing jumping. we also see the laser spectrum and intensity change for the trains that are offset in timing.

we called laser OCD at ~16:30. they have been trying to fix the problem by readjusting the instability zone multiple times and by adjusting the pockels cell, but this did not change the timing jumps. at 21:30 they concluded that there was nothing more they could do tonight and a meeting should happen in the morning to discuss how one can proceed.

effectively, this means that we lost the entire late and night shifts today for the pump-probe beamtime and have some data of questionable quality from the early shift as well.

Laser status:

After the laser issues listed in https://in.xfel.eu/elog/OPERATION-2024/99 the late shift was finally able to find a timing setting for the amplifier trigger that was beyond stable settings under usual conditions, but which allowed to continue taking data runs for the rest of the beamtime reliably. The users were already able to reveal indications of a pump-probe signal in their data.

The issue itself remains but didn't show up in form of unstabilities anymore. After the beamtime LAS took some effort to investigate and narrow down to either a faulty trigger going into the front-end amplifier or on the front-end amplifier itself. To the best of our knowledge this circumstance will need to involve EEE support or even a service from the manufacturer and cannot be solved solely by LAS. LAS ensured that the highest priority will be to deliver stable laser pulses for next week's beamtime as well.

FEL status:

we found the FEL poynting to be very stable troughout the last weeks beamtime (despite some short down-times), which allowed to reliably verify the spatial overlap.

Today in the afternoon, SASE1 was tuned (a description of the activities in att 4). We now realize that this has in some form changed the SQS pulse parameters.

att 1 shows ion spectra recorded under nominally the same photon parameters before (run 177) and after (run 247)

it's also visible in the history of the SASE viewer that the behavior of the SASE3 pulse energy is different before and after 15:45, which is right when SASE1 was tuned. it clearly started fluctuating more. however, we don't think that the pulse energy fluctuations themselves are the reason for the large change in our data, we rather suspect a change in the pulse duration and/or the source point to cause this. however, we have no means to characterize this now after the fact.

it is particularly unfortunate that the tuning happened exactly BEFORE we started recording a set of reference spectra on the SASE3 spectrometer, which were supposed to serve as a calibration for the spectral corelation analysis to determine the group duration, as well as for a training data set to use the PES with machine learning as a virtual spectrometer for the large data set taken in the shifts before.

We asked the machine for two color mode in the morning. Tuning gave good results at first (1.6mJ second color, 100uJ first color), but as soon as any of the colors was switched off and on again the pulse energy dropped (~600uJ) and BKR had to retune to even get a fraction of the pulse energy back back. The same happened when changing the delay

31.01.2024 (Day shift)

X-ray delivery

• 10 keV
• 1200 - 400 uJ (significant pulse energy slope within train)

Optical laser delivery

• None

Achievements/Observations

• Flat fields and timing scans were measured for AGIPD (1.1MHz, normal CDS mode)

Issues

• Interlocks issue causing beam to be interrupted when the beamline shutter was open https://mylog.connect.xfel.eu/#narrow/stream/111-SPB_900429_RichardBean/topic/Issues/near/32289
• Opening EXP hutch gate valves resulted in vacuum issue https://mylog.connect.xfel.eu/#narrow/stream/111-SPB_900429_RichardBean/topic/Alignment/near/32195
• Pulse pattern decoder recorded empty arrays, even though data were visible in Karaboo https://mylog.connect.xfel.eu/#narrow/stream/111-SPB_900429_RichardBean/topic/Issues/near/32299
• AGIPD timing issue caused by the wrong Trigger Source https://mylog.connect.xfel.eu/#narrow/stream/111-SPB_900429_RichardBean/topic/Detector.20Status/near/32408

To do for next shift

• Continue with the detector program

At 21:29:20 today, the number of bunches in SASE1 was increased from 1 to 352 without intervention of the SPB/SFX shift crew. 30 seconds later, the number of bunches was reduced to 1, again without intervention from the SPB/SFX shift crew (see attachments 1 and 2).

We were in the middle of a measurement at that time and the sudden increase from 10 mW to 3.5 W average power resulted in the destruction of the sample we were using.

AGIPD and JUNGFRAU are safe and were not at risk at the time of the incident, but this is merely by luck.

The origin of the change in number of bunches is unclear to us and needs to be investigated. It goes without saying that this is extremely dangerous for equipment and the damages could have been even worse.

X-ray delivery

• 1-352 pulses at 10 keV, 1200 - 400 uJ.
• Significant pulse energy slope within the train. Action on the kickers was eventually performed leading to some reduction of the slope, though remaining still far from ideal. Decision to take the action took a very long time (2.5 hours).
• Several beam delivery interruptions through the afternoon due to trips of A16 before is was taken off beam.
• At 21:29:20, the number of bunches in SASE1 was changed from 1 to 352 without intervention of the SPB/SFX shift crew resulting in the destruction of a sample (see elog:7).

Optical laser delivery

• 800 nm, 50 fs, 1-113 pulses at 564 kHz.
• Brief intensity and spectral fluctuations as already observed in the past with non-optimal AMPHOS regulation. Detail will be communicated to LAS.

Achievements/Observations

• Measured flat fields for AGIPD for all remaining configurations.
• Confirmed timing of JUNGFRAU.
• Established pump-probe timing at IRDa.

Issues

• Several issues with AGIPD controls: timeout when trying to stop sending data, two hemispheres in different states, etc. Experts informed.
• Destruction of the sample following number of pulses incident in the evening caused a considerable loss of time to replace the sample and resume operation.
• XTD2 500 um Si is damaged (leftover from last run).

X-ray delivery

• 1-64 pulses at 10 keV, ~1.1 mJ.
• Overall stable delivery.

Optical laser delivery

• NA

Achievements/Observations

• Measured flat fields for JUNGFRAU for all configurations. Detector program completed.

Issues

• Mishap of lit frame finder with custom bunch patter.
• Apparent issue to process JUNGFRAU darks caused by lack of disk space. Solved by DOC/CAL.
• Ghost run in myMdC upsetting the run controller. Solved by DOC and cause under further investigation.

X-ray delivery

• 1-350 at 12.45 keV, 1700 uJ. Stable delivery

Optical laser delivery

• None

Achievements/Observations

• fixing many issues with the cube and helium leaks
• collected many quality diffraction data for both jet with and without samples
• data on Nendu mutant (~2.5 hours of data collection) with two different conditions

Issues

• AGIPD data aggregator stopped responding and it took about 1 h to be fixed by DOC, which interrupted the data collection

• some strange streak diffraction appears on JF4M and this has to be removed by scanning X- and Y- nozzle motors carefully

X-ray delivery

• 1-350 at 12.45 keV, 1700 uJ. Stable delivery

Optical laser delivery

• None

Achievements/Observations

• collected about over 15 hours of quality data from different samples (thanks to FXE) with extremely stable jet.

Issues

• Issue with data processing in the morning and from the day before due to DAQ; only half of the detector data were calibrated; this was solved by switching to the old DAQ

X-ray delivery

• 1-350 at 12.45 keV, 1750 uJ. Stable delivery

Optical laser delivery

Achievements/Observations

• completed Uwe and Petra data.
• the jet is almost established for <mixing experiments

Issues

X-ray delivery
1-350 at 12.45 keV, 1700 uJ. Stable delivery

Optical laser delivery

None

Achievements/Observations

- collected data at different delay times 14s and 18s with over 7 hours of data collection
- the jet was stabilized again by removing the catcher and decreasing transmission to ~30%
- removing the catcher is critical to get stable jet, that allowed us to run for over 20 runs without stop.

Issues

- The jet was extremely unstable for 1.5 shifts; we needed to stop the runs intermittently with intervals less than 10 mins. 
- It seems difficult in principle to go for longer delay times with the current setting; may be elongating the distance in Y-axis with 400 pixel if the jet stable may help to reach 20s.

X-ray delivery

• 1-350 at 12.45 keV, 1600 uJ. Stable delivery

Optical laser delivery

• None

Achievements/Observations

• collected data at different delay times 20s and 40s
• we installed new mixing injector to allow for longer delay times; and collected complete dataset with it
• completed four different delay times for the users 14s, 18s, 20s and 40s

Issues

• spent longer times to make stable jet which cause many interruptions and hutch searches almost every 5 mins for several hours.
• it was extremely difficult to mix with delay at 20s with original setup.
• nozzles broken during the day, costing several hours to prepare new ones

Like after the previous winter maintenance period, the shutter of FXE appears as opened in DOOCS panels even though it is closed (https://in.xfel.eu/elog/OPERATION-2023/58).

X-ray delivery

• 1 to ~332 pulses at 6 keV, 4.5 mJ
• Stable delivery
• Number of pulses limited by the Photon Beam Power local limit that is set to 16 W. Is this intentional? All other limits are set to 40 W

Achievements/Observations

• Aligned NKB trajectory
• Achieved NKB focus
• Reduced background scattering on AGIPD
• Started data collection

Issues

• Identified a couple of problems with the new DAQ that we were testing. Controls group informed and investigating

X-ray delivery: stable and flat over 250 pulses per train. 1.1 MHz with 400 uJ

Optical laser delivery: NA

Achievements/Observations:

• Beam optimization and characterization
  - Short pulses tuned
  - HIREX setup and used to characterize the number of SASE SPIKES per pulse (~10)

Issues

• 10 SASE spikes per pulse is more than desired
  - discussion with BKR and AccRC regarding possibilities for optimization (feedback in the morning)
  -HIREX data proc is not correct as dark processing fails

  - DOC informed and working on it
  - one issue was gothard II appeard to have timed out - does the number of triggers need increasing?
  -sample jetting setup

  - electrospray working but saw exactly 1 hit in run 32
  - tried to alighn with silver nano-cubes but had to switch to sucrose to increase hit rate to aid alignment
  - no hits with sucrose seen with scanning X +- 1mm

X-ray delivery: stable and flat over 190 pulses per train. 1.1 MHz with 200 uJ short intervention needed to improve intensity after it dropped

Optical laser delivery: NA

Achievements/Observations:

• Beam optimization and characterization
  - Short pulses tuned and number of spikes reduced
• X-ray focus FWHM calculation found to have a bug in pixel size used
  - rest to 0.2 FWHM of focused beam now better than 1.5 microns
• Laser illumination installed to enable particle visualization to aid alignment and feedback
• Sample injection and data collection for several hours
  - Hits seen from Silver cubes
  - very low hit rate from copper
  - cubes, and spheres

Issues:

• DAQ - multiple crashes and restarts needed throughout the shift
• Proc pipeline - some runs fail to processwhile adjacent runs collected with same conditions (run repeater) process without errors
• Sample injection - copper samples continue to be difficult to inject
  - very low hit rates (few hits seen per run)
  - clogging issues
• Focus drift - focus was tuned at around 3 pm, by 9 when we could check again focus was >3 microns FWHM
• Beam intensity reduced during the shift and BKR needed to tune it back
  - returned intensity to 200 uJ - they asked if we wanted to continue but did not as this could have resulted in more SASE spikes

X-ray delivery:

stable and flat over 190 pulses per train. 1.1 MHz with ~200 uJ

Optical laser delivery:

NA

Achievements/Observations:

Beam optimization and characterization
Sample injection and data collection for ~10 hours

reasonable hit rates observed but highly fluctuating, making alignment optimisation slow

Issues:
DAQ - restarts needed throughout the shift
Sample injection - copper samples dont give high hit rate and increaseing sample concentration led to clogging
Focus drift - focus was tuned at around 3:30 pm ( from 4 um back to 1.5 um)
-again at 19:30 when we checked again focus was >2microns FWHM returned to 1.5

X-ray delivery
9.10 keV; 190 pulses per train @1.1 MHz with roughly 140-200 uJ; observed some fluctuation in the photon flux

Optical laser delivery
ns laser for particles visualization

Achievements/Observations

• Beam optimization and characterization
• X-ray focus FWHM calculation found to have a bug in pixel size used
• Sample injection and data collection for >10 hours
• collected data on copper cubes with average hit rates of ~5%

Issues

• DAQ - multiple crashes and restarts needed throughout the shift; at certain point we encountered three crashes within less than an hour; and called DAQ to intervene
• Gotthard II at Hirex stopped acquiring and we had to change the number of the acquisition time to ~10 days to allow continuous running until the end of the beamtime.
• hit rates were low and sonication of samples needed to be done several time to prevent clogging issues
• Focus drift - we observed significant drift in the beam; a refocus then had to be check several times to focus back the beam to nm
• beam intensity fluctuated a lot during the shift

X-ray delivery

• 9.10 keV; 190 pulses per train @1.1 MHz with roughly 170-200 uJ; stable delivery

Optical laser delivery

• ns laser for particles visualization

Achievements/Observations

• Sample injection and data collection for ~11 hours
• collected data on copper cubes with better hit rates than previous shifts; ~15%

Issues

• We had laser interlock issues, the magnet in door went to error state and this was fixed by Klaus Witt call around 22:15.
• data migration issues; it was very slow and DOC did not figure out the cause for this.

The status of the XTD2 attenuators has been checked during the experiment last week and many were found to cause disturbances in the wavefront ranging from mild to severe.

Attachment 1: No attenuator
Attachment 2: 75 um CVD - Fringes visible in beam
Attachment 3: 150 um CVD - Strong fringes in beam
Attachment 4: 300 um CVD - Looks more or less ok
Attachment 5: 600 um CVD - Weak distortion in the beam profile
Attachment 6: 1200 um CVD - Ok
Attachment 7: 2400 um CVD - Ok
Attachment 8: 500 um Si - Not ok

X-ray delivery

• 10 keV, up to 160 pulses at 1.1MHz, 1.8 mJ

Optical laser delivery

• NA

Achievements/Observations

• Instrument aligned up to IRDa at 10 keV.
• Focus optimised.
• JUNGFRAU timed.

Issues

• We could only start using the beam at 1 am.
• More mirror movement needed to get the beam down the instrument than expected.
• Several AGIPD power cycling needed due to issues with MFPGA2

X-ray delivery

• 10 keV, up to 128 pulses with custom bunch pattern, 1.6-2.0 mJ

Optical laser delivery

• NA

Achievements/Observations

• Collected first SFX data, but background on JUNGFRAU is rather high and needs further optimisation.

Issues

• AGIPD having frequent problems with both MFPGAs (once even both at the same time). Frequent restarts were needed.
• Horizontal drift of the beam pointing.

The number of bunches during the last night shift kept being changed down to 24 by the device SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020 acting on SA1_XTD2_BUNCHPATTERN/MDL/CONFIGURATOR. This was not the case during the previous shifts, even though the pulse energy was comparable. Looking at the history of SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020, it looks like this device was started at 16:26:25 on Friday, 1st March.

• 10 keV beam
• Detector: JUNGFRAU in the burst mode (16 memory cells)
• Achievements:
  • multiple sfx data sets (lysozyme) with 4 and 8 pulses (at 1.1MHz) per exposure taken

We checked the XTD2 attenuators following the intervention from XRO (compar ewith elog:37, before the intervention).

The wavefront shows similar distortions with the 75 and 150 um as earlier, which is not too surprising since the cracks are vertical.

Attachments 1-5: Beam on SPB_XTD9_SCR with XTD2 attenuator CVDs

Attachments 6-7: Beam on FEL imager with XTD2 75 and 150 um CVDs

X-ray delivery

• 6 keV, 2.6 mJ/pulse, stable delivery

Optical laser delivery

• 800nm PP laser, stable delivery

Achievements/Observations

• NKB focus optimized and PSLITS optimized
• Beam going through IBS camera
• PP laser rough timing checked w.r.t. X-ray
• Catcher and injector motors (X, Z) are aligned well with about 1.5mm offset in X

Issues

• Manifold-MDL issue (reported in Issues tap)
• SPB_IRU_LIQUIDJET/CAM/1 (IRU nozzle insertion camera) is not working. Power cycle required.
• SPB_EHU_Valve closed 2-3 times due to vacuum interlock
• Space (KF50 pipes) in the super-structure was too long for the user injector (nozzle rod). After taking off small spacer, nozzle is very close to the injector valve during insertion. Please check it out whether we can find any better solution or not.

X-ray delivery

6 keV, 2.4 mJ/pulse, stable delivery

Optical laser delivery

800nm PP laser, stable delivery

Achievements/Observations

NKB focus cslits optimized
Some progress with AGIPD background although we still need to improve
Catcher and injector motors (X, Z) are aligned well; with defroster, had under 5mm offset in X
Managed to get a jet and overlap with X-rays.

Issues

Beckhoff flowmeters were disconnected and we were not sure which ports to connect, maybe spent 45min to connect to the right ports and restart device.
Helium flow meter was on but not updating, had to restart it too.
Marco noted actual measured flow at beckhoff VS HPLC set flow was perhaps too low, not sure
Vacuum with injector improved, issue was related to nozzle preparation, however even after this fix the chamber pressure was just over the AGIPD interlock. After tuning gas/liquid flow managed to go under it, but possibly this can also be improved.
Often number of pulses was being changed without us requesting it.
One AGIPD module stopped working after moving quadrant.
DAQ data aggregator crashed a few times, Steffan Hauf will look into this issue after we finished the shift.
Could not move one of the AGIPD quadrant (Q4) due to interlock
Background in Q1 and Q3 needs to be optimized, and/or quadrants needs to be moved.

There are now two watchdogs for the XTD2 attenuator running at the same time

• SA1_XTD2_WATCHDOG/MDL/ATT1_MONITOR020
• SA1_XTD2_OPTICS_PROTECTION/MDL/ATT1

that are configured differently, which results in different pulse limits (see attachment).

**X-ray delivery**

- 6 keV, 3.2 mJ/pulse, stable delivery

**Optical laser delivery**

- 800nm PP laser for jet illumination, stable delivery

**Achievements/Observations**

- NKB focus checked and IUT_Slits optimized
- AGIPD quadrants position finalized
- Measured a lot of water BG runs and few runs with Au nanoparticles (sphere with ~50 nm in diameter)

**Issues**

- SideMic data not saved properly (empty array).
- Found relatively high BG when nozzle is IN since the distance in between nozzle tip & the interaction region is ~100um (even 80 um in some cases).
- DA issue (DAQ was stuck in changing) although all the DA looked healthy. DOC fixed the issue (temporarly) together with Egor, Florian, and Steffen.
- Runs callibration were not working

X-ray delivery
- 9.3 keV, >3 mJ/pulse, stable delivery 

Optical laser delivery
- 800nm PP laser for jet illumination, stable delivery

Achievements/Observations
- Optimised hexapod position for SFX
- Collected granulovirus data for geometry calibration
- Started user sample delivery but clogged the nozzle a few times

X-ray delivery
- 9.3 keV, >3 mJ/pulse, beam down around 30min

Optical laser delivery
- 800nm PP laser for jet illumination, stable delivery

Achievements/Observations
- Collected a lot of data (from midnight to 6am) with different sample concentrations and mixing delays with the users

X-ray delivery

• 12.72 keV, 10 Hz, NOT stable delivery, pointing drastically changed in the afternoon, presumably due to the train-on-demand operation at SASE2

Optical laser delivery

• 487 nm ns laser (not used)
• 800 nm PP laser for sample illumination

Achievements/Observations

• Beam is aligned for IRDa
• lysosyme date was collected for JF4M geometry

Issues

• X-ray pointing changed in the afternoon, we have to figure out why and whether we can avoid losing trajectory
• Beam is currently clipped and EHC_SCR, @Romain Letrun , @Adam Round please take a look tomorrow
• Detector background was not optimal, might require more work tomorrow
• Roadrunner RX motor did not move, in Karabo, was giving error messages @Adam Round please take a look, maybe resetting the roadrunner outside of Karabo can help
• AGIPD MFPGA2 issue (keep acquiring) which was solved by power cycle

Pulse energy fluctuation observed after full train operation in SA2 starts.

X-ray delivery
- 12.72 keV, 10Hz, ~1.4 mJ/pulse, stable delivery 

Optical laser delivery
- 487 nm ns laser for pump-probe
- 800 nm PP laser for jet illumination, stable delivery

Achievements/Observations
- AGIPD BG optimized (DiffPump realigned)
- IRDA focus optimized (EHC_CRL arm10 (1mm pinhole) inserted)
- Completed dark state measurement of the main sample (more than 4 hours of measurement time)
- Collected PP (pump-probe) dataset with 5 us time delay (about 3 hours of measurement time; Jet was quite stable during the 3 hrs measurement)

Issues
- AGIPD module Q3M2 was not healthy. After another power cycle, it recovered.
- X-ray pulse energy was relatively low in the beginning of the shift. It was partially recovered after running the launch optimization by BKR.

DOOCS panels are again incorrectly reporting the FXE shutter as opened when it is in fact closed. This had been fixed after it was last reported in February (elog:24), but reappeared now.

Continuous pointing drift in horizontal direction.

It may trigger a vacuum spike at SPB OH (MKB_PSLIT section) at 21:07 pm.

X-ray delivery

- 12.72 keV, 10Hz, ~1.4 mJ/pulse, continuous beam drift in the horizontal direction observed

Optical laser delivery
- 487 nm ns laser for pump-probe
- 800 nm PP laser for jet illumination, stable delivery

Achievements/Observations
- Collected PP (pump-probe) dataset with 5 us time delay more than 12 hours
- Collected 3 runs (5 minutes run) of dark state in the end of the shift
- Jet was quite stable most of the time

Issues
- Continuous beam pointing drift in the horizontal direction observed
- Vacuum spike at MKB_PSLIT section trip the beam shortly which might be related to the beam pointing drift

X-ray delivery

• 12.72 keV, 10Hz, ~1.4 mJ/pulse, some beam drift in the horizontal direction observed (probably mirror related)

Optical laser delivery

• 487 nm ns laser for pump-probe
• 800 nm PP laser for jet illumination, stable delivery

Achievements/Observations

• Collected PP (pump-probe) dataset with 5 us time delay more than 12 hours
• Jet and data collection was stable most of the time

In-house/commissioning beam time summary night 1

X-ray delivery:
12.7 keV, up to 208 pulses at 1.1MHz, 1.1-1.5 mJ

Optical laser delivery:
PP laser for illumination

Achievements/observations:
Minimised air scattering background on Jungfrau
Collected serial crystallography data with 8, 10 and 13 pulses per memory cell and good hit rates

Issues:
Sample growing again from the catcher cone

In-house/commissioning beam time summary night 2

X-ray delivery:
12.7 keV, up to 288 pulses at 1.1MHz, ~1.2 mJ

Optical laser delivery:
PP laser for illumination

Achievements/observations:
Collected a lot of data: continued 8 pulses per memory cell, around 1h30-2h 1 pulse and 3 runs with 18 pulses.
 

Staff: MS, RB, PS, RdW

In-house/commissioning beam time summary night 3

X-ray delivery:
12.7 keV, up to 288 pulses at 1.1MHz, ~1.3 mJ

Optical laser delivery:
PP laser for illumination

Achievements/observations:
Collected a lot of data with Jungfrau, burst mode 16 memory cells: 
- 1, 4 and 18 pulses/memory cell 
- one run of each setting 1, 4, 8, 10, 13 and 18 pulses/memory cell one after the other
- 8 pulses with 32% transmission
- 4 pulses with half the repetition rate (0.5MHz)

In-house/commissioning beam time summary night 4

X-ray delivery:
12.7 keV, up to 128 pulses at 0.5MHz, ~1.3 mJ, one very short beam down.

Optical laser delivery:
PP laser for illumination

Achievements/observations:
AGIPD timing scan, fine and coarse
A few runs with Jungfrau

Issues:
Some sample delivery issues
Some Jungfrau pipeline issues (will be investigated by DOC during the day)

In-house/commissioning beam time summary night 5

X-ray delivery:
12.7 keV, up to 128 pulses at 0.5MHz, ~1.3 mJ, one very short beam down.

Optical laser delivery:
PP laser for illumination

Achievements/observations:
- AGIPD1.2 tests with different trigger delays, gain modes, signal levels ...
- inhouse proghram completed

Issues:
Some sample delivery issues

Staff: JK, SB, JE, AS, RW, FK

**X-ray delivery:**
- 21 keV, ~500 uJ

**Optical Laser:**
- None

**Achievements / Observations:**
- Measured several Schimadzu runs

**Issues**:
- the beam intensity was fluctuating significantly during the day; 
- Changed the objective of Schimadzu for wider field then the controller stopped working which took over 1 h; before starting working again.
- One of the attenuators has features on the scene; Si 3200

End of User beamtime 5157

X-ray delivery

• 9.3 keV, 3.3 mJ, 190 pulses/train

Optical later delivery

• For jet illumination

Achievements/Observations

• Aligned NKB mirrors, slits
• Collected 8 adaptive gain and 22 fixed medium gain runs with lysozyme
• Collected Copper foil flatfield data and glass scattering with different attenuation and sample-to-detector distances

Issues

• XTD2 attenuator watchdog limited the number of pulses to ~190 with XTD2 CVD inserted

X-ray delivery

• 9.3 keV, 3.3 mJ, 190 pulses/train

Optical later delivery

• For jet illumination

Achievements/Observations

• Aligned NKB mirrors, slits again
• Removed the AGIPD shield
• Moved slit3 and slit4 to a fresh position after damage

Issues

• Litframe finder blocked collecting dark images for AGIPD. The problem was solved after resetting the Litframe finder.
• Slit3 and slit4 were damaged

X-ray delivery

- 9.3 keV, 3.1 mJ, 214 pulses/train at 1.1 MHz and 132 at 0.5 MHz

Optical later delivery
- For jet illumination

Achievements/Observations
- Aligned IRU_SLIT-3 again
- Collected PCS data in fixed MG: a lot of diffraction for 5458, only very few for 7559.
- Collected Lysozyme data in fixed MG and adaptive gain.

Issues
- IRU_SLIT-3 right blade was too close to the beam. Physically shifted it by a few mm towards the dark side and improvised lead aperture added between IRU_SLIT-3 and inlinemic.
- LitFrameFinder cannot recognize dark runs for fixed medium gain. One needs to check box Disable DAQ Filtering (on LitFrameFinder scene) or enable Global Passthrough Mode in LitFrameFinder properties, when takes darks for fixed MG.

X-ray delivery

• 11.56 keV, 1-352 pulses @ 1.13 MHz, 2.6-3.1 mJ.
• Average intensity slowly decaying through the night.
• Quite some intensity gradient through the train (~35%).

Optical laser delivery

• 840 nm, 50 fs, 564 kHz.
• Stable delivery.

Achievements/Observations

• Collected data with AGIPD in various scenarios to better understand the behaviour of the 1.2 modules.

Issues

• SoundPlayer alarms not playing. Only way to fix it was to start the server manually via CLI.

X-ray delivery

• 11.56 keV, 1-352 pulses @ 1.13 MHz, 2.6-3.1 mJ. Stable delivery.
• Intensity gradient with long train could not be reduced by BKR.

Optical laser delivery

• 840 nm, 50 fs, 564 kHz.
• Stable delivery.

Achievements/Observations

• Completed AGIPD programme.
• HIREX aligned for our trajectory and 11.56 keV photon energy. Thanks Naresh!
• Calibration of HIREX.

Issues

• Calibration of AGIPD data is slower than usual. Up to close to three hours calibration time observed for some runs and others taken around 2:30 still not calibrated as of now.

X-ray delivery

• 11.56 keV, 1-352 pulses @ 1.13 MHz, 2.4-3.1 mJ. Stable delivery.
• Intensity gradient with long train better than previous shifts (~20% drop).

Optical laser delivery

• 840 nm, 50 fs, 564 kHz.
• Stable delivery.
• Issue with Treacy compressor motion. Could be solved, but required intervention on site by LAS and EEE OCD.

Achievements/Observations

• Calibration of HIREX GOTTHARD II data working after intervention of DOC/DRC.
• Finished PP laser setup.
• PP laser timed.
• Collected SFX data sets using AGIPD in different gain modes.

Issues

• All AGIPD dark processing attempts failed with random missing modules showing up in the reports.
• We were not able to reach the DRC to discuss the above issue earlier during the shift on either 98089, 98088, or 01715330920. DOC informed at end of the shift.
• HIREX had been removed from the SASE1 DAQ, preventing the GOTTHARD II preview from running, and the DAQ was left into passive state after someone had used it during the day with the XTD9 Shimadzu camera.
• Problem with the Treacy compressor delayed the final PP laser preparation, which then had to be done during beam delivery.

X-ray delivery

• 11.56 keV, 1-352 pulses @ 1.13 MHz, 2.4-3.1 mJ. Stable delivery.
• Intensity gradient with long train now slightly less than 20%.

Optical laser delivery

• 840 nm, 50 fs, 564 kHz.
• Stable delivery.

Achievements/Observations

• Collected TR-SFX data.

Issues

• Challenges with sample delivery.

X-ray delivery

• 11.56 keV, 1-197 pulses @ 564 kHz, 2.4-3.1 mJ.
• Beam down a bit over 1 hour due to accelerator interlock problem.

Optical laser delivery

• 840 nm, 50 fs, 50 pulses @ 141 kHz.
• Stable delivery.

Achievements/Observations

• PP laser timed.
• Completed TR-SFX dataset at 30 ps delay.

Issues

• Scantool stuck in ACQUIRING state. Recovered by restarting the device.
• HIREX data missing for most of the night. GOTTHARD II control and receiver devices found down in the morning. Server on which they are reunning appears to have crashed at 1:36. DOC informed in the morning.

X-ray delivery

• 11.56 keV, 1-197 pulses @ 564 kHz, 2.6 mJ.

Optical laser delivery

• No PP laser output after shift change.

Achievements/Observations

• Collected static SFX data.

Issues

• Unavailability of PP laser prevented collection of TR-SFX data, which was the main goal of the shift.

X-ray delivery

• 6.0 keV, up to 202 pulses at 564 kHz and 101 at 282 kHz, 0.8-1.6 mJ
• Low pulse energy after beam handover. Improved around 10:00.
• ~1.5 hour downtime due to issue with A5.
• Several timing jumps of up to 1.5 ps visible on BAM and confirmed by X-ray/optical laser timing measurement at the instrument.

Optical laser delivery

• 770 nm, up to 202 pulses at 564 kHz and 101 at 282 kHz.
• Stable delivery.

Achievements/Observations.

• Collected time-resolved diffraction from gold and gold-platinum nanorods at multiple laser fluences and over long and short delay scans.

Issues

• AGIPD interlocked overnight and again during the morning due to temperature.
• Had to perform timing measurement and correction multiple times throughout the day due to timing jumps.

X-ray delivery

• 6.0 keV, up to 202 pulses at 564 kHz, 1.8-1.9 mJ.
• Stable delivery except for downtime at start of shift (until 7:15) and two more hours in the morning.

Optical laser delivery

• 770 nm, up to 202 pulses at 564 kHz.
• Stable delivery.

Achievements/Observations

• Collected time-resolved diffraction from gold and gold-platinum nanorods at multiple laser fluences.

Issues

• Vacuum spike in EHU_PSLIT section.
• DAMNIT backend process stuck. A couple of Maxwell nodes were unaccessible. Solved by DOC and DA OCD.
• Issue with two AGIPD modules noticed around 21:20. Could not be recovered before end of the shift. To be further investigated tomorrow morning.

X-ray delivery

• 6.0 keV, up to 202 pulses at 564 kHz 1.3-1.8 mJ.
• Decaying intensity through the day. Recovered once at 15:15 and slowly decaying again after that.

Optical laser delivery

• 770 nm, up to 202 pulses at 564 kHz and 101 at 282 kHz.
• Wavelength drift due to problem with BCC piezo stage. Mitigated by LAS.

Achievements/Observations

• Collected time-resolved diffraction from gold and gold-platinum nanorods/spheres at multiple laser fluences.

Issues

• Problem with the four ASICs in module Q2M1 of AGIPD could not be solved. Operated without those for today. Will be further investigated later on.

X-ray delivery

• 7.0 keV, up to 202 pulses at 564 kHz, ~1.7 mJ.
• Number of pulses initially limited to 178. Promptly addressed by BKR.

Optical laser delivery

• 800 nm, up to 202 pulses at 564 kHz.
• Oscillating wavelength due to known faulty DMBCC stage, but otherwise stable delivery.

Achievements/Observations

• Aligned instrument and collected data with test samples as well as various background signals in preparation for next suer experiment.

Issues

• Changing the ROI of a Zyla appears to affect its timing.

X-ray delivery

• 7.0 keV, up to 202 pulses at 564 kHz, 1.6-1.9 mJ.
• Average pulse energy tuned up to 1.9 mJ at 14:45, but slow continuous decay after that down to 1.7 mJ at the end of the shift.
• 45 minutes downtime in the morning and another 15 minutes to reset the IBFB later on, as coordinated by the PRC.

Optical laser delivery

• 800 nm, up to 202 pulses at 564 kHz.
• Large intensirty gradient through the burst compared to previous days. Recovered after moving the RF phase shifter. LAS informed.

Achievements/Observations

• Collected calibration and background data for upcoming user experiment.

X-ray delivery
- 7.0 keV, up to 202 pulses at 564 kHz, 1.0 mJ.
- Average pulse energy was about 1.9 mJ on Tuesday and Wednesday. It has to be discussed with BKR tomorrow morning.

Optical laser delivery
- ns laser for jet illumination

Achievements/Observations
- BG measurement with water, buffer low salt
- Lysozyme measurement for geometry file
- Data collection for six different sample concentrations [2 x 15 minutes runs]

Issues
- Pulse energy is very low as compared with Tuesday and Wednesday in this week.
- Damage found on IRU_SLIT4 Top and Bottom blades.
- NKB focus moved by ~20um at around 15:00 (during the handover period). SideMic X moved +20um to compensate it.

X-ray delivery
- 7.0 keV, up to 202 pulses at 564 kHz, 1.0 mJ.
- X-ray beam pointing changed at around 18:12 and it affects nano KB focus (from ~200nm to ~1um).

Optical laser delivery
- ns laser for jet illumination

Achievements/Observations
- Detector moved the most upstream position (~1.5 m sample to detector distance)
- BG measurement with water, buffer low salt
- Silver Behenate measurement for geometry file (mostly for detector distance)
- Data collection for different sample (pure) concentrations

X-ray delivery

• 7.0 keV, up to 202 pulses at 564 kHz, 1.0 mJ.

Optical laser delivery

• ns laser for jet illumination

Achievements/Observations

• Data collection for user sample through column

Issues

• Pressure spike while rotating nozzle tripped AGIPD
• Temperature of ASICs of Q2M1 module too low and not easy to turn it on (issue solved after changing the bath temperature to -27 degreeC)
• Julabo chiller issue again (In the end, set temperature changed to -29.5 degreeC and continued the experiment)
• Another 2 ASICs in Q2M2 and Q2M3 seem unhappy

**X-ray delivery**

• 7.0 keV, up to 202 pulses at 564 kHz, 1.0 mJ.

**Optical laser delivery**

• ns laser for jet illumination

**Achievements/Observations**

• Data collection completed for all the requested samples from user.

**Issues**

• Run repeater behave once abnormally.  Run repeater started with 30 minutes duration, but the first run (run #238) stopped after 40 seconds and next run (run #239) started.

X-ray delivery
- 9.22 and 11.0 keV, 1-352 pulses @ 1.1 MHz, 600-1400 uJ.
- Large intensity gradient with long trains (>70% drop with 352 pulses @ 1.1 MHz).
- Very unstable delivery from 20:30 until 23:00.

Optical laser delivery
- 800 nm, 50 fs, 1-113 pulses @ 564 kHz.
- Stable delivery

Achievements/Observations
- Instrument alignment.
- AGIPD timing scan.
- AGIPD flat field measurement at 1.1 MHz.

Issues
- Twice more time required to collect AGIPD flat fields due to large intensity gradient in the train.
- XTD2 1.2 mm CVD unusable. Initially stuck in the beam, but could be retracted with the help of DOC/PLC OCD.

X-ray delivery

• 11.0 keV, 1-352 pulses @ 1.1 MHz, 600-1100 uJ.
• Even larger intensity gradient with long trains than yesterday.

Optical laser delivery

• 800 nm, 50 fs, 1 pulse.
• Stable delivery.

Achievements/Observations

• Finished AGIPD programme.
• Started JUNGFRAU programme.

Issues

• Poor beam delivery with more than a few bunches per train.

X-ray delivery

• 11.0 keV, 1-351 pulses @ 1.1 MHz, 300-1100 uJ.
• Continuously decaying average intensity through the day.
• Large intensity gradient with long trains.

Optical laser delivery

• 800 nm, 50 fs, 1-113 pulses @ 564 kHz.
• Stable delivery.

Achievements/Observations

• Completed detector programme.

Issues

• Difficulties with AGIPD online calibration. Solved by DOC/CAL.

The SASE1 HIREX interlock is not working as expected.

At the position 'OUT' (step 2 in HIREX scene) and while none of the motors are moving, the interlock SA1_XTD9_HIREX/VDCTRL/HIREX_MOVING is triggered, which prevents the use of more than two pulses.

SA1_XTD9_HIREX/VDCTRL/HIREX_MOVING remains in INTERLOCKED state while moving HIREX in (expected behaviour) and finally goes to ON state when HIREX is inserted, thus releasing the MODES and MODEM interlocks and allowing normal beam operation.

Summary 06.08 - 10.08

X-ray delivery

• 9.3 keV, 1-352 pulses @ 1.13 MHz and 564 kHz, 1.4-1.8 mJ
• 11.56 keV 1-352 pulses @ 1.13 MHz and 564 kHz, 0.7-1.0 mJ.
• Large impact on SASE1 when SASE3 used pulse on demand at the start of the week. Ok since we were setting up, but would be problematic during measurement.
• Overall stable delivery except for the point above.
• Notable downtimes:
  • Thursday afternoon 08.08, beam down during ~1 hour for a ZZ in XTD9 to fix the monochromator motion.
  • Thursday evening 08.08, beam down during the last 45 minutes of the shift following wavelength change request of SASE3.
  • Friday afternoon 09.08, beam down for ~2 hours due to ZZ to fix communication with a device.
  • Saturday morning 10.08, beam down during 45 minutes due to problems with A5.

Optical laser delivery

• 800 nm, 50 fs, 1-202 pulses @ 564 kHz and 141 kHz.
• Laser delivered from Thursday afternoon, as agreed with LAS. One of the issues addressed, but performances still sub-par and more work is required.

Achievements/Observations

• Collected data for all but one PCS proposals. Sample for last PCS is expected to be delivered to EuXFEL on Monday.
• Setup of HIREX for 9.3 and 11.56 keV.
• Completed laser setup for IHR.
• Established laser/X-ray timing.

Issues

• Multiple issues with the online calibration pipeline. It is regrettable that a new version including breaking/major behaviour changes has been deployed after the bring up days, where everything had been succesfuly tested.
  • Constants failing to load after changing detector configuration in the calibration manager. Hotfixed by CAL OCD.
  • AGIPD double pixels mask not correctly applied. Hotfixed by CAL.
  • Changing number of memory cells in the calibration manager, but without loading new constants, changes drastically what is displayed in the AGIPD corrected preview. DOC informed, fix pending.
  • Independent settings for the different pipeline preview data have been implemented, but not communicated. This is very dangerous for the detectors. We have implemented a workaround to restore the previous behaviour.
  • Difficulties in reliably getting a preview of all AGIPD modules even with settings we used with the previous version of the calibration pipeline.
  • HIREX correctino devices had been renamed in the SASE1 topic, but no information was provided to instruments resulting in missing clone devices in instrument topics.
• Offline calibration throws error for most runs because there are sequence files that contain no valid trains produced at the end of the runs. This is not really a problem and would require a less ambiguous statement in myMdC as this makes it difficult to spot when an actual error occurs. DOC/CAL informed.
• DAMNIT could not be started on non-display Maxwell nodes. It turns out an additional argument is required following the OS update during the summer maintenance period, but this is not documented. The solution was provided by DA OCD.
• Ice detector generating many false positive and sending of report not working. To be addressed with DA on Monday.
• Graphical bug in Karabo Image Graph widget when the same data are displayed in two scenes. More detail will be provided to CTRLS.
• XTD9 PBLM at the nominal 'out' position was not fully out and the beam could still be seen on it.
• HIREX interlocks not working as expected and preventing the use of more than two pulses when HIREX is supposedly extracted from the beam path. Mitigated by the PRC.

X-ray delivery

11.56 keV 202 pulses @0.5 MHz, ~1.0 mJ

some downtime due to fire alarm in the tunnel

Optical laser delivery

800 ns laser for jet visualization

PPLaser for time-resolved data collection

Achievements/Observations

• PPLaser timing and alingment done
• Collected several hours of PYP sample with 30ps laser delay

Issues

• DAMNIT issue solved with OCD DA
• some online calibration issues from before persisted

X-ray delivery

12.40 - 12.82 keV, ~1.1 mJ

Optical laser delivery

800 ns laser for jet visualization

Achievements/Observations

• 800 ns laser for jet visualization (aligned but not timed)
• Jungfrau timing done
• IRDa focus done
• IRDa fully aligned
• Sample delivery ready
• Powder data collected

X-ray delivery

12.40 - 12.82 keV, ~1.1 mJ

Optical laser delivery

ns laser for jet visualization
ns laser for pumping ready but not used

Achievements/Observations

• Focus IRD
• ns time and spatial overlap
• Collected lyzo data
• Collected dark rhodopsin data
• New opaque foil installed on cube X-ray exit side

Issues

• Roadrunner motor issues solved after manual intervention and computer restart

X-ray delivery

-12.40 - 12.82 keV, XDT9 XGM ~0.81 mJ@ end of shift

Optical laser delivery

-800 nm laser for jet visualization
-487 nm ns laser for pumping

Achievements/Observations

-Focus IRD
-Collected 10 ns SFX data

Issues

-AGIPD chiller was down

X-ray delivery

-12.40 - 12.82 keV, XDT9 XGM ~0.85 mJ, 3 hours interruption by vacuum issue on XTD9

Optical laser delivery

-800 nm laser for jet visualization
-487 nm ns laser for pumping

Achievements/Observations

-Focus IRD
-Collected SFX data
 

Issues

-AGIPD chiller was down
-Issue with tunnel vacuum valves, at around 7 pm, ended shift early

X-ray delivery

-12.40 - 12.82 keV, ~1.3 mJ

Optical laser delivery

-800 nm laser for jet visualization
-525 nm ns laser for pumping

Achievements/Observations

-Focus IRD
-Collected SFX data

Issues

-SPB_EHC_VAC/DCTRL/GAUGE_FR_1_Value triggered interlock, we could not open gate valve : SPB_IRU_VAC/VALVE/VALVE_INLINE_3

X-ray delivery

• 12.78 keV, ~1.3 mJ

Optical laser delivery

• 800 nm laser for jet visualization

Achievements/Observations

• Collected SFX data
• Calibration date for HIREX measured (at different X-ray energy)

X-ray delivery

12.8 keV, ~0.8 mJ

Achievements/Observations

- Data collection with fixed target

Issues

- M1 and M2 position on SPB/SFX Setpoint J was far from optimal
- Beam position in X was drifting to minus direction all time 

X-ray delivery

• 12.8 keV, 1 pulse, ~0.7 mJ

Achievements/Observations

• Except for some downtime due to beam down, majority of the time was for data collection
• 6 windows of a chip collected, with variations in shots spacing (30 um and 60 um) and rotation

Issues

• Something is still causing beam drift constantly, even though we can correct it with the feedback.

X-ray delivery

• 12.8 keV, 1-20 pulses at 47 kHz, 500-700 uJ.
• Pulse energy constantly decaying since ~3:00.

Achievements/Observations

• Continuous SFX data collection since 00:10.

Issues

• Apparent mechanical issue with one motor. To be investigated before the restart of beam delivery.
• Beam horizontal drift still present, but not as pronounced as earlier this week.

X-ray delivery

8.951 keV, 1.6 mJ

Achievements/Observations

Beam trajectory and focusing established for 8.951 keV
HIREX inserted for energy reference for calibration tomorrow
Temporary beamstop verified position for direct beam.
Mirror tank inserted to beam path

On Shift: TS, FT, CK, AR, PS

X-ray delivery
8.951 keV, 1.6 mJ up to 200 pulses at 0.5 MHz (intesity drops off at end of train)

Achievements/Observations
Beam trajectory and focusing re-established at 8.951 keV
Temporary beamstop verified position for sweeping - confirmed by eric
Beam sweeping in synchronisation with pulse trains observed!
slit window assembly remounted and beam is going through to back of hutch in perparation for optimisation next shft

X-ray delivery
8.951 keV, 1.2 mJ up to 30pulses at 113 kHz

Achievements/Observations
Beam trajectory at 8.951 keV
IRDa slits left/right blades connected
Sweeped pulses could be observed on fluorescent paper in front of the Downstream screen
Mirror synchronisation parameters could be found
Jungfrau halves were closed and the downstream cone installed
Background was minimised using the IRDa slits

X-ray delivery
8.951 keV, 1.6 mJ up to 16 pulses at 113 kHz

Achievements/Observations
More than 1h of beam sweeping with lysozyme crystals
Several runs of vitamin C beam sweeping data

Issues:
Several synchronization methods were tested for the mirrors but without improving it
 

600um thick CVD diamond (XTD2 attenuator) seems damaged

Attachment 1: 600um thick CVD diamond (Arm 4) OUT

Attachment 2: 600um thick CVD diamond (Arm 4) IN

X-ray delivery

• 6 keV, ~2.8 mJ, stable delivery

Optical laser delivery

• ns laser for particle visualization

Achievements/Observations

• NKB focus optimized
• particle injection tested with gold nanoparticles
• ns laser roughly aligned with particle beam
• Julabo chiller (AGIPD) is still on target (-32 degree)

Issues

• XTD2_ATT Arm4 (600um thick CVD) is damaged (most probably last week)

X-ray delivery

• 6 keV, ~2.7 mJ, stable delivery

Optical laser delivery

• ns laser for particle visualization

Achievements/Observations

• NKB focus optimized
• ns laser aligned with particle beam
• particle injector position optimized
• measured 3 samples: roughly 2 hours per each sample
• one sample was measured with and without He gas
• Julabo chiller (AGIPD) is still on target (-32 degree)

Issues

• Beam drift in vertical direction: Rz has to be corrected during the measurement and moved by 0.2 mm in total in whole day (~18 hours)

X-ray delivery

• 6 keV, ~3.0 mJ, stable delivery

Optical laser delivery

• ns laser for particle visualization

Achievements/Observations

• measured 1 sample roughly for 9 hours (hit rate went down in the afternoon)
• Julabo chiller (AGIPD) is still on target (-32 degree)

Issues

• NKB mirror drifted in vertical direction: it was optimized again at around 9pm
• extremely low hit rate even with sucrose (extremely few particles were observed in sidemic, 1 particle within a few seconds): maybe the aerodynamic lens should be cleaned

**X-ray delivery**

- 6 keV, ~2.8 mJ, stable delivery

**Optical laser delivery**
- ns laser for particle visualization

**Achievements/Observations**
- measured user sample for almost whole day (about 100 runs x 5 minutes)
- Julabo chiller (AGIPD) is still on target (-32 degree)

**Issues**
- MKB Ion pump error (issue fixed after resetting the device)
- NKB focus loosed. Focus reoptimized twice during the d2 shift

X-ray beam is continuously drifting in the vertical direction (XTD9_XGM)
It affects our NKB focus and focused beam is also drifting continuously.

**X-ray delivery**

- 6 keV, ~2.8 mJ, continuous beam drift in vertical direction (both XTD2 & XTD9 XGMs)

**Optical laser delivery**
- ns laser for particle visualization

**Achievements/Observations**
- measured sample for almost whole day (about 100 runs x 5 minutes)
- Julabo chiller (AGIPD) is still on target (-32 degree)

**Issues**
- Continuous beam drift in the vertical direction: NKB focus was optimized almost every 2 hours
- Sudden intensity drop in SA1 when SA2 or SA3 changes the number of pulses.

X-ray delivery

    6 keV, ~2.8 mJ, continuous beam drift in both direction

Optical laser delivery

    ns laser for particle visualization

Achievements/Observations

    DAta collection for CDI
    Julabo chiller (AGIPD) is still on target (-32 degree)

Issues

    Continuous beam drift in the both direction: NKB focus was optimized almost every 2 hours

X-ray delivery

12.4 keV, ~1.4 mJ

Achievements/Observations

Beam was focused on a YaG, alignment finished up to JF4M

X-ray delivery

12.5 keV, ~1.6 mJ

Achievements/Observations:
Collected almost 40 runs of the priority sample with 7 pulses per train using the Polypico injection system.

Issues:
Jungfrau controller issue when changing back to single cell adaptive to collect powder data with new geometry (solved together with DOC).
Pipeline issue after requesting single cell correction - still the same bug in the calibration pipeline, the pipeline has to be re-instantiated after the change.

End of in-house 8496

X-ray delivery

15.13 keV 30 pulses @0.5 MHz and 64 pulses (customized pattern), 0.85-1.0 mJ stable delivery

Achievements/Observations

• We used the customized pattern and it worked well, thanks to everyone involved
• One fuse was blew, TS OCD had to come on site
• Interlock was brocken accidently with the shutter open, which tripped the beam in all SASEs

X-ray delivery

19 keV, ~550 uJ, 1 - 128 pulses, stable delivery

Optical laser

N/A

Achievements/Observations

Sample test with both setup
Siemens star resolution measurement 

Issues

No major issue

X-ray delivery

19 keV, ~450 uJ, 1 - 128 pulses, stable delivery

Optical laser

N/A

Achievements/Observations

Lots of data collection

Issues

No major issue today

X-ray delivery

19 keV, ~ 400  - 450 uJ, 1 - 128 pulses, stable delivery

Optical laser

N/A

Achievements/Observations

Lots of data collection, 659 runs total, lot of nice results, first obsertvations of 3D printing process

Issues

No major issue today

Moning shift summary (31.01 and 01.02)

Acievements:
- SRA recalibrated, cross position on FEL imager found
- VSLIT in SA3 check calibration
- LEPM angle changed for LRGR. Calibration of photon energy server done with help of Ne in GATT.
- Several configurations for SCS have been adjusted/established
 

Late shift 07.02.24
Laurent, Martin

Continued with laser preparation

• optimized SHG
• changed lens to f = 600 mm
• knife edge scans
• found optimum lens position (95 mm) for w0 = 70 um
• fluence characterization --> up to more than 2 J /cm^2
• Coarse timing
• Fine timing

night shift summary 07.02.24

Andreas, Giuseppe

program was to determine horizontal an vertical beam size and then create some single laser shot patterns to determine fluence settings for LIPSS.

PROBLEMS:

• knife edge insensitive to beam sizes smaller than 20um due the large beam jitter of the FEL.
• moved to the camera and curved gratings but then encountered vacuum issue with scroll pump down since noon 07.02. fixed by replacing scroll pump.
• useful camera data could not extracted as we cannot control the TPI shutter and have multiple exposures of the camera. STPulser Macro need to be fixed.
• SCS_XTD10_TPI/DCTRL/SHUTTER in the SCS topic is not found after shutting device down when tracking down device issues.

Night shift summary
Andreas, Giuseppe

• we finished to measure sample 17 and 41 in CARD 06_Y2018F04
• we transferred sample Card_07_Y2018F19
• we started measuring sample 500 nm Ti #26

Problems:

• the frame could not be grabbed properly from the transfer arm, so we decide to vent and transfer manually
• we realized that FEL beam drifted horizontally, we corrected by moving M2 pitch. (The beam did not move at XGM in XTD10, so it must have been some mirror movement)

shift summary
Andreas, Giuseppe

• continued pumpprobe measurements
• around 5:30 we could not see any image at the MTE3 camera (after sample transfer), we tried to fix it but did not work
• we tried to remove the samples from the LLC, the LLC is vented but the lift cannot be moved, needs to be repaired during the day.

Late Shift summary

Achievements:
- recovered the beam overlap
- changed lens position to have larger OL beam
- spatial overlap on YAG
- measured RIXS data delay scan at fixed photon energy

Problems:
- none

## Shift summary night July 28th-29th

Achievements:
- we saw signal on both MCPs (with a voltage of around 1500 V). We used the MCPs to measure XAS at Cu L edge at different theta angles from 120 deg (specular to MCPs) to 70 deg (facing hRIXS). At grazing angles the spectrum is a bit distorted, but just to see the edge is perfectly fine. If there is a diode on the way, the MCP sees  still some fluorescence.
- we saw the direct beam at APD 8 with transmission 0.02 % and 55 V bias voltage.
- we saw signal also on diode 4 and APD5
- the detectors connected are: diode 2, diode 4, APD 5, MCP_BIG and MCP_SMALL
- we looked for signal at rixs on CoO, but it was too weak so we asked BKR to go to vertical polarization
- beam with vertical polarization was transported to the sample
- continie the optimization of hRIXS at Su and hr grating

Problems:
- XAS processor was not working, tried to change date source, but it didn't help.
- diode 6 could not see any signal, probably because it is obstractad by thte antenna back

Program for next shift:
 - finish optimization at Cu edge hr grating

- optimize for Fe, at this geometry or rotated hRIXS.

RC / BKR informs : Beam is down (since 12h10), all beamlines affected

The reason is failure of a quadrupole (power supply) downstream of the dogleg.
The MPS crew is working on recovery, access to XTL ongoing.
RC will inform when new estimates come in on when we can restart beam operation.

Currently affected on Photon side:
- SA1: Naresh (XPD) and Marco (DET) working on Gotthard-II commissioning. Naresh is present at BKR.
- SA2/SA3: Antje + Mikako (XRO) working on SA2 mirror flipping tests and SA3 exit slit check. Both are located at SXP control hutch.
Other XRO members will come in for the afternoon/evening shift work. Antje will inform them via chat.

A ZZ is taking place today, 31.01 at 11:30, in XTD9 to verify and fix the M3 Pop-in screen which is currently blocking the beam. It is not yet clear if this is an electrical or mechnical problem.  

** Update 13:05: ZZ intervention is finished and successful. A mechanical issue due to a loose coupling between motor and spindle was fixed. Machine is running again and SPB/SFX can use the beam.

-Machine status

the cold compressor failure occurred at ~16.30hr today. And it will take around 35 hr to fix it. I will update via email (local contacts for experiments) and Operation Elog tomorrow with more details.

Naresh

SASE1 bunch number was reduced to 10 pulses for no obvious reason. 

Traced it back to a malfunctioning of the watchdog for the solid attenuator (see attached error message)

Deacttivated the watchdog for SASE1 solid attenuator. Re-activated it again.

Issue: SASE1 (SPB?) was requesting 200 pulses at 6 keV with full power with all solid attenuators in the beam. This was a potentailly dangerous situation for the solid attnenuator (!)

After that the team left to go home. Nobody is available at FXE.  Set pulse number to 1.

Checked status of solid attenuator. Seems that plate 1&2 have some vertical cracks (see attached picture)

11:15 SASE1/SPB called that they observed and issue with the SASE1 XGM/XTD2: about every 35 seconds the power reading jumps from 4 Watts to zero and back. It seems to be related to some automated re-scaling of the Keithlyes, which coincides. The fast signal and downstream XGMs do not show this behavior. The concern is that this may affect the the data collection later today. 

12:15 The XGM expert was informed via email, however, there is no regular on-call service today. If this cannot be fixed in the next few hours, the PRC advises to use in addition the reading of the downstream XGM for data normalisation. 

13:15 Theo fixed the problem by disabling auto-range. Needs follow-up on Monday, but for this weekend should be ok. 

BKR called PRC around 06:10AM to inform on vacuum issues at SA1 beamline leading to a lost beam permission for the North Branch.

VAC-OCD found an ion pump in the PBLM / XTD9 XGM presented a spike in pressure that triggered valves in the section to close.

After reopening valves the beam permission was restored; further investigations and monitoring in coming days will continue. Currently all systems work normally.

https://in.xfel.eu/elog/Vacuum-OCD/252

Yesterday two motors of the SASE2 seeding mono were damaged. The mono was exchanged in the afternoon and pumped down. Currently, an issue with the vertical cystal motion is being investigated.  The vacuum improved much faster than expected after this intervention, so the hope is that the tunnel can be searched soon and electrons could go through SASE2 already this afternoon. Watch for updates in the program for SASE2 in the DESY logbook under 'Schedule'.     

The intra-bunchtrain feedback (IBFB) was not active until today afternoon.
Although not strictly required or requested for the current week experiments,
it was decided to turn on this feedback, which should generally be active to
stabilize and improve the arrival time of the photon pulses along pulse trains.

In order to avoid complications during long data acquisitions, the PRC checked back with all ongoing experiments,
identified a suitable time slot, and the IBFB was activated without issues at aound 16h45.

20.4.2024 / 22h51:
DOC informed PRC that the SASE2 topic broker had to be restarted.
The EPS power limit was reset by PRC to the previous value of 40 W.

DOC and DRC are working on the restart. HED is in hardware setup configuration change (not currently taking beam, thus no harm to experiment),
will start taking beam with new users sunday 7am, but would need before this some test runs for the detectors. 

Currently the SA2 topic watchdogs (for ATT, CRL, imagers) cannot be restarted - they crash quickly after restarting.
DOC/DRC are investigating this issue. Until the watchdogs are successfully restarted, extra care must be taken when using ATT, CRL, imagers in SA2.

Update 21.4.2024 / 1h43 by DRC:
In a similar fashion to the incident of last 5.04, the Broker process for the SA2 and LA2 topic has restarted.
This was noticed by the DOC crew and by the HED crew in parallel.
With the help of the CTRL OCD (Micheal Smith) we restarted both topics. As a lesson learned from the last interruption, the POWER_LIMIT
middlelayer device has been set up by the PRC, so that when the DOOCS bridges will be restarted by the DESY-MCS personnel,
the maximum power will not be set to 0.
Please check in due time if all systems are working as intended.

V0 valve for SA2 closed as the pump P22090I at HED branch enters in wrong state. This closed nearby vacuum valves for HED branch and consequently V0. Vacuum levels at the section remained in 10^-9 all time.

Thanks to VAC-OCD Denis Finze we removed this pump signal from interlock to allow operation in SA2 at MID branch. Further investigation will follow on Friday.

840 pulses not possible at SXP due to big brother settings for the EPS Power Limit rule. The minimum pulse energy was assumed to be 6 mJ while the actual pulse XGM reading is 550 uJ.
Solution: The minimum assuemd pulse energy had been lowered from 6 mJ to 2 mJ. This shall be reverted on Tuesday next week.

Dear all,

Due to a probable issue at the SA1 Transmissive imager, the "OUT" and "YAG" position indication signals seems to be not responsive.

After the FXE team noticed that MODE S was interlocked, the root cause has been narrowed down to this component. Later, further investigation with A. Koch & J. Gruenert (XPD) has confirmed this fact. Apparently to recabling works performed during yesterday's ZZ may have something to do with this issue.

Once that the complete retraction of all the screen has been jointly confirmed via DOOCS and Karabo, the affected interlock condition (see attached picture) has been disabled for the rest of the week to allow normal operation of SA1 (release of MODE S and MODE M).

Further investigation and the eventual issue resolution is expected to happen on Tuesday next week via ZZ-access.

With best regards,

Raul [PRC@CW19]

In SASE2 undualtor cell 10 has issues. Suren K. will do a ZZ at 11:30 to fix it (if possible). Estimated duration: 1-3 hours. 

Update 12:30: The problem is more severe than anticipated. Suren needs to get a replacement driver from the lab and then go back into the tunnel at about 14:00 

In principle, the beam could be use dby HED until 14:00

Update 13:30: Beam was not put back, because energizing and de-energizing the magnets would take too much time. Whenever Suren is ready, he will go back into XTD1 and try to fix the problem. 

Udate 17:30: Suren and Mikhail could do the repair and finish now th zz. A hardware piece (motor driver) had to be exchanged and it turned out that the Beckhoff software on it was outdated. To get the required feedback from Beckhoff took considerable time, but now everything looks good.  

The control rack of the soft mono at SASE3 has some failure. Leandro from EEE will enter the XTD10 at 12:00. 

Beam will be interupted for SAS1 and SASE3.

Estimate to fix it (if we are lucky) is one hour. 

Udate 12:30: Problem is solved (fuse was broken, replaced, motors tested)

Update 13:30: Beam is still not back, because BKR has problems with their sequencer

SASE1: Solid attenuator att5 is in error. When it is tried to move it out, the arm lifts, but does not reach the final position before the software time-out. We  increased the software time-out from 4 seconds to 7 seconds, which didn't help. We suspect that there is a mechanical problem that prevents the att5 actuator to reach the out position. 

SPB states that it is ok to leave the attenuator arm in over the weekend, because they run mostly detector tests and don't require full intensity.  

EPS Interlock change for Mode30:
the 3 HIREX Interlock conditions had been disabled

Issue with SA3 Gas attenuator (SA3_XTD10_GATT)

9h06: SQS informs PRC that the Argon line in GATT is not reacting on pressure change commands, no GATT operation, which is required to reduce pulse energies for alignment. VAC-OCD is informed.

ZZ access is required to check hardware in the tunnel XTD10.
SA1 / SPB is working on other problems in their hutch (water leaks) and don't take beam.
SA3 / SQS will work with the optical laser.
SA2 / MID will not be affected: there is also an issue in the hutch and MID currently doesn't take beam into the hutch, and would anyhow only very shortly be affected during the moment of decoupling the beamline branches.

9h45: BKR decouples North Branch (SA1/SA3) to enable ZZ accesss for the VAC colleagues.

ZZ access of VAC colleagues.

10h12: The problem is solved. A manual intervention in the tunnel was indeed necessary. VAC colleagues are leaving the tunnel.

• Beam down SASE1: 9h43 until 10h45 (1 hour no beam), tuning done at 11h45, then 1.7mJ at 6keV single bunch.
• Beam down SASE3: 9h43 until 10h56 (1.25 hours no beam). Before intervention 4.2mJ average with 200 bunches, afterwards 2.5mJ at 400 eV / 3nm single bunch.

Bunch pattern SA1 / SA3

SQS requests to change the bunch pattern.
Currently SA1/SA3 run interleaved as 131313...
The first bunch in the train (for SA1, non-lasing in SA3) nevertheless is visible as a small pre-peak in the diode signal which SQS uses to check the timing.
Therefore SQS requests to change to 31313131....

It was checked with SA1 (SPB/SFX) and the accelerator experts that this should not cause any issues (AGIPD detector is running on dynamic trigger),
therefore it was agreed that SPB/SFX informs BKR when there is a good moment without data acquisition, and BKR will make this change after informing SQS.
This change will happen within the next hour (as of 15h20).

A2 failure - downtime

The machine / RC informs (11h22) that there is an issue with the accelerator module A2 (first module in L1, right after the injector).
This causes a downtime of the accelerator of 1 hour (from 10h51 to 11h51), affecting all photon beamlines.

Beam is back. Current status:

SASE1 (SPB/SFX)

• 2.72 mJ, 352 bunches/train, 6 keV

SASE2 (MID)

• 880 uJ, 1 bunch/train, 8 keV

SASE3 (SQS)

• 4.56 mJ, 1 bunch/train, 400 eV (3.1nm)

14:40hr

FXE and HED reported the laser synchronization issue. Informed to Machine RC. 

LbSync team couldn't fixed this issue remotely and on there way to Schenefeld to fix this issue.

Naresh 

PRC wk 41

Desy Laser group has fixed the issue about the laser Synchronization.

https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2024/41/07.10_M&pos=2024-10-07T14:51:12

Naresh

PRC

wk41

- While FEL IMG with YAG screen was inserted in SA1, EPS/MPS was not limited to the expected 2 bunches, but to 30 bunches; 
- It was found the condition #3 ("SA1_XTD2_VAC/Switch/SA1_PNACT_Close") at the EPS Mode-S was disabled which should be always enabled; 
- Condition #3 has been enabled again and now SA1 is limited to 2 bunches, as expected when YAG screen is inserted. 

Further investigations will be done to find why and when it was disabled. 

SA2 cell#5 undulator segment moved at ~09:45AM Fri to 25mm without aparent reason and undulator experts need a ZZ to exchange power supply; FEL instensity dropped from 230uJ to 60uJ s consequence.

RCs are preparing south branch for the iminent ZZ. HED is informed.

**Update: ZZ finished by 12:10; recovered performance and re-started delivery by 13:30.

20h16: beam down in all beamlines due to a gun problem
20h50: x-ray beams are back
Shortly later additional 5 min down (20h55 - 21h00) for all beamlines, then the gun issue is solved.

After that, SA1 and SA3 are about back to same performance, but SA2 HXRSS is down from 130 uJ before to 100 uJ after the downtime.
Additionally, MID informs at 21h07 that no X-rays at all arrive to their hutch anymore. Investigation with PRC.

At 21h50, all issues are mostly settled, MID has beam, normal operation continues.
Further HXRSS tuning and preparation for limited range photon energy scanning will happen in the night.

Broken Bridges

Around 21h20 the XPD group notices that several karabo-to-DOOCS bridges are broken.
The immediate effect is that in DOOCS all XGMs in all beamlines show warnings
(because they don't receive important status informations from karabo anymore, for instance the insertion status of the graphite filters nrequired at this photon energy).

PRC contacts Tim Wilksen / DESY to repair these bridges after PRC check with DOC, who confirmed that on karabo side there were no issues.
Tim finds that ALL bridges are broken down. The root cause is a failing computer that hosts all these bridges, see his logbook entry;
https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2024/47/22.11_a&pos=2024-11-22T21:47:00
That also explains that HIREX data was no longer available in DOOCS for the accelerator team tuning SA2-HXRSS.

The issue was resolved by Tim and his colleagues at 21h56 and had persisted since 20h27 when that computer crashed (thus 1.5h downtime of the bridges).
Tim will organize with BKR to put a status indicator for this important computer hosting all bridges on a BKR diagnostic panel.

Broken Radiation Interlock

Beam was down for all beamlines due to a broken radiation interlock in the SPB/SFX experimental hutch.
The interlock was brocken accidently with the shutter open, see https://in.xfel.eu/elog/OPERATION-2024/203
Beam was down for 42 min (from 3h00 to 3h42) but the machine came back with same perfomrmance.

SA1 Beckhoff issue

SA1 blocked to max. 2 bunches and SA3 limited to max 29 bunches
At 9h46, SA1 goes down to single bunch. PRC is called because SA3 (SCS) cannot get the required >300 bunches anymore. (at 9h36 SA3 was limited to 29 bunches/train)

FXE, SCS, DOC are informed, DOC and PRC identify that this is a SA1 Beckhoff problem.
Many MDLs in SA1 are red across the board, e.g. SRA, FLT, Mirror motors. Actual hardware state cannot be known and no movement possible.
Therefore, EEE-OCD is called in. They check and identify out-of-order PLC crates of the SASE1 EPS and MOV loops.
Vacuum and VAC PLC seems unaffected. 

EEE-OCD needs access to tunnel XTD2. It could be a fast access if it is only a blown fuse in an EPS crate.
This will however require to decouple the North branch, stop entire beam operation for SA1 and SA3.
Anyhow, SCS confirms to PRC that 29 bunches are not enough for them and FXE also cannot go on with only 1 bunch, effectively it is a downtime for both instruments.

Additional oddity: there is still one pulse per train delivered to SA1 / FXE, but there is no pulse energy in it ! XGM detects one bunch but with <10uJ.
Unclear why, PRC, FXE, and BKR looking into it until EEE will go into the tunnel.

In order not to ground the magnets in XTD2, a person from MPC / DESY has to accompany the EEE-OCD person.
This is organized and they will meet at XHE3 to enter XTD2 from there.

VAC-OCD is also aware and checking their systems and on standby to accompany to the tunnel if required.
For the moment it looks that the SA1 vacuum system and its Beckhoff controls are ok and not affected.

11h22: SA3 beam delivery is stopped.
In preparation of the access the North branch is decoupled. SA2 is not affected, normal operation.

11h45 : Details on SA1 control status. Following (Beckhoff-related) devices are in ERROR:

• SA1_XTD2_VAC/SWITCH/SA1_PNACT_CLOSE and SA1_XTD2_VAC/SWITCH/SA1_PNACT_OPEN (this limits bunch numbers in SA1 and SA3 by EPS interlock)
• SA1_XTD2_MIRR-1/MOTOR/HMTX and HMTY and HMRY, but not HMRX and HMRZ
• SA1_XTD2_MIRR-2/MOTOR/* (all of them)
• SA1_XTD2_FLT/MOTOR/
• SA1_XTD2_IMGTR/SWITCH/*
• SA1_XTD2_PSLIT/TSENS/* but not SA1_XTD2_PSLIT/MOTOR/*
• more ...

12h03 Actual physical access to XTD2 has begun (team entering).

12h25: the EPS loop errors are resolved, FLT, SRA, IMGTR, PNACT all ok. Motors still in error.

12h27: the MOTORs are now also ok.

The root causes and failures will be described in detail by the EEE experts, here only in brief:
Two PLC crates lost communication to the PLC system. Fuses were ok. These crates had to be power cycled locally.
Now the communication is re-established, and the devices on EPS as well as MOV loop have recovered and are out of ERROR.

12h35: PRC and DOC checked the previously affected SA1 devices and all looks good, team is leaving the tunnel.

13h00: Another / still a problem: FLT motor issues (related to EPS). This component is now blocking EPS. EEE and device experts working on it in the control system. They find that they again need to go to the tunnel.

13h40 EEE-OCD is at the SA1 XTD2 rack 1.01 and it smells burnt. Checking fuses and power supplies of the Beckhoff crates.

14h: The SRA motors and FLT are both depending on this Beckhoff control.
EEE decides to remove the defective Beckhoff motor terminal because voltage is still delivered and there is the danger that it will start burning.

We proceed with the help of the colleagues in the tunnel and the device expert to manually move the FLT manipulator to the graphite position,
and looking at the operation plan photon energies it can stay there until the WMP.

At the same time we manually check the SRA slits, the motors are running ok. However, removing that Beckhoff terminal also disables the SRA slits.
It would require a spare controller from the labs, thus we decide in the interest of going back to operation to move on without being able to move the SRA slits.

14h22 Beam is back on. Immediately we test that SA3 can go to 100 bunches - OK. At 14h28 they go to 384 bunches. OK. Handover to SCS team.

14h30 Realignment of SA1 beam. When inserting IMGFEL to check if the SRA slits are clipping the beam or not, it is found after some tests with the beamline attenuators and the other scintillators that there is a damage spot on the YAG ! See next logbook entry and ATT#3. This is not on the graphite filter or on the attenuators or in the beam as we initially suspected.
The OCD colleagues from DESY-MPC and EEE are released.
The SRA slits are not clipping now. The beam is aligned to the handover position by RC/BKR. Beam handover to FXE team around 15h.

Damage on YAG screen of SA1_XTD2_IMGFEL

A damage spot was detected during the restart of SA1 beam operation at 14h30.

To confirm that it is not something in the beam itself or a damage on another component (graphite filter, attenuators, ...), the scintillator is moved from YAG to another scintillator.

ATT#1: shows the damage spot on the YAG (lower left dark spot, the upper right dark spot is the actual beam). Several attenuators are inserted.
ATT#2: shows the YAG when it is just moved away a bit (while moving to another scintillator, but beam still on YAG): only the beam spot is now seen. Attenuators unchanged.
ATT#3: shows the situation with BN scintillator (and all attenuators removed).

XPD expert will be informed about this new detected damage in order to take care.

Beckhoff communication loss SA1-EPS

12h19: BKR informs PRC about a limitation to 30 bunches in total for the north branch.
The property "safety_CRL_SA1" is lmiting in MPS.
DOC and EEE-OCD and PRC work on the issue.

Just before this, DOC had been called and informed EEE-OCD about a problem with the SA1-EPS loop (many devices red), which was very quickly resolved.
However, "safety_CRL_SA1" had remained limiting, and SA3 could only do 29 bunches but needed 384 bunches.

EEE-OCD together with PRC followed the procedure outlined in the radiation protection document 
Procedures: Troubleshooting SEPS interlocks
https://docs.xfel.eu/share/page/site/radiation-protection/document-details?nodeRef=workspace://SpacesStore/6d7374eb-f0e6-426d-a804-1cbc8a2cfddb

The device SA1_XTD2_CRL/DCTRL/ALL_LENSES_OUT was in OFF state although PRC and EEE confirmed that it should be ON, an aftermath of the EPS-loop communication loss.
EEE-OCD had to restart / reboot this device on the Beckhoff PLC level (no configuration changes) and then it loaded correctly the configuration values and came back as ON state.
This resolved the limitation in "safety_CRL_SA1".

All beamlines operating normal. Intervention ended around 13h30.

Beckhoff communication loss SA1-EPS (again)

14h00 Same issue as before, but now EEE cannot clear the problem anymore remotely.
An access is required immediately to repair the SA1-EPS loop to regain communication.
EEE-OCD and VAC-OCD are coming in for the access and repair.

PRC update at 16h30 on this issue

One person each from EEE-OCD and VAC-OCD and DESY-MPC shift crew made a ZZ access to XTD2 to resolve the imminent Beckhoff EPS-loop issue.
The communication coupler for the fiber to EtherCat connection of the EPS-loop was replaced in SA1 rack 1.01.
Also, a new motor power terminal (which had been smoky and removed yesterday) was inserted to close the loop for regaining redundance.
All fuses were checked. However, the functionality of the motor controller in the MOV loop could not yet be repaired, thus another ZZ access
after the end of this beam delivery (e.g. next tuesday) is required to make the SRA slits and FLT movable again. EEE-OCD will submit the request.

Moving FLT is not crucial now, and given the planned photon energies as shown in the operation plan, it can wait for repair until WMP.
Moving the SA1 SRA slits is also not crucially mandatory in this very moment (until tuesday), but it will be needed for further beam delivery in the next weeks.

Conclusions: 

In total, this afternoon SA1+SA3 were about 3 hours limited to max. 30 bunches.
The ZZ-access lasted less than one hour, and SA1/3 thus didn't have any beam from 15h18 to 16h12 (54 min).
Beam operation SA1 and SA3 is now fully restored, SCS and FXE experiments are ongoing.
The SA1-EPS-loop communication failure is cured and should thus not come back again.
At 16h30, PRC and SCS made a test that 384 bunches can be delivered to SA3 / SCS. OK.
Big thanks to all colleagues who were involved to resolve this issue: EEE-OCD, VAC-OCD, DOC, BKR, RC, PRC, XPD, DESY-MPC

Timeline + statistics of safety_CRL_SA1 interlockings (see ATT#1)
(these are essentially the times during which SA1+SA3 together could not get more than 30 bunches)

12h16 interlock NOT OK
13h03 interlock OK
duration 47 min

13h39 interlock NOT OK
13h42 interlock OK
duration 3 min

13h51 interlock NOT OK
13h53 interlock OK
duration 2 min

13h42 interlock NOT OK
15h45 interlock OK
duration 2h 3min

In total, this afternoon SA1+SA3 were about 3 hours limited to max. 30 bunches.
The ZZ-access lasted less than one hour, and thus SA1/3 together didn't have any beam at all from 15h18 to 16h12 (54 min).
SA3 had beam with 29 or 30 pulses until start of ZZ, while SA1 already lost beam for good at 13h48 until 16h12 (2h 24 min).
SA3/SCS needed 386 pulses which they had until 11h55, but this afternoon they only got / took 386 pulses/train from 13h04-13h34, 13h42-13h47 and after the ZZ.

Status

All beamlines in operation.
The attachment ATT#1 shows the beam delivery of this afternoon, with the pulse energies on the left and the number of bunches on the right.
SA2 was not affected, SA1 was essentially without beam from 13h45 to 16h15 (2.5h) and SA3 didn't get enough (386) pulses in about the same period.
Beam delivery to all experiments is ok since ~16h15, and SCS is just not taking more pulses for other (internal) reasons.

Beam delivery Status well after tuning finished (which went faster than anticipated).

goals achieved:
alignment of the beam in tunnel and hutches including refinement of the beam in the center of rotation 
check of cameras and optical components
initial check of the mirror chamber (successfully)

issues:
smarACT motors in the DES tricky - some are in error state and the filter wheel is moving and does not stop - positioned it and removed afterwards the cable - EEE intervention next week necessary
camera of Pop-in imager II45-2 in XTD6 went into error - DOC solved the issue

Goal Achieved.
- Finished flat field measurement for AGIPD at different scenarios for upcoming user experiments. 
- Aligned the spectrometer at DES and obtained the energy spectrum of 9 keV SASE.
- In order to investigate the sample to detector distance of AGIPD, we collected SAXS patterns from three different type of SiO2 NPs at different sample positions.
- Finished preparations for tomorrow.
 

check of the optimized focussing options at 10 keV using only one arm in CRL2 - results still to be evaluated
successful check of the mirror chamber in the experiment hutch of MID - it was possible to reflect the direct beam down and use it under a shallow angle on the sample position

Stable seeding with 1mJ + 400mJ SASE
HIREX triggered interlock when all attenuators are removed 
successful test of nano focusing setup
successful test and alignment of x-ray microscopes
Alignment of split and delay line
Laser x-ray timing at sample position

instrument alignment including NaFos, DES spectrometer, diamond detector, PUMA & test samples
seeding makes a good impression based on the DES spectrometer

Weekend summary:

Overall stable beam delivery with 300bunches @18keV and 750uJ

-issue with XGM on Saturday night/morning

-The intensity in SA2 over the train is dropping of when SA3 is changing some parameters. BKR can fix it within a couple of minutes

-The users found suitable working conditions to overcome the limitations from the 10bar He supply line.

alignment of the instrument including collimation and beam shaping
test of the tempus detector using standard samples with different rep rates, intensities & # of pulses
start commissioning of the PhotonArrivalMonitor (temporal & spatial overlap, white light spectrum)

continue with Tempus tests
continue with PAM commissioning (first results of the working device at MID)

comments:
changed to 10 keV in order to use only one focussing lens in CRL2 and generating a smaller focus

issues:
vacuum issues in the MPC (reason: open roughing valve)
FSSS (hall sensor broken - exchange of the entire device planned for tomorrow)

optimization of seeding (still not as good as needed)

set position for the two color mode (difference 3-4 ev)

timing of the direct beam with diode and laser
timing of the direct beam with YAG and laser
alignment of the lower branch of the SDL
timing pf the lower branch beam with YAG and laser

issues:
intensity and spectral purity of the seeded beam not really good
alignment of the mirrors have been tricky (previous values did not work)
 

optimized the SDL alignment

found timing of the upper (-520 ps) and lower branch (-514.5 ps) of the SDL

moved the upper branch by 3 ps (positive) and lost the beam on the merger crystal - difficult to get it back

Found the delay time from upper and lower branches of SDL with a frosted-YAG.
The upper branch is -512.8 and the lower branch is -513.28.
The delay time difference is about 500 fs. 
The Mid SDL delay tuner should be investigated further more. 
The next step is find the spatial overlap on the Suna microscope and measure the fringes. 
 

Found the spatial overlap on the Suna microscope with Zyla.
Found the SDL motor position giving 200 fs time difference between the upper and lower branches. 
The encoder and current values of DC2_CASZ and DC1_CASZ are not 100 % correlated. 
The SDL delay scan using ScanTool has an issue due to the incorrect motor movement. 
The manual delay scan has been attempted, which was not successful. 
 

Found spatial overlap at Suna microscope with Zyla cam.
The splitting ratio was obtained about 1:1. 
The spatial jittering from upper and lower branch were investigated.
The temporal overlap and fringe will be investigated further.
 
 

Found the interference fringe at Suna microscope with Zyla cam from MID split-and-delay line. 
The fringe patterns were recorded as a function of delay controlled by the upper branch motor position. 
Obtained the interference fringe patterns with two pulses with different repetition rate of machine. 
BKR provided the two color HXRSS seperated by a few eV, and SDL was aligned accordingly. 
The fringe patterns were recorded as a function of delay with two X-ray photon energies.

- DAQ was crashed after taking darks of AGIPD. It was solved by DOC and OCD.

- Laser could not switched from optical to RF lock. Solved by OCD (using doocs).
- Energy scan was attempted, which was not successful. 
- Seeding optimization was ongoing. It will continue tomorrow morning. Still there is intensity loss during the energy scan.
- We might see second (maybe higher) photon energy component on the beam. (https://mylog.connect.xfel.eu/#narrow/stream/201-MID_6877_UrsStaub/topic/.E2.9C.94.20Troubleshooting/near/72056)
- The beam was focused with CRL 2 arm 4 and 5. A precise beam size measurement would be useful. 
- The timing of XFEL and PP laser has been found from a frosted-YAG on the X-ray eye. T0 = 1374.94.
- The air gap between the MPC and DES are closed. 
- The first user sample is mounted and delivered in the PUMA setup.
- The spatial overlap of XFEL and PP laser has been found at the sample position with YAG crystal.
- The substrate peak (th = 20.7, tth = 41.4) was found on the fluorescent screen. 
- The SDL diamond detector was aligned on the beam, and now move it out.

 
Tomorrow
- Continuing of optimization of seeding and improving the energy scan.
- Cool down the sample and search the film peak on AGIPD.

- The seeding optimization result: Total 450 uJ (Seeding of 150 uJ  + SASE of 300 uJ).  BKR will be able to start optimization on Saturday around noon if necessary. 
- The pump-probe (PP laser and PuMa pulse) experiment data on the substrate peak was collected. 
- Found the film peak on the AGIPD. Intensity optimization was not successful. We keep losing diffraction peak intensity over the measurement time and number of pulses. We assume the X-ray-induced heating can affect the diffraction intensity.
- Rotating Phi angle by moving the manipulator Z direction helped to bring the peak out of the detector gap. 

- The seeding optimization is ongoing, which was not successful. Now we have a total of 170 uJ including SASE and seeding. 
- The option to switch to SASE is relatively not complex and can be done with the support of BKR without a seeding expert.  BKR expects to provide about (more than) 1 mJ. To be discussed with user later. 
- An investigation of the asymmetric film peak is ongoing to optimize the measurement conditions, such as diffraction angle, sample temperature dependence, and pump laser effect. Still, the PP laser effect on the diffraction peak on the asymmetric peak is unclear.  The higher sample temperature suppresses the diffraction intensity.  Unclear intensity changes over time have been observed. 
- The AGIPD moved closer to the sample to cover the more solid angle. 
- The MDL of the coupled motion of the hexapod and puma manipulator is not working reliably. 
- The delay time between the X-ray and the PP laser was cross-checked to ensure its correctness. No timing drift was confirmed. 
- Laser pump and X-ray probe experiment is being conducted at the vicinity of subtract peak. After optimizing the spatial overlap by scanning, the pump laser effect was clearly achieved. 
- The user program will be ongoing over the night. 

- Pump-probe experiments are being conducted using PuMa and PP laser. 
- The user program is ongoing over the night.

Tomorrow, 
- Rotating the sample phi angle. The sample holder will be retracted from the MPC.
- Search the asymmetric film peak and conduct the PuMa and PP laser experiment.

Started the day with vacuum & detector issues
real start of the experiments only at about 6 p.m. 
measurement of a selection of samples (urea, ludox) using the levitation device

issues: vacuum (behind AGIPD), motor (bend2 of the DES was in error - used for the beamstop)

started initial alignment for the ptycho experiment 
Continued with XPCS measurement using the levitation device

issues: contribution of the 2nd harmonic of 8 keV at 16 keV visible on the Jungfrau detector - surpressing via mirror alignment not successful

continued with the evaluation of the XPCS analysis packages
investigated the effect of alignment optimizations and forced misalignments on the contrast directly with the online XPCS tool
checked additional standard samples 

01:49

Worked on the XPCS analysis packages
measured the TetraPack samples
continued with the levitation device

issues:

1. Jungfrau detector had an error - R. Shayduk restarted the device - it should be power cycled as described in the manual
2. The bunch number selctor does not work - the system didn't accepted 150 pulse but moved it back to 30 pulses - the machine switched off the user selection - should be followed up tomorrow

12.389keV, HXRSS, 1-400+ bunches, 450uJ

• Seeding alignment
• Beamline alignment (Mirror + foccussing)
• Measured energy and moved it the requested value
• Cryos for sample and monochromator are prepared
• Aligned DES spectrometer
• Started with DCCM characterization (440)

beam parametes: 8.5-9 kev, 1.2 mJ, 1-2 pulses

• HED trajectory established from XRO setpoint, shape and transmission optimized
• monochromator aligned
• 5 sets of CRLs including nano-focus aligned
• ZZ yesterday morning to fix cameras, connect motors..
• DAQ rolled back to legacy version, works fine
• instrument spectrometers aligned and calibrated
• We perform some longer scans with monochromated beam imaging on wires / Siemens star
• Work on shimadzu timing
• Debugging new JNGFR_DARK_CHAR middle layer with controls - done

*Achievements :
- Generally rather successful with very high quality beam parameters
- We had a nearly steady 800 uJ+ HXRSS flux during the user run and ~ 4mJ SASE during the setup at HED.
- Beampointing was rather stable throughout the HXRSS operation.
- Users took several shots (>2600 FEL and FEL+ReLAX shots) on target. The beamline setup was very quick prior to that and users were able to get preliminary data to prototype parameters already a day before formal handover to them and could straight go to harvest mode on Saturday morning.  
-----------------------------------
* Issues :
- Users still found large Bremsstrahlung fluxes on their background, but this is rather on our side, in conjunction with the users.
- On the last day, we had issues with JF calibrations due to the high run numbers in the proposal. DOC promptly helped out.  
- Initially, we observed fluctuations in the FEL flux. BKR was very prompt to correct for this and user operation was not impacted seriously. Slow drifts in pulse energy were also observed, but again, could be compensated from time to time by them.
-----------------------------------

Shift summary 2024-02-23 07:00 - 2024-02-24 10:00

We got self-seeding at 8.5 keV very quickly after changeover from 10.5 keV operation finished at 2024-02-23 07:00.
Pulse energy ~ 600 uJ. 

Progress is in general good. Most of planned tasks for the day are done.
We called BKR whether we can get a bit more energy/pulse this morning, but it appears difficult. Otherwise, having a bit more energy would be helpful to have enough signal-to-noise ratio for X-ray imaging. 

Achievements: 

• Insert nanofocus lens, pump IC1 sample chamber into vacuum
• Establish X-ray axis, measure source point and divergence
• Set up the CRL scheme overfilling CRL4b nanofocus (the settings used in Feb at 8.2 keV were CRL1 arm 3,5 and CRL2 arm 1)
• Bring the imaging systems to the X-ray axis
• Focus and calibrate the x-ray spectrometers
• Laser-xray timing verified on YAG
• DAMNIT setup 
• Nanofocus imaging optimization (Z-scans) with Cu 25um wire and Siemens star
  • Best focus was identified
• Sample alignment 

Shift summary for 2024-02-24 from 07:00 to 2024-02-25 at 14:00:

Self-seeding at 8.5 keV, with approximately 500 μJ per pulse. Following re-tuning around 10 am on the 25th, the energy per pulse increased to around 600 μJ.

The x-ray intensity and pointing stability have been excellent, resulting in a substantial amount of data being acquired by users just two days into the beamtime. Users have expressed satisfaction with the performance thus far.

There were a couple of brief beam downtimes, but they had minimal impact on beamline operations.

Achievements:

• Successful laser delay scan (0 - 1000 ps) of wire compression dynamics under ultra-intense laser interaction (>10^21 W.cm-2).
• Nanofocus Z scan imaging completed.
• Online analysis tools (DAMNIT) are functioning properly.
• Some minor issues with the automatic shot procedure using MDL were encountered but mostly resolved.
• Ongoing issues persist with the off-axis-parabola mirror X axis (motor movement not stopping), which will require follow-up after the beamtime.
• A minor issue was identified with laser timing jumps, possibly stemming from a trainID distribution issue with the ToD system. Increasing the sequence delay of the master timer from 10 to 30 appears to have mitigated this problem.

Shift summary for 2024-02-25 from 11:00 to 2024-02-25 at 14:00:

Self-seeding at 8.5 keV, with approximately 500 μJ per pulse. 

Overall, the performance of x-ray self-seeding remains good, and users are happy. Pointing stability is also exceptional. However, enhancing photon energy slightly (600 uJ as we had yesterday) would further improve the signal-to-noise ratio.

Key Achievements and Current Challenges:

1. Ongoing data accumulation.
2. Issue with the sample motor (stepper motor). The encoder ceases to update values although the motor is actually moving. While a temporary workaround has been implemented, a thorough investigation is needed post-beamtime. More details: [Link]

• proposal 5700 finished on 04.05. The experiment used 352 pulses at 4.5 MHz and 18 keV to X-ray heat pressurized samples in diamond anvil cells. Overall succesful on several samples.
• Issues:
  • the requested 20.5 keV photon energy had too low pulse intensities so we had to go down to 18 keV
  • pulse energies at 18 keV (300-400 uJ) were insufficient for some samples
  • the ramp in pulse envelope has not been delivered consistently throughtout the experiment (eventually we gave up due to the average low pulse energies)

beam parameters: 2 colors at around 6.5 keV, 1 pulse

achievements:

user program continues with data acquisition at slightly varying photon energies (tens of eV both colors simultaneously) and at different delays between the colors

Issues:

• users have scheduled datasets to be measured overnight with larger delays up to 350 fs, however, the BKR was able to go only up to 100 fs delay despite having larger delays earlier in the experiment (even during nights)

Beam conditions:

• Used photon energies between 6 and 7 keV, including short undulator scans
• Continuously got 1 mJ despite the short pulse mode that's running
• Bandwidth was found to be 18 eV FWHM at 6.07 keV (0.3%) by monochromator scan against fixed undulator setpoint

Achievements:

• Cr foil scan for energy calibration of the FXE mono
• Prepared setpoints for XAS scans during next week's user expts. at: Ce L3 (5.72 keV), Ce L2 (6.16 keV) and Mn K (6.54 keV)
• IPM calibration against XGMD

Issues:

• ATT MDL (class JJAttenuator) got stuck in INIT state and then did not show that absorber was inserted in the beam path (ticket to be submitted to Ctrls)
• DA3 of FXE DAQ (FXE_DAQ_DATA/DA/3) got stuck in changing state (it got too many parameters inserted). The restart by shutting down the server and waiting for the device to come back up took around 20 mins, see 22.2.2024, 4:24 am to 4:43 am.

Not sure if this is the cause of the fringes reported before, but at 5.6 keV there are clear cracks visible on the XTD2 Attenuators 75 umm and 150um CVD, see attached. There is also a little bit of damage on the 600 um CVD, but not as strong as the others.

At 5.6 keV we will need to use the thin attenuators. I will mention again in case this causes problems downstream.

Edit 8:50h: We do not observe additional interference effects in the downstream beam profile (see attachement 4)

5.61 keV setpoint in SA1 was confirmed to correspond to 5.660 keV (+/- 0.003 keV), see attached.

The user expt #5644 can proceed as planned.

Preparation for user experiment transient grating, Cris Svetina

• Used 7.12 keV from ACC with very good performance, above 4 mJ
• Got spatiotemporal overlap with transient gratings using monochromatic beam
• One issue solved by DOC: Picomotors could not be scanned in karabacon, even though this worked well two/three weeks ago

User experiment transient grating, Cris Svetina. (LTP)

• Used 7.12 keV from ACC with very good performance, above 4 mJ.
• Collected pump-probe data on user sample
• Sudden timing jump of around 20 ps. Could not find out why this happened some time between 9h today and 18h.
• Needed to add camera to DAQ. Solved this on instrument side using the DAQ assistant. Great tool!

User experiment transient grating, Cris Svetina. (LTP)

• Used 7.11 keV from ACC with very good performance, above 4 mJ.
• Collected pump-probe data on user sample most of the time today, only minor interruptions due to accelerator outages
• After the outage at 21h, SASE came back with 17 eV lower photon energy, so we had to re-calibrate the undulator spectrum
• FXE will operate in 24h mode for the rest of the delivery week

User experiment transient grating, Cris Svetina. (LTP)

• Used 7.11 keV from ACC with very good performance, above 4 mJ.
• Collected pump-probe data on user sample most of the time today
• FXE will operate in 24h mode for the rest of the delivery week

Measurements to check the impact of bunch compression on FXE mono transmission were successful. We could change the peak current between 4 and 12 kA and saw changes in bandwidth between 17 and 100 eV (at 14 keV). Results are under evaluation.

SA1 brought to safe state:

• CRLs moved out
• Mono moved out
• CVDs in XTD2 ATT inserted
• PBLM-based feedback was started and kept running to avoid beam drifting off of M3
• Hirex shutter closed
• The M1/M2 offset was decreased a bit as compared to saturday mornings' value (was very close to total reflection limit and to safely use 100 bunches, it was decreased to be on the safe side). Should not affect further measurements for XFELO on sunday.

Used 9.4 keV with 94 kHz wed and thu nights. Received approx. 0.9 mJ with large fluctuations in pulse energy, however acceptable for the tasks below.

Achievements:

• X-ray optics and goniometer aligned for next user experiment 5725
• Transmission through Mono #2 recovered and calibrated to match Mono #1
• Understood uneven beam profile of FXE PI originating from the thick XTD2 ATT CVDs. Without them, the beam looks nice.
• Observed that overbending M2 gives better beam profile than underbending under the present conditions

Issues encountered:

• Major issue: M3 RY piezo actuator not working any more. This means beam feedback cannot run and beam keeps drifting off on the instrument apertures. This cannot be compensated by M2 piezo due to the small aperture of M3. The issue was reported wednesday night to PRC, XRO and FXE group and is being worked on by XRO
• One limit switch was found swapped; fixed by FXE engineers
• Some motors moved in the opposite direction as compared to previous commissioning, even though parameters were saved and restored. Not understood, but was easy to work around.

We observe a continuous horizontal beam drift in SA1 after the M1 incident. It seems not to converge, so one can wonder where this will go....

Attached screenshot shows the M2 piezo actuator RY that keeps the beam at the same position on the M3 PBLM. These are about 130 m apart, The drift is 1V/12 hours. 0.01V corresponds to a beam motion by 35 um. So the drift velocity is 0.3 mm/hour. In the past days this had at least once lead to strong leakage around M2. But the two feedbacks in the FXE path are keeping the beam usable most of the time.

Finally, the drift velocity of the pointing reduced approx a factor of 2 as compared to yesterday, see attached. Possibly it will still converge...

FXE received the beam and did not change accelerator settings.

Beam alignment to FXE was re-established using the new values for the old M1/M2 offset.

Monochromator aligned to the SPB energy setpoint.

THz source aligned to Xrays and several reference positions for Xray pointing taken. Further steps require THz radiation, which is not available at this point.

Xrays realigned back to SPB XTD9 screen.

The xray beam drifted significantly on wednesday and thursday (18. and 19.) but then was not drifting on friday (20.), see attached trendline of the M2 actuator that kept the beam at fixed position on M3.

Today also seems that beam pointing is stable. But unclear what happened earlier...

This is to report some findings during operation that should be looked into (mostly to avoid in the future, because both negatively affect data quality of the user expt at FXE):

• At the moment, the 99:1 mode affects 4 to 5 pulses out of 100 in SA1, see first screenshot that shows in the top right the IPM reading with each blue point denoting the average pulse energy in a train. This was not always the case. After initial setup on thursday it did affect only a single train out of 100.
• On 27.10., the switching of SA2 between 1 and multiple bunches affected SA1 beam heavily, see the jumps in SA1 pulse energy in the second screenshot shown here:
  • The pulse energy dropped by 15% and
  • the pointing changed vertically by 150 um on the FXE PI (data not shown, but available from karabo trendlines)

Edit 8.11.: Added screenshot (grafana/ctrend) to show that the vertical displacement was permanent (not corrected by the feedback)

We had some trouble with the DAQ. It was not able to start runs properly.

We got the following error messages:

initially:

[00:01:27]: Problem of communication with data manager

[00:01:30]: Start new run ...

[00:01:38]: Failed starting data recording

DM: error at registerRun (Request to MDC timeouted)

after trying to start the run again:

[00:05:35]: Start new run ...

[00:05:35]: Failed starting data recording

DM: error at registerRun (Problem of communication with MDC

#Info: MDC-1100: MDC web server error: Error creating run

#Details: {'run_number': ['must be unique per proposal', 'has already been taken'], 'experiment_id': ['has already been taken']})

Cycling the DAQ resolved the issue.