SPB

On monday there was an issue with a detector related PLC due to misconnection of one EtherCAT cable. The issue has been solved and it "just" resulted in 3 of hours delay in  the AGIPD calibration activity.

FXE, SCS, SQS, XPD, XRO,ITDM no issues reported

AE related

uTCA in XTD 6 are installed 

VAC-Related

Marc Messerchmidt reported  abouit an entry in the DESY Logbook about a pump that triggered the interlock in the SASE1 tunnels. According to Nerea this this seems to be related to the old issue already known. VAC group is working on that. 

 CAS

CYoungman is working on the priority list for the components in SASE2 tunnels. Double checking with HSinn happening soon.

Access for Tue/Wed next week to XTD1 will be requested by CY through HS

Beam is in SPB/FXE this week due to the HERCULES project.

No new issues to be discussed where reported by any groups.

The status and issues reported on friday are still valid. 

Photon Run Coordinator (H. Sinn):

1.) SASE1: User experiment SPB could be finished sucussfully after several problems on the weekend (thank to FXE for providing night shift). Big progress in improving throughput of FXE insrument. Next D3 material test with 300 bunches on April 3, 9am. 

2.) SASE2: Transmission of electron beam to T5D dump (SASE2 beamline) this week sucessful. Switching between electron beamlines 

works nicely (minor problem witth dark current during switching).

Now air coil measurement going on. BBA needs to done before first lasiing starting on May 2nd. 

To be clarified today: To we need access to XTD in week 12/13.

3.) SASE3: Photon commissioning (with 24/7 shifts) finished sucessfully. Now specific tasks are done parasitically to SASE1 operation 

during regular working hours (organized by photon run coordinator). 

4.) General (stimulated by DESY operators): EPS system can lead to shutdown of DESY accelerator through MPS by many different channels (PLC in tunnels, Fast valves, PBLM etc). 

On DESY side, MPS provides overview of inputs, but structure difficult to understand and labels are not always conclusive, 

On XFEL side, no general overview panel for all SASEs+Instruments (scene) exists, but it required for facility operation. 

SPB

No issues/no report 

SQS:

- Cleaning of the hutches is ongoing. From 19.3. on, access will be limited. Furthermore, shoe covers have to be worn in E.06 and E.07.

- Electricity in E.22 is delayed. AE and ITDM will be informed about the status by PSPO.

- Vacuum will start the installation of the transfer pipe between the tunnel and E.07 on 19.3.

- SQS will start the installation of their instrumentation on 19.3.

- Some of our beamline components are currently stored at Hera South and must be transported to Schenefeld next week. However, there is an issue with the crane at Hera South. Kai-Erik has to look into this!

FXE (P. Zalden / A. Galler):

No issues/no report 

SCS (J.T. Delitz)

No issues/no report 

PP-Laser:

"The laser group has nothing particularly noteworthy to report, only that installation is still going as planned"

Vacuuum:

Installations in XTD6 are ongoing. Damaged vacuum pipes have been 

exchanged/ordered. Vacuum in XTD1 is closed. Control System in XTD1 is 

working and locally tested, Interlock-tests will happen after Easter.

XPD:

No issues/no report 

Advanced Electronics:

No issues/no report 

SASE2 Tunnels (C. Youngman)

Going according to plan. 1 week after Easter the movement and EPS loops are supposed to be brought up. 

ITDM (K. Wrona):

No issues/no report 

Detectors (J. Sztuk-Dambietz, P. Lang):

A module for AGIPD was replaced. 

No issues/no report

CAS (H. Santos):

DEMO for the coordinated motion was done during the week. Further tests needed. 

Schedule and discussion about the Karabo rollout process. See presentation bz H. Santos given in the technical meeting.

SPB (Weinhausen/Round/Mancuso):

• Again on the fast valve was triggering the MPS interlock system. For XRO was not completely clear what happened. for AE everything is clear and it depends on the wrong MPS configuration with respect to missingl cables. The will discuss this offline and eventually bring back the issue. Linked to this, it is reported that the MPS panel is difficult to decipher for the experiments...

• FXE (Galler):

• 2 ion pumps went in error and the interlock was triggered. Issue clarified.

SASE2 Tunnels (Youngman)

• Access schedule sorted out. Works in the SASE2 tunnels will be carried out in the period 4-19.04 plus the week before Easter. Discussion with AE on the status of the loops ongoing.

AE (Jardon)

• It is reported by SPB (March Messerschmidt) that people in the BKR are confused about MPS signal that they are getting from XFEL. The different lables assigned to the signals have to be sorted out and communicated. Nerea and Marc will try to start this clarification offiline.
• GATT Loop in SASE3 will be updated today
• Check of the inficon gauges ongoing- eventually update required.

ITDM (Wrona)

• Lots of data (14 Tb) collected during the last SPB shift.
• Issue concerning speed of offline data transfer reported by SPB
• For the user runs in May data will need to be deleted from the offline cluster. A procedure has to be established to clearly communicate with GLs and User groups.
• SASE3 IT installations have been handed over to XFEL. IT setup ongoing. Network will be available in the experiments by the end of the month.

SPB (Weinhausen):

• During the night there was an interlock issue. The fast shutter was triggering the MPS interlock system. After several calls and request of support from SPB to the photon run coordinator and the machine run coordinator the problem was found and the interlock signal bypassed. This was due to some incomplete cabling of the fast valve. While this was knwe, after the last updated the correct configuration was not set up in the controlling PLC.
• In general one has to note that the experiments do not have access to the MPS panels and there is no well established procedure when MPS interlocks are concerned. This will be reported in the PSO meeting.

FXE (Gawelda):

• Before the beam shutdown FXE observed that the beam was very unstable.
• 2 ion pumps went in error and the interlock was triggered. Again in this case is not clear who is responsible for the re setting of the interlock and the pumps.
• Servers had to be restarted due to wrong train id after the cooling water accident
• 0.5 MHz operation of the LPD successfully tested. 

SASE2 Tunnels (Youngman)

• It is reported that the technical coordination of the necessary activities is missing.
• A list of activities was sent by CYoungman to TTschentscher concerning SASE2 tunnels. In total 10 working days are needed to complete these tasks. This includes 1 week implementation and 1 week commissioning. Carefully planning with DESY is needed to avoid that the necessary 10 working days are scheduled around Easter, which will result in less effective days.

CAS (Santos)

• HSantos is preparing a deployment plan for Karabo whoch includes the requirements and deadlines to be fulfilled by the other support groups (ITDM, DET, AE). The idea is to have this plan presented on Friday for discussion.

Photon Run Coordinator (L. Samoylova):

Commissioning during machine studies Tue and Wed and parasitically during SA1 user operation at 
 SASE3 on Thursday and Friday 

SASE1

SPB

User operation, current status (Adrian): got few data from a test system on Saturday. The users’ 
sample delivery technology now seems to deliver nice & stable droplets, so if SPB 
get beam early enough today to set up for a measurement, a full data set could be collected. 
 

FXE

• Instrument Commissioning
• Measure beamline transmission (unfocused 550 µJ per pulse at the sample position, with lenses 
  115 uJ)

 Machine studies

• Wed afternoon: a set of K-mono FEL spectra taken

https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/07.03_a&pos=2018-03-09T17:56:20 
 SASE3

 Thu, Friday - a first experience of the SASE3 commissioning in 30 bunch mode during user operation: 

• minimal number of undulators (12) for lasing, low intense SASE (XGM shows 10 uJ, linear 
  regime), SRA closed to 2 mm
• MCP2 @ SASE3 - signal from 30 SASE pulses, consistently decrease with closing the slit down 
  to 0.5 mm

https://ttfinfo.desy.de/elog/XMLlist?file=/XFELelog/data/2018/10/08.03_a&xsl=/elogbook/xsl/elog.xsl&picture=true#2018-03-08T18:24:14 

• Fri afternoon GATT calibration with XGM and MCP ongoing, in 30b mode, optimizing SASE3 lasing 
  for these measurements needed - too weak and not homogenous signal in the center, see 
  https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/08.03_a&pos=2018-03-08T18:26:50

  Problems:

 'unexpected' from photon side closing XTD1 two working days earlier, explicit schedule 

• for the south branch tunnels access would be appreciated
• power off in XHPSC on Thursday at 21:15, no cooling water in experimental hall that led to 
  increasing the temperature in the rack with the MicorTCA crate, the CPUs of the two MicroTCA 
  crates in SPB and one of the two crates in FXE were automatically switched off. SPB crate 
  1 did not come automatically back afterwards, as the result: no train IDs for SPB PLC 
  loops, no MPS for SPB, no Clock and Control for AGIPD, no triggers for SPB systems on 
  Friday
• Saturday morning the MPS interlock takes into account the signal from one of three fast valve 
  on SPB shutter although these valves are disconnected to SPB EPS interlock. The MPS 
  bit had to be masked

Correction from SPB [12.03.2018 10:10]: This issue was not related to the XTD9 shutter, but to the 
optics hutch shutter. The fast valve in XTD9 is connected to the EPS, the fast valve 
in the OH is not yet connected to the EPS. After the restart of the EPS/MPS on Thursday 
/ Friday after the cooling water issue, the condition for the fast valve in the optics 
hutch was wrongly included in the MPS. 

SPB

1) Thursday 08.03.2018 (8 am - 8 pm) - SPB/SFX commisioning shift

    Notes:

          - everything running

          - RAW data overview very responsive with  data filtering as suggested by Steffen

          - Corrected data preview started very nicely and seem to provide only limited update rate after a few hours running

          - karabo DAQ deices (dataaggregators) need several rests, resulting in a full DAQ restart around 7pm

          - sometimes updates in KARABO were lost

          - preparation for users experiment - over night

2) Thursday 08.03.2018 (around 9 pm) - "cooling water" incident

     Notes:

         - vacuum pumps affected

         - AGIPD warmed up

         - uTCAs have tripped/shutdown (some infrastructure needs to replaced)

3) Friday 09.03.2018 - morning (User beamtime)

         - damaged uTCAs equipment was replaced but trigger setting needs to be input/ adjusted =  loss of the user beamtime

         - User experiment progresses

SQS:

Hutch cleaning Mid. March

KB tanks installation in parallel with SCS

9-14 air filter will be installed. After that date only clean works have to be carried out

Collision with KB tanks installation and other activities like Rad. Interlock test to verify

FXE (P. Zalden / A. Galler):

Setting up the experiment

IPM and XGM data manualy correlated. Would be nice to have the train id working to have the "live" correlation

SCS (J.T. Delitz)

• cleaning of hutch next week (wk11)
• we will start moving first components wk12-13, ie. ALAS and DPS
• KB installation (positioning only, no AC+filter required) beginning April-depends on commissioning of precision climate zone (currently discussed with PI)

PP-Laser (M. Lederer):

SASE1 commisisonin gonoing.

Some misalignment due to the cooling water accident. To be investigated 

09.04 SASE2/3 laser tables will be installed.

Vacuuum (R. Villanueva):

SASE2--------->Good progresses on XTD1 control system deployment: Differential Pumping sectors are ready for first beam // First five vacuum sectors under vacuum and locally tested (already with Karabo). XTD6 tunnel installation phase progresses at good pace.

SASE1--------->In general, nothing additional to remark. We are working in a more robust interlock definition to prevent false triggering due of ion pump control signals.

SASE3--------->Complete vacuum system is working normally. Gas Attenuator is now under routinary operation up to 2-3 mbar with a UHV interface pressure after its DPS sector < 1*10e-9 mbar.

XRO 

SQS/SCS/XRO collaboration for the mirror coating ongoing

Installation of M5 in preparation

XPD (T. Maltezopoulos ):

XGM@XTD2, @XTD9, and @XTD10 are online.

XGM@XTD1 is in the loop and accessible from Karabo

(together with AE+CAS).

Currently I test XGM@XTD1 components from DOOCS.

XGM@XTD9 shows slightly higher energy then in XTD2.

I calibrate all XGM components twice, but still it remained.

Note that the beam pass the XTD9 XGM with -7 mm offset

(in beam direction 7 mm to the left, to the south).

With this high offset absolute calibration is not guaranteed.

Advanced Electronics (P. Gessler):

1 CPU for SPB broken due to the cooling water accident. Already replaced. Being configured

FXE grating integrated in the vac loop

SASE2 Tunnels (C. Youngman)

More coordination is needed, especially concerning the access to the tunnels. For the technical details, please see slides on the indico page https://indico.desy.de/indico/event/20115/

ITDM (K. Wrona):

No problems with overrheating

Problems with Train ID reliability after the heating. Should be checked and monitored online

Documentation of single devices have to be improved. + there is need for a tool to make the sanity check

Detectors (J. Sztuk-Dambietz, P. Lang):

LPD commissioning OK + DAQ working. Now LPD off because of cooling water problems.

CAS (H. Santos):

• Sandor is in a meeting with the Detector group to align requirements regarding AGIPD and DA
• We started to prioritize and address the (agile) missions of SPB and FXE. As part of this effort:
  • Alessandro started to develop the scenes link on the GUI. This feature is really required and has been present in several SRDs (For example, to link vac sections/loops).
  • We implemented the GUI combo box to select different GUI Server - it is part of the SPB Mission (Leonce)
• Hamed and Sergey made good progress on the Digitizer. They will work on a demo/talk for the next PBS Meeting.
• Wajid, Sergey, and Dennis fixed the X2Timer issue reported by FXE (hash/data inconsistency). It is already deployed and running.
• Regarding the cooling incident. Wajid was the first contacted yesterday. After that Alessandro at 01:00. Today, during the morning, we reinstall Karabo at SPB-RR-SYS-UTC-1.

SPB:

• Monday morning Karabo failed. It was not possible to operate anything. Aftrer restart it took 2 hrs. to have the tunnel loop and Lopp 5 again operable . Other issues connected to that. CAS is on it.
• AGIPD can be operated with some workaroung concerning the control system. No final fix available yet.
• CSS can be operated locally.
• Coordinated motion test probably on 14.03.

FXE:

• IPM/XGM correlation data process ongoing. 
• DAQ worked reliably
• Scan tool operational for almost every component. Data storage works. Test ongoing.
• PG points aout that no MPS definitions were given and no MPS is available for FXE yet. To be checked.

SQS

• Corrdination of installations in the hutch with the TUV tests for Rad. Interlock needed.

SCS

• Nothing ton report

ITDM:

• FXE LPD slow data separation will be done today after the meeting
• Some devices are setting the train id as "0". CAS is informed.

SASE3

MCP HV - Crate restarted. Now ok. There was a misconfiguration of Power/Karabo settings. This meeds to be avoided increasing the communication between people performin operations on the XPD components and the technical support groups.

AE

• uTCA crate in SASE2 for timing being prepared, as well as PLC for VAC loop
• Replacement of a uTCA crate in the balcony room for SASE2 needed

Below some clarifications from R. Bean on the issues reported in today's dispatch meeting:

- AGIPD suffered many problems which affected the operability (not really a problem of the scene). It was just operable, although not at all efficiently. Problems with the pipeline / DAQ and Karabo controls for the AGIPD suggest that some reasonable amount of work is required to find and solve the problems which may not be independent from each other

- CAS OCD point already addressed with previous email: They were available, just we needed them rather more than we should have done

- The CSS is the large rail that carries the detector. We cannot move it via the integrated system because of a misunderstanding in the implementation of limit / safety switches into the eplan design of the crate. The cabling between rack room / device seems to be fine (but of course not really tested properly because the crate is not correct). We have to use the test crate to move it right now, which is rather tedious since the encoders / limits cannot be handled properly by that system. I am available to discuss the proper implementation of switches / buttons such that we don’t need a fourth attempt at this crate

- For the smaracts we still have axes that are not controllable via the PLC. It is these that would be beneficial if they are fully integrated 

SPB

• Last night the beam disappeared between 4 and 8. 
• AGIPD scene showed many problems, both concerning Karabo and Data preview. Detector is not operable.  To solve the issue a joint effort of all the groups involved, as well as the physical presence of experts during the shift is requested. 
• CSS has to be moved out of the beam. As of now there are problems with cabling that need to be solved
• Smaract stages are misbehaving. More support from AE (Mahadi) is requested
• Status of the coordinated motion test was asked. Should be ongoing but the experts were not there.

FXE:

• Testing fast intensity monitors and LPD 
• DAQ worked very well
• Scan tool must me tested
• Correlation between LPD and fast intensity monitor is problematic due to train id problems. There is a new device to be deloyed on the XGM. Today or tomorrow will be seen whether this solves the issues.
• BKR reported that the beam tripping was due to some FXE device close to the M3, but FXE is not aware of that. Info will be asked to Harald today during the PSO meeting.

Advanced Electronics (P. Gessler):

• MCP HV problem in SASE3 - the power supply failed. The fixing shoud be done according to the priorities, Photon run coordinator decides.

ITDM (K. Wrona):

• FXE slow data DAQ issues: these will be solved splitting the LPD slow data from the rest. small update of the software needed. 
• Simultaneous acquisition from LPD and cameras is now available and working well.

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIPD will operate with the old firmware due to problems with the new one
• No calibration data available as of today
• AGIPD is in a safe state

Photon Run Coordinator (N. G. Kujala):

• SASE1
  • Machine delivery 300 pulses upto XTD2 without M1 mirror in beam on Wednesday evening.
  • On Thursday 300 pulses deliver to FXE for radiation test.
  • Collected data using DAQ with 100, 200 and 300 pulses.
  • Large area Detector40 x 40 mm was set-up for radiation test at FXE for .
  • Damage test with 300pulses in FXE hutch with 0.5mm steel, 0.2mm copper and B4C block   
  • Vacuum at XTD2 absorber was stable and temp was also stable for 300 pulses
  • Harald setup piezo motors for M3 of SASE1.
  • Yesterday night beam delivery to SPB with single bunch.
  • ZZ for XTD9 on Tuesday by SPB for CRL test/check.
  • Vacuum on FXE branch in XTD9 was tripped on Tuesday early at 1am, which lost beam permission. Manually I have reset interlock.
• SASE3
  • SASE3 commissioning is going on.
  • Commissioning the XGM and IMG correlations with Gas ATT and without Gas ATT;
  • SoftMono mirrors is commissioning
  • Gas attenuator commissioned and Raul made look-up table for pressure setting
• 30bunches were operated in SASE3 with Gas attenuator

SPB

• Ready for first commissionig
• AGIPD power and cooling procedures OK
• Feedback problems between Karabo and AGIPD. Being worked out

SQS:

• Hardware installation starting Mid. March after  cleaning of the hutch
• FMB will install the mirror tanks in April. Has to be synchronized with the remaning work in the hutches
• Meetings with AE soon to strat planning the commissioning

FXE (P. Zalden / A. Galler):

• Ready to accept first beam
• Radiation tests done

SCS (J.T. Delitz)

• Removing of wood cover: 05.02.2018

• Cleaning: end of week 11.

• End of march beginning of the installation of the first components (ALAS and TIM)

• Waiting of the exact date for Interlock test to specify more exact the dates for the KB mirror system installation.

PP-Laser (M. Lederer):

• LAS is now commissioning 3-stage NOPAs for both FXE and SPB.
  ​so far all in plan to finish in March

Vacuuum (R. Villanueva):

• SASE1 
  • Nothing remarkable: the whole vacuum system works in a very stable way. We only had one incident on the output of 1-2 ion pumps that trigger once the interlock. For that we are investigating the causes .
• SASE 3.
  • The whole vacuum loop (as required for the first lasing) down to the instrument front-end is under control and no special issues have appeared yet.
  • The SASE3 Gas Attenuator is under manually operation and most of the Photon Commissioning Members are familiar with it. Next steps are aligned with the deployment of the Pressure Feedback Control PID control available at Karabo level and to continue with the software-related tasks.
• SASE2.
  • XTD1------->The complete vacuum string down to the front-end in the shaft building is under vacuum but in local mode operation (including XGM subsystem)
  •          ------->During this almost past week we already initiated the PLC local tests (i.e. US & DS Differential Pumping are already done), and in KW10 we aim to move on with the rest of the system (ion pumps and vacuum sector gate valves).
  •          ------->Absorber, V0 and Fast Valve wiring are in good progress and we think they could be ready for local tests by the beginning of next week.
  •          ------->The interlock tests are planned also for KW10.
  •          ------->Depending of the progresses of AE and CAS, we may even be able to deploy the first control scenes in Karabo.
  •  
  • XTD6 ------->The assembly works of the whole beamline there progress adequately and without remarkable "show-stoppers" that couldn´t be managed.
  •          ------->Vacuum Group and Optics group (N. Kohlstrunk) are having also special attention and dedication to the SDU vacuum chamber conditioning and preparation works for its upcoming in-vacuum mechanics installation.

XPD (T. Maltezopoulos ):

• XGM in XTD1 is connected to DOOCS (together with DESY)
• Synchronization of XGM@XTD1 microTCA next week
• Local tests at XTD1 XGM were finished (together with AE)
• XTD2, XTD9, and XTD10 XGMs are running 24/7
• Change to 300 bunches went without problems
• XTD9 XGM need SRG recalibration ... done meanwhile
• Wajid, Joakim, Tommaso, and Theo will repeat  SASE3 DAQ Imager/XGM correlation today (need single shot operation)

Advanced Electronics (P. Gessler):

• Broken terminal between SASE3 MOV2 PLC Loop and Master Loop were replaced last Friday without any side effects. The Train ID is now also provided from the MOV2 loop.
•  So far, no PLC related export from the SASE2 Tunnel eplans of the vacuum, beam transport and diagnostics PLC crates are possible. Therefore we will build up the XLS file for the start up of the VAC and XGM loops next week. However, the related group plus AE and EETF should meet to evaluate, if an export is possible until end of March in order to use it for further installation work. Otherwise XLS files have to be crates manually again with support from all involved groups. Also consistent documentation is required between diagnostics on one side and vacuum and beam transport eplans.

SASE2 Tunnels (C. Youngman)

• Table with the set priorities and deadlines available in Indico.

ITDM (K. Wrona):

• Run Configuration Groups required for the user experiments for FXE and SPB/SFX have been defined and tested
• Discussion after the meeting: Since the Karabo-DOOCS interface for XGM will be changed and it will affect also how the data arrive at DAQ data aggregators we need to modify and (re)test the BEAM_COND group.

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIPD will operate with the old firmware due to problems with the new one
• No calibration data available as of today
• AGIPD is in a safe state

CAS (S. Brockhauser):

• Gotthard for HIREX being operated
• Progress on GATT commissioning by Valerii

SPB:

• Ion pumps running in the Mirror chamber. Vacuum level OK to run.
• D.02 is now interlocked and is a clean area.
• Mahadi and Alessandro are setting up a test system for the coupled movement. The test will probably be done between the 2 users periods.
• 2 hours of parasitic beam can be used today to check the alignment of the CRLs. The CRLs will be then moved out of the beam to allow the FXE radiation tests tomorrow.
• Primary limits for coupled motion not included in E-plan, safety switches are implemented but SPB prefers to have the primary limits included. CSS crate to be modified after clarification with AE/EET.
• Test of the karabo brigde will be performed with users. It is not yet clear exactly when since this needs synchronization with several groups involved (CAS, ITDM, DET). 

FXE:

• Radiation test tomorrow
• LPD test measurements in thhe WE
• Von Hamos spectrometer alignment until monday

ITDM:

• SASE1 and 3 have the last DAQ update now. Stable in SASE1 so far.
• Data sources arrangement being worked out. Today a 45 min session is foreseen with SPB.
• SPB needs slow data acquisition from 2 cameras for users, apart from AGIPD.

VAC/Detector/XPD/CAS: nobody attended

SPB:

• Ion pumps running in the Mirror chamber.
• Starting test with mirrors together with CAS this week.
• In general everything in good position to receive beam.
• Primary limits for coupled motion not included in E-plan, safety switches are implemented but SPB prefers to have the primary limits included. This will suppose a hardware modification.

AE:

• SASE3: train IDs in loop2 are working, the problem was fixed last Friday.

ITDM:

• Measurements for commissioning continues.

CAS:

• H.Santos inform that CAS has not resources to run tasks for SASE2, this might affect the date for first lasing in SASE2. Priorities are SASE1 operation and SASE3 commissioning.
• G. Wellenreuther will bring this topic to the PSO meeting.

FXE/VAC/Detector/XPD: nobody attended

Photon Run Coordinator (L. Samoylova):

• SASE1 tuning is ongoing (350 microJ)
• SASE3 3000microJ
• SASE1 interlocks enabled, SASE3 only vacuum interlock enabled
• Soft monochromator is being commissioned
• K-mono studies were carried out
• HiRex was being commissioned
• SASE1 and SASE3 DAQ was used
• Gain curves should now be possible

SPB/SFX (M. Messerschmidt):

• Recorded data with detector integrated in Karabo
• problems with coupled motions persist, ran into & were not able to get out of end switches using Karabo
• leak at mirror tank still has to be localized

FXE (P. Zalden / A. Galler):

• Will be ready for beam by the end of the week
• chamber was pumped, crystals mounted
• I0 monitor was tested and commissioned
• only have permission for <=2 bunches
• scantool: Train IDs are not captured
• DAQ-tests look good for LPD

PP-Laser (M. Lederer):

• Final commissioning of the laser is ongoing
• integration of the laser into DAQ during the April shutdown

Vacuuum (M. Dommach):

SASE1

• nothing to report; Vacuum system ready of beam operation

SASE3:

• XTD10 was open last Tuesday
• Together with Diagnostic group Exchange of Turbo pump on XGM
• Gas Attenuator commissioning ongoing

SASE2:

• Working on cabling of XTD1 components and vacuum installation in XTD6
• Installation of cleantent at HED-SDU ongoing.

XPD (J. Grünert):

XGM SASE3 

• tuesday XPD made XTD10 access to repair SASE3 XGM vacuum
• one turbo pump failed Saturday noon, decided to repair during access tuesday, exchanged pump
• tuesday detected another vacuum trip in the differential pumping around the XGM
• wednesday afternoon VAC made ZZ-access to „repair“ what tripped the vacuum interlock
• wednesday to thursday night: checking proper and stable SASE3 vacuum operation - is ok
• XGM is back in operation since thursday morning 8am

XGM status overall:

• XTD2 and XTD10 XGMs in operation
• When will the XGM in XTD9 be needed? ==> Harald and Liuba believe that beam might be sent to XTD9 this weekend, therefore they request to start XTD9 operation today afternoon

Imagers 

• SQS Exit slit imager has been set to beam position and can be used for some basic commissioning (limited functionality)

thursday work with beam:

• SASE1 KMONO studies: two cells studies, height adjustment of single cells using DAQ (until it crashed at noon), data analysis today
• HIREX commissioning
• SASE1 beam conditions recorded with DAQ

Correlation of beam intensity monitored by XGM and imagers

• Required e.g. for the current Gas Attenuator Transmission Calibration
• need to prepare a simple table of calibration values for all imagers which are used as supplementary intensity monitors
• will discuss offline among photon run coordinators

SASE2

• need to rearrange space in one rack to be able to put our last crate into tunnel XTD1: Wolfgang will contact Ulf and VAC
• Request to CAS: 
• DOOCS devices shall be created for SASE3 (as in SASE1) for undulator and phase shifter and electron beam energy

Advanced Electronics (P. Gessler):

• SASE3: No train IDs are being delivered in loop 2. Repair could potentially / in the worst case affect all electronics. ==> Should be fixed right now to get it done.

ITDM (K. Wrona):

• Configuration / Testing of DAQ continues

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIDP ready for 100/64 bunch operation
• there will be a two-day training for AGIPD-operators
• LPD was tested with new Karabo / DAQ
• LPD has hardware problem regarding power line for one module, to be fixed

CAS (S. Brockhauser):

• The issue reported in the beginning of the week regarding the TrainIDs was identified. We have an integer overflow problem when connection is lost with the timeserver. That happens sporadically, and we are working to fix it since yesterday.
• We released Karabo Hotfix 2.2.2.2 to fix an issue on the RunConfiguratorGroup at SA1 (ITDM already checked and it is working)
• We have improved the communication protocol with AE on handling loops/PLC updates and related (re)establishment of the Karabo services.
• AGIPD library is extended to supply information if detector is in stop or acquire state. The next step will add the "acquisition time" to enable scantool integration
• The old digitizer is running and integrated with DAQ. Also, we will add the "acquisition time" to enable scantool integration
• We started to call the Sprint Meetings (for the moment we focus on updating the priorities of FXE and SPB topics)

AoB:

E. Boyd: A test @ FXE with Cu-Target and 300 bunches is required. This will affect the areas / hutches around FXE

A. Mancuso: Will try to work around this, but in general a message one week beforehand would be required / appreciated.

SPB

• KB chamber being pumped down
• Smaract commissioning ongoing
• Coordinate motion for VAC ready to be tested (probably tomorrow). It has to be agreed with Richard Bean
• Clean-up slits Karabo middle layer deployed and working. - The same hase to be done for the power slits.

 FXE

• DAQ test ongoing - fixing of bugs
• After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Still the case.

• ​XPD

• XGM in SASE3 fixed (A turbo broke - replaced)
• One exit slit Imager in XTD 10 commissioned

AE

• PBLM moto problem was found to be due to a wrong Karabo-PLC linking. ==> FIXED
• Soft mono commissioning started and AE support was required. DONE

ITDM

•  Main problems with Train ID solved. Some problems still remain for some beckhoff loops in the tunnels==> AE on that.
• Some way to document the change of data format for devices has and will be found by IDTM
• Data source running for FXE. Changes have to be clearly communicated. 

CAS

• Soft fix improving the GUI deployed. No 2.2.2 ready.
  

• ​Issue with Karabo device returning the sensors data (e.g. cooling AGIPD) is being investigated. The Bug was found also in test environment but still random.
   

SASE3 - CY

• GATT works
• Inficon device to be improved (loop restart needed)
• CAS will deliver a method to deal with the configuration changes asked by the experiments, redirecting the changes to the proper platform (Firmware (AE), Karabo...)

Small correction:

The coordinated motion is the KB-mirror's, not the Vacuum.

The postfix is 2.2.2.1 and includes features that were not included in 2.2.2

Karabo 2.2.2 is ready

 FXE

The FXE beamshutter key has been installed into the interlock system after satisfactory completion of the FXE Chicane and Radiation Protection Sticker Checklist. The FXE Experiment hutch is now ready to receive xfel beam as required by the FXE instrument group.

However, the bunch mode MUST be limited to 1 or 2 bunches until the new user chicane is tested with thirty bunches using the copper target. This can be done whenever the parameters can be practically put into place. The test will be similar to the hutch commissioning where once it has been confirmed that the beam is on target. I will take radiation measurements at the chicane. This is the only place where radiation measurements will be required. Please provide me as much notice as possible prior to the date and time that 30 bunches will be brought into the FXE Experiment hutch.

I have also place 3 additional area monitoring dosimeters in the FXE Experiment. One on the robot arm where the D3 pandora place holder is. One above the old user chicane on the D.05 side and one inside the new user chicane on the control room side.

SPB

• KB chamber being pumped down
  

• interaction with AE is ognoing concerning update pf Loop 5 and AGIPD cooling procedure ongoing
• Smaract commissioning ongoing
• The goal is to be ready to start commissioning with beam on 01.03.
• Slit Karabo middle layer - integtration and final fixing being finalized by A. SIlenzi

 FXE

• DAQ test ongoing - fixing of bugs
• After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Being discussed with CAS
• Permission to use parasitic beam next week requested to the radiation safety group.

​XPD

• XGM in SASE3 requires physical intervention to be fixed. Access request for tomorrow.
• DAQ groups definition ongoing and being clarified with ITDM

AE

• PBLM moto problem was found to be due to a wrong Karabo-PLC linking. If the tunnel will be accessible tomorrow the issue will be solved. HSinn to decide on priority.
• Soft mono commissioning started and AE support was required. It is not clear what priority does it have. Mahadi is anyway working on that today ad 14:00. If the intervention is found to require more time one has to see.

ITDM

•  During the FXE DAQ tests many problems were found:
  • issues with digitizers
  • issues with scan
  • issues with images acquisition. In this case the problem has been identified (Train ID was 0). KWrona is in contact with CAS to fix the bug.

CAS

• The issue reported by SPB about no data being read from AGIPD temperature sensors has been identified by Gero (CAS) and the fixing is ongoing. 

Report of the XTD10 interventions from last week - Tuesday and Wednesday - C. Youngman 

Many thanks to those who contribute to the "SA3 'further technical coordination push" onTuesday (no-beam XTD10 closed) and Wednesday (XTD10 open) mini-shutdown, below is brief summary. The status items were not highlighted during PSPO's readiness meeting AoB section but are useful to know. Those issues that were raised are contained in the conclusion section. 

We leave the SA3 beamline with a functioning interlock, but you should additionally watch and test your component and ensure that beam operation is possible.

The principle task on Tuesday was

• for ITDM to shutdown and update f/w of the control server CPU hosts in the balcony room, and
• for CAS to replace the Karabo 2.1.18, running since the Oct-Nov shutdown 2017, with Karabo 2.2.1 which is already running on the SA1 installation. 
  • A number of problems during the installation were addressed and a stable deployment was finally reached early in the evening.

The principle task on Wednesday were

• test the updated GATT Ebara pump updated f/w and its s/w counterpart
• test that GATT Bronkhorst Mass-Flow valve f/w worked: correcting conversion errors and the incorrect setting of property r/w attributes
• test that GATT Inficon gauge f/w did not crash the loop and correct  conversion and r/w errors as with the the Mass-Flow valves
• perform Beckhoff-local tests on one or more the DOD motor axes as they were expected to be wrongly wired (as seen in the Jan-Dec shutdown in the VSLIT)
• update the s/w to reflect changes in the GATT f/w
• address difference in the s/w introduced by the move to Karabo 2.2.1

The key-point task listed above have been attained with the following caveats:

• There is a h/w fault which prevents the second Ebara pump being fully controllable remotely, it is currently locally controlled.
• Bronkhorst f/w and s/w is usable, the conversion is correct, but additional s/w work is required to align the s/w
• The Inficon f/w works (it does not crash the loop), the f/w conversion error is identified but not enough time was available to modify it.
• some s/w issues remain which are being addressed (tagging of Ebara, Bronkhorst and Inficon s/w, correct configuration of the vacuum-section..)
• the automatic update of project and scenes was not completely successful so project and scene owners should check and correct their scenes. The auto update procedure will be improved.
• Some data logging information was lost, but e now have regained a standard implementation (not hot-fixed) which should improve the this.
• After measuring that the DOD axes (1 of 10) were incorrectly wired it was concluded that all were incorrectly wired (as had been the VSLIT in the Dec-Jan shutdown)
  • Benoit and Daniele managed to rewire all 10 DOD axes during the mini shutdown and test 1 axis with Karabo.
• PES tests of multi-board digitizer s/w and f/w were continued (with measurements at the crate), these are promising as 'raw' and 'processed' data are now concurrently available.

In conclusion - good progress was made and this type of 'commissioning' mini-shutdown (anything longer than ~4 hours which includes somebody elses longer ZZ) should be performed as often as possible.

This requires

• that the photon run coordinator team know that there is a "list of task" to be performed which are not in the ZZ list
• that a selection of these tasks can be performed within the time budget of a mini-shutdown and that these do not compromise SA3 tunnel operation
• that the photon run coordinators inform the SA3 technical commissioning team (contact a member of the "SA3 tech coord" CAS team setup by Sandor, e.g. Chris, Valerii or Wajid ) so that they can organize the work in time (or report that they cannot as something or someone is not available).

During the next few days a "list-of-task" will be created. This could include technical commissioning actions for SA3: PBLM, ExitSlits, very end of beamline imagers where the status needs updating, etc.

Also remember that the CAS team tries hard to satisfy as many requests as possible, but prioritization will be done, so make your case thoroughly and don't expect it to simply happen (it has to be on the radar screen). In particular h/w, f/w and related s/w tests needing access will be done first and non-critical s/w work will be left until after tunnel closure. And remember to check your projects, scenes, and macros before you need them.

Here is a information from Sandor on CAS OCD Numbers
(this info is copied from the SASE1 commissioning logbook): 

during the day: 98053   (8:00 - 16:00)

overnight: 98054  (11:00 - 8:00 next day) 

during the weekend it is #53 or #54  (Saturday #53 8:00 - 8:00 next day, Sunday #54 8:00- 8:00 next day)

Photon Run Coordinator Report to Readiness Meeting

The maintainance on Wednesday was very successful and all goals were reached (and some extras). 

Thanks to everybody involved in planning and execution! 

- Installation of Karabo 2.2.x in XTD10 went smoothly. 

- SASE1 tuning studies are ongoing. Still there are some things to be learned on compression and 

optimizations will go on Friday. Currently around 250 uJ.

Recent achievments in commissioning: 

SASE1: 

-Peter aligned CRLs in XTD2 to beam. 

-Naresh could work on Hirex

-Liuba and Ilia started pointing studies with imagers (?)

Problem: We found out that gate vales of vacuum are not connected to DESY MPS

(waek beam was hitting one closed valve behind Hirex for about 2 hours)

SASE3: 

- Gas attenuator commissioning started

- Calibration of SRA to optical center of beamline

- The undulator beam was steered 4 mm horizontally. The vertical steering requires movement of undulator center, 

which will be tried possibly on Friday

- Cross calibration of XGM to FEL imager started last night. 

SPB

- KB mirrors

        - installation of horizontal micron KBs finished (approx 8 working days were required)

        - pump carts attached, expected that one week is required before valves can be re-opened (Vacuum group needs ZZ to reconnect valves on the XTD9 side)

        - installation of vertical micron KBs in April

        - D02 will remain cleanroom during the intervening time, i.e. all companies / groups (internal and external) coming on site should be briefed. No access without permission and accompaniment

        - controls tests on elmos completed via Karabo, help from Nerea / Mahadi to get everything working as expected

- configuration of smaract motors

        - commissioning setup will be used during first beam week in March

        - Re-config of smaract controllers by Marcin / Grant / Mahadi mostly completed

- AGIPD

        - firmware update scheduled for this week will not be completed as DESY detector group firmware expert is not available

        - remain limited to 32 patterns per train

        - automatic cooling and powering procedures are not working reliably (requires investigation poss. CAS?)

        - AGIPD chiller fell into an error state after removal of a separate crate BKF00072 (SPB_IRU_LIC), loop 5 redundancy requires investigation (AE)

- Karabo

        - GUI is slow to update or does sometimes not update (request investigation by CAS)

        - devices (e.g. those reporting pressure / temperature) have to be re-started before updates are seen in Karabo (request investigation by CAS)

        - Users have questions about any changes in the implementation of the 'Karabo bridge’. Technical information on how information is broadcast from Karabo is required in order to ensure that users can adapt their monitoring software 

- data trial run

        - Robert suggested trial run for DAQ / user software plugin before the first experiment. SPB can contribute to the execution of these tests and involving users. Requires also action from CAS and ITDM teams

FXE

• Diamond grating in XTD9 can be controlled remotely now
• DAQ test ongoing
• LPD calibration to be  performed

Laser:

•  Installation according to the plan, no major delays. 

Vacuum

SASE1:     ZZ access to connect SPB valves

SASE3:     tripping of interlock seems to be understood, interlock conditions changed,

                if it happens again, the two effected gauges will be removed from interlock until April

           Gas Attenuator is operational, Raul did some tests yesterday, up to 1 mbar

           No feedback control on gas pressure, only manual setting of gas flow

           The device can only be operated by experts so far

Optics

• see PRC section 

XPD

Last wednesday tunnel access:

• „repaired“ SA1 IMGSR trigger (with AE)
• „repaired“ SA3 XGM, restarted (but intentionally without HAMP)
• better routing / fixing of purple Beckhoff loop Ethernet cables and fiber connections is needed

SASE3 vacuum interlock trips (which eventually led to XGM failure on Saturday and reoccured monday night again) were strongly investigated by VAC : now understood ?

work on cross-calibration between XGMs and Imagers (as backup intensity/position monitors when XGM is out of order) with Laurent

DAQ: work with Krzysztof and Luis to get SASE1 DAQ back in operation, and to start SASE3 DAQ (can we test today ?)

Current status of SASE delivery (max. µJ in SA1/SA3): yesterday 390 / 200, this morning 190 / 600

Request to CAS: please plan to send for "training" or "experience" some CAS people to BKR during beam operation, especially the people in CAS who do OCD.

On behalf of Natalia : report to AE on detected issues with the ELMO controller settinsome hours gs of the PBLM motors in SASE1

AE

- stuck trigger timing module in XTD2 restarted (related to SR imager)

- Gas Attenuator PLC loop updated

- PLC modules in SPB rack with SmarACT system rearranged according to updated documentation

- Two bugs in PLC Framework identified, which are fixed now in the current version, but is not deployed yet to most systems:

     - Access rights are not correctly transferred to Karabo. This leads to the fact, that in cases of very special parameters only read access is possible via Karabo (these are automatically injected parameters in Karabo). If you come to the problem of not being able to write such a parameter, please let AE know and we can set it directly to the PLC. This will be solved with the next PLC loop update.

     - A bug was identified, which could in the worst case lead to a full crash of the related PLC. This could be caused by a „re-init“ on a device on the PLC. Therefore do no re-inits on PLC devices. If you really see the need for a re-init of a certain device, please let AE know about this and we see, if this is required and how this could be done. Also this issue will be fixed with the next scheduled update of the PLC loop.

ITDM

•  Cameras issue solved.
• SASE3 DAQ testing to start
• XGM data acquisition to be tested due to karabo/DAQ possible lost of information recently debugged.

CAS

+ Karabo 2.2.2 is released

+ 3rd control pc is set up in BKR

+ SASE3 is running Karabo 2.2.1

+ Karabo 2.2.1 is tested with DAQ

+ as we moved into weeks of commissioning, CAS OCD is regularly (basically every night) involved in and provides support for partly software and partly hardware issues

+ as experiments are coming, Data Analysis integration requests (eg. on Karabo Bridge) shall be channeled to Hans (and to the newly setup DA contacts: Thomas for SPB and Cyril for FXE) directly

+ the gui slowness problem reported by SPB needs to be checked if it comes from the GUI client or from the server and shall be tested with the new Karabo hotfix 2.2.2, too.

DET

LPD canlbration analisis ongoing

LPD Long integration time acquisition being tested

AGIPD: colntrol optimized and tested. Firmware being tested, the aim is to ioncrease the acquisition speed.

AGIPD: next week calibration data to be taken. 

AGIPD: Sensors in the centra area that underwent scattered beam to be checked.

CYoungman - SASE3

 - good progress was made and this type of 'commissioning' mini-shutdown (anything longer than 6 hours) should be performed as often as possible (Wed. shutdown). Next work items would involve: PBLM, ExitSlits....

LBatchelor - XTD6

XTD6: both CRLs (MID CRL2 and HED CRL2) ready for installation (lenses installed and aligned, chambers sealed). Installation should happen next week.

SPB

• MIddle layer karabo devices are responsible for the det. Karabo problems. Issue being addressed.
• 2 mirrors in D.02 installed.
• Test of the mirrors movements is done.
• Mirrors cooling bars being installed

Vacuum

• SASE3: 

  • Spikes of the downstream DPS gauges are responsible for the interlocks tripping. Probable cause is related to the beckhoff terminals of the DPS (different from the other gauges). Ongoing.

XPD

• Pop-in monitor filters in XTD10 being changed
• Exit slit imager after Mono XTD10 has ot be commissioned and scintillator has to be put out of the beam way.

AE

•  It can happena that some parameters cannot be set using Karabo due to lack of writing rights. A fix has been found but we would need to update all the loops. Since the problem is just occurring from time to time, AE asks to report specific cases for whoch a solution will be found. Same things happens with the re initialization of some devices. In this case the PLC crashes.
• Timing device in XTD 2 stopped working. Will be fixed today either by remote re start or by local intervention.

ITDM

• Broken server replaced
• Tomorrow there will be an update of the cluster between 9:00 and 10:00. DAQ will be down.
• Reboot of the online cluster needed. To be agreed with Detector group. 1 hour downtime also in this case.

CAS

•  In SASE3, deployment of new (2.2.1) Karabo happened yesterday and seems stable, but brittle. Analysis of the reason ongoing.
• Today GAT final fixes plus testing will be performed

SPB

• During operation Karabo crashed. As soon as detector is being operated the problem appears. Fixing is still ongoing.
• 2 mirrors in D.02 are being installed.
• Test of the mirrors movements has been performed.

FXE

• Debug of implemented devices ongoing
• Diamond grating in XTD9 has to be moved out of the beam way manually (no remote controls implemented due to long lead time of required beckhoff teminals). Access requested for Wednesday.
• Vac group will check whether beckhoff terminals are available to allow the remote control of the dimaond grating

Vacuum

• During the weekend the XGM in SASE3 was damaged. Details still unknown. 

AE

•  SASE2 XGM local test ongoing
• GAT loop update in SASE3 being done.
• Local checks in XTD10 on Wed.

ITDM

•  Slow data acquisition issue affecting imagers: solved
• One server died over the week end. ITDM notified only today.