Photon Run Coordinator Report to Readiness Meeting

The maintainance on Wednesday was very successful and all goals were reached (and some extras). 

Thanks to everybody involved in planning and execution! 

- Installation of Karabo 2.2.x in XTD10 went smoothly. 

- SASE1 tuning studies are ongoing. Still there are some things to be learned on compression and 

optimizations will go on Friday. Currently around 250 uJ.

Recent achievments in commissioning: 

SASE1: 

-Peter aligned CRLs in XTD2 to beam. 

-Naresh could work on Hirex

-Liuba and Ilia started pointing studies with imagers (?)

Problem: We found out that gate vales of vacuum are not connected to DESY MPS

(waek beam was hitting one closed valve behind Hirex for about 2 hours)

SASE3: 

- Gas attenuator commissioning started

- Calibration of SRA to optical center of beamline

- The undulator beam was steered 4 mm horizontally. The vertical steering requires movement of undulator center, 

which will be tried possibly on Friday

- Cross calibration of XGM to FEL imager started last night. 

SPB

- KB mirrors

        - installation of horizontal micron KBs finished (approx 8 working days were required)

        - pump carts attached, expected that one week is required before valves can be re-opened (Vacuum group needs ZZ to reconnect valves on the XTD9 side)

        - installation of vertical micron KBs in April

        - D02 will remain cleanroom during the intervening time, i.e. all companies / groups (internal and external) coming on site should be briefed. No access without permission and accompaniment

        - controls tests on elmos completed via Karabo, help from Nerea / Mahadi to get everything working as expected

- configuration of smaract motors

        - commissioning setup will be used during first beam week in March

        - Re-config of smaract controllers by Marcin / Grant / Mahadi mostly completed

- AGIPD

        - firmware update scheduled for this week will not be completed as DESY detector group firmware expert is not available

        - remain limited to 32 patterns per train

        - automatic cooling and powering procedures are not working reliably (requires investigation poss. CAS?)

        - AGIPD chiller fell into an error state after removal of a separate crate BKF00072 (SPB_IRU_LIC), loop 5 redundancy requires investigation (AE)

- Karabo

        - GUI is slow to update or does sometimes not update (request investigation by CAS)

        - devices (e.g. those reporting pressure / temperature) have to be re-started before updates are seen in Karabo (request investigation by CAS)

        - Users have questions about any changes in the implementation of the 'Karabo bridge’. Technical information on how information is broadcast from Karabo is required in order to ensure that users can adapt their monitoring software 

- data trial run

        - Robert suggested trial run for DAQ / user software plugin before the first experiment. SPB can contribute to the execution of these tests and involving users. Requires also action from CAS and ITDM teams

FXE

• Diamond grating in XTD9 can be controlled remotely now
• DAQ test ongoing
• LPD calibration to be  performed

Laser:

•  Installation according to the plan, no major delays. 

Vacuum

SASE1:     ZZ access to connect SPB valves

SASE3:     tripping of interlock seems to be understood, interlock conditions changed,

                if it happens again, the two effected gauges will be removed from interlock until April

           Gas Attenuator is operational, Raul did some tests yesterday, up to 1 mbar

           No feedback control on gas pressure, only manual setting of gas flow

           The device can only be operated by experts so far

Optics

• see PRC section 

XPD

Last wednesday tunnel access:

• „repaired“ SA1 IMGSR trigger (with AE)
• „repaired“ SA3 XGM, restarted (but intentionally without HAMP)
• better routing / fixing of purple Beckhoff loop Ethernet cables and fiber connections is needed

SASE3 vacuum interlock trips (which eventually led to XGM failure on Saturday and reoccured monday night again) were strongly investigated by VAC : now understood ?

work on cross-calibration between XGMs and Imagers (as backup intensity/position monitors when XGM is out of order) with Laurent

DAQ: work with Krzysztof and Luis to get SASE1 DAQ back in operation, and to start SASE3 DAQ (can we test today ?)

Current status of SASE delivery (max. µJ in SA1/SA3): yesterday 390 / 200, this morning 190 / 600

Request to CAS: please plan to send for "training" or "experience" some CAS people to BKR during beam operation, especially the people in CAS who do OCD.

On behalf of Natalia : report to AE on detected issues with the ELMO controller settinsome hours gs of the PBLM motors in SASE1

AE

- stuck trigger timing module in XTD2 restarted (related to SR imager)

- Gas Attenuator PLC loop updated

- PLC modules in SPB rack with SmarACT system rearranged according to updated documentation

- Two bugs in PLC Framework identified, which are fixed now in the current version, but is not deployed yet to most systems:

     - Access rights are not correctly transferred to Karabo. This leads to the fact, that in cases of very special parameters only read access is possible via Karabo (these are automatically injected parameters in Karabo). If you come to the problem of not being able to write such a parameter, please let AE know and we can set it directly to the PLC. This will be solved with the next PLC loop update.

     - A bug was identified, which could in the worst case lead to a full crash of the related PLC. This could be caused by a „re-init“ on a device on the PLC. Therefore do no re-inits on PLC devices. If you really see the need for a re-init of a certain device, please let AE know about this and we see, if this is required and how this could be done. Also this issue will be fixed with the next scheduled update of the PLC loop.

ITDM

•  Cameras issue solved.
• SASE3 DAQ testing to start
• XGM data acquisition to be tested due to karabo/DAQ possible lost of information recently debugged.

CAS

+ Karabo 2.2.2 is released

+ 3rd control pc is set up in BKR

+ SASE3 is running Karabo 2.2.1

+ Karabo 2.2.1 is tested with DAQ

+ as we moved into weeks of commissioning, CAS OCD is regularly (basically every night) involved in and provides support for partly software and partly hardware issues

+ as experiments are coming, Data Analysis integration requests (eg. on Karabo Bridge) shall be channeled to Hans (and to the newly setup DA contacts: Thomas for SPB and Cyril for FXE) directly

+ the gui slowness problem reported by SPB needs to be checked if it comes from the GUI client or from the server and shall be tested with the new Karabo hotfix 2.2.2, too.

DET

LPD canlbration analisis ongoing

LPD Long integration time acquisition being tested

AGIPD: colntrol optimized and tested. Firmware being tested, the aim is to ioncrease the acquisition speed.

AGIPD: next week calibration data to be taken. 

AGIPD: Sensors in the centra area that underwent scattered beam to be checked.

CYoungman - SASE3

 - good progress was made and this type of 'commissioning' mini-shutdown (anything longer than 6 hours) should be performed as often as possible (Wed. shutdown). Next work items would involve: PBLM, ExitSlits....

LBatchelor - XTD6

XTD6: both CRLs (MID CRL2 and HED CRL2) ready for installation (lenses installed and aligned, chambers sealed). Installation should happen next week.

Photon Run Coordinator (L. Samoylova):

Commissioning during machine studies Tue and Wed and parasitically during SA1 user operation at 
 SASE3 on Thursday and Friday 

SASE1

SPB

User operation, current status (Adrian): got few data from a test system on Saturday. The users’ 
sample delivery technology now seems to deliver nice & stable droplets, so if SPB 
get beam early enough today to set up for a measurement, a full data set could be collected. 
 

FXE

• Instrument Commissioning
• Measure beamline transmission (unfocused 550 µJ per pulse at the sample position, with lenses 
  115 uJ)

 Machine studies

• Wed afternoon: a set of K-mono FEL spectra taken

https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/07.03_a&pos=2018-03-09T17:56:20 
 SASE3

 Thu, Friday - a first experience of the SASE3 commissioning in 30 bunch mode during user operation: 

• minimal number of undulators (12) for lasing, low intense SASE (XGM shows 10 uJ, linear 
  regime), SRA closed to 2 mm
• MCP2 @ SASE3 - signal from 30 SASE pulses, consistently decrease with closing the slit down 
  to 0.5 mm

https://ttfinfo.desy.de/elog/XMLlist?file=/XFELelog/data/2018/10/08.03_a&xsl=/elogbook/xsl/elog.xsl&picture=true#2018-03-08T18:24:14 

• Fri afternoon GATT calibration with XGM and MCP ongoing, in 30b mode, optimizing SASE3 lasing 
  for these measurements needed - too weak and not homogenous signal in the center, see 
  https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/2018/10/08.03_a&pos=2018-03-08T18:26:50

  Problems:

 'unexpected' from photon side closing XTD1 two working days earlier, explicit schedule 

• for the south branch tunnels access would be appreciated
• power off in XHPSC on Thursday at 21:15, no cooling water in experimental hall that led to 
  increasing the temperature in the rack with the MicorTCA crate, the CPUs of the two MicroTCA 
  crates in SPB and one of the two crates in FXE were automatically switched off. SPB crate 
  1 did not come automatically back afterwards, as the result: no train IDs for SPB PLC 
  loops, no MPS for SPB, no Clock and Control for AGIPD, no triggers for SPB systems on 
  Friday
• Saturday morning the MPS interlock takes into account the signal from one of three fast valve 
  on SPB shutter although these valves are disconnected to SPB EPS interlock. The MPS 
  bit had to be masked

Correction from SPB [12.03.2018 10:10]: This issue was not related to the XTD9 shutter, but to the 
optics hutch shutter. The fast valve in XTD9 is connected to the EPS, the fast valve 
in the OH is not yet connected to the EPS. After the restart of the EPS/MPS on Thursday 
/ Friday after the cooling water issue, the condition for the fast valve in the optics 
hutch was wrongly included in the MPS. 

SPB

1) Thursday 08.03.2018 (8 am - 8 pm) - SPB/SFX commisioning shift

    Notes:

          - everything running

          - RAW data overview very responsive with  data filtering as suggested by Steffen

          - Corrected data preview started very nicely and seem to provide only limited update rate after a few hours running

          - karabo DAQ deices (dataaggregators) need several rests, resulting in a full DAQ restart around 7pm

          - sometimes updates in KARABO were lost

          - preparation for users experiment - over night

2) Thursday 08.03.2018 (around 9 pm) - "cooling water" incident

     Notes:

         - vacuum pumps affected

         - AGIPD warmed up

         - uTCAs have tripped/shutdown (some infrastructure needs to replaced)

3) Friday 09.03.2018 - morning (User beamtime)

         - damaged uTCAs equipment was replaced but trigger setting needs to be input/ adjusted =  loss of the user beamtime

         - User experiment progresses

SQS:

Hutch cleaning Mid. March

KB tanks installation in parallel with SCS

9-14 air filter will be installed. After that date only clean works have to be carried out

Collision with KB tanks installation and other activities like Rad. Interlock test to verify

FXE (P. Zalden / A. Galler):

Setting up the experiment

IPM and XGM data manualy correlated. Would be nice to have the train id working to have the "live" correlation

SCS (J.T. Delitz)

• cleaning of hutch next week (wk11)
• we will start moving first components wk12-13, ie. ALAS and DPS
• KB installation (positioning only, no AC+filter required) beginning April-depends on commissioning of precision climate zone (currently discussed with PI)

PP-Laser (M. Lederer):

SASE1 commisisonin gonoing.

Some misalignment due to the cooling water accident. To be investigated 

09.04 SASE2/3 laser tables will be installed.

Vacuuum (R. Villanueva):

SASE2--------->Good progresses on XTD1 control system deployment: Differential Pumping sectors are ready for first beam // First five vacuum sectors under vacuum and locally tested (already with Karabo). XTD6 tunnel installation phase progresses at good pace.

SASE1--------->In general, nothing additional to remark. We are working in a more robust interlock definition to prevent false triggering due of ion pump control signals.

SASE3--------->Complete vacuum system is working normally. Gas Attenuator is now under routinary operation up to 2-3 mbar with a UHV interface pressure after its DPS sector < 1*10e-9 mbar.

XRO 

SQS/SCS/XRO collaboration for the mirror coating ongoing

Installation of M5 in preparation

XPD (T. Maltezopoulos ):

XGM@XTD2, @XTD9, and @XTD10 are online.

XGM@XTD1 is in the loop and accessible from Karabo

(together with AE+CAS).

Currently I test XGM@XTD1 components from DOOCS.

XGM@XTD9 shows slightly higher energy then in XTD2.

I calibrate all XGM components twice, but still it remained.

Note that the beam pass the XTD9 XGM with -7 mm offset

(in beam direction 7 mm to the left, to the south).

With this high offset absolute calibration is not guaranteed.

Advanced Electronics (P. Gessler):

1 CPU for SPB broken due to the cooling water accident. Already replaced. Being configured

FXE grating integrated in the vac loop

SASE2 Tunnels (C. Youngman)

More coordination is needed, especially concerning the access to the tunnels. For the technical details, please see slides on the indico page https://indico.desy.de/indico/event/20115/

ITDM (K. Wrona):

No problems with overrheating

Problems with Train ID reliability after the heating. Should be checked and monitored online

Documentation of single devices have to be improved. + there is need for a tool to make the sanity check

Detectors (J. Sztuk-Dambietz, P. Lang):

LPD commissioning OK + DAQ working. Now LPD off because of cooling water problems.

CAS (H. Santos):

• Sandor is in a meeting with the Detector group to align requirements regarding AGIPD and DA
• We started to prioritize and address the (agile) missions of SPB and FXE. As part of this effort:
  • Alessandro started to develop the scenes link on the GUI. This feature is really required and has been present in several SRDs (For example, to link vac sections/loops).
  • We implemented the GUI combo box to select different GUI Server - it is part of the SPB Mission (Leonce)
• Hamed and Sergey made good progress on the Digitizer. They will work on a demo/talk for the next PBS Meeting.
• Wajid, Sergey, and Dennis fixed the X2Timer issue reported by FXE (hash/data inconsistency). It is already deployed and running.
• Regarding the cooling incident. Wajid was the first contacted yesterday. After that Alessandro at 01:00. Today, during the morning, we reinstall Karabo at SPB-RR-SYS-UTC-1.

General

"In the beginning of this meeting SM congratulated the whole staff of the European XFEL and in particular the commissioning team with the very important achievements 
- the first SASE3 lasing / bi-lasing at SASE3/SASE1. It happened yesterday late evening. More information will be given during our PBS meeting next week."

Photon Run Coordinator Report to Readiness Meeting

• parasitic operation in SASE3 (closed 3 undulator segments) for photon commissioning with SR-beam
  • V0 opened, vacuum open down to XHEXP-shutters
  • photon beam detected down to FEL imager, yesterday until SR absorber beyond mirrors M1/M2 and MCP
• photon systems supported tuning/search for SASE1 lasing
• Progress in SASE3: 
  • First Lasing in SASE3/XTD10 
  • simultaneous lasing SASE1 and SASE3

SPB

• During operation Karabo crashed. The problem is not clear yet. Det. loop was affetcted ==> AGIPD tests stopped!
• 2 mirrors in D.02 are being installed.
• Test of the mirrors movements is ongoing
• Motors issues are being checked by AE (Mahadi)

FXE

• Gotthard det. and digitizers back to work
• Leak close to the diamond window being fixed

Laser:

• Timing system has been upgraded - currently we can have any repetitive laser pattern for each laser, number of laser pulses in not limited to number of electron bunches, also we can have lasing when there is no x-ray (electron bunch)

•  Installation according to the plan, no major delays. No commitment to experiments for laser availability in March.

Vacuum

• SASE2:

  • XTD1 under vacuum up to XS2 shutter

• SASE3: 

  • interlock problems solved, runs without problems since yesterday

  • GasAttenuator loop needs update. when is the best time to do that? to be agreed with PRC

Optics

• issues mainly in the XPD section 

XPD

• SASE1 : photon systems supported tuning/search for SASE1 lasing
  • tests of DAQ in SASE1 ongoing
• parasitic operation in SASE3 (closed 3 undulator segments) for photon commissioning with SR-beam
  • V0 opened, vacuum open down to XHEXP-shutters
  • photon beam detected down to FEL imager, yesterday until SR absorber beyond mirrors M1/M2 and MCP
• karabo update to 2.2.x in SASE3 tunnel scheduled for next tuesday morning
• SASE3 DAQ urgently required
• issues with in-vacuum goniometer motor control of KMONO
• issues with IMGFEL to be understood - found a scintillator which works when we have FEL radiation

AE

•  The need to properly plan the activities close to the tunnels closing is pointed put.
• The need of a specific technical coordinator as in SASE3 for the SASE2 tunnels is pointed out

ITDM

•  Sase3 DAQ not ready since Karabo is not yet up-to-date
•  Slow data acquisition issue affecting cameras: being worked out.

CAS

•  In SASE3, deployment of new (2.2.1) Karabo versionwas  delayed because of many people getting the flu.
•  2.2.2 release test cycle ongoing

Photon Run Coordinator (N. G. Kujala):

• SASE1
  • Machine delivery 300 pulses upto XTD2 without M1 mirror in beam on Wednesday evening.
  • On Thursday 300 pulses deliver to FXE for radiation test.
  • Collected data using DAQ with 100, 200 and 300 pulses.
  • Large area Detector40 x 40 mm was set-up for radiation test at FXE for .
  • Damage test with 300pulses in FXE hutch with 0.5mm steel, 0.2mm copper and B4C block   
  • Vacuum at XTD2 absorber was stable and temp was also stable for 300 pulses
  • Harald setup piezo motors for M3 of SASE1.
  • Yesterday night beam delivery to SPB with single bunch.
  • ZZ for XTD9 on Tuesday by SPB for CRL test/check.
  • Vacuum on FXE branch in XTD9 was tripped on Tuesday early at 1am, which lost beam permission. Manually I have reset interlock.
• SASE3
  • SASE3 commissioning is going on.
  • Commissioning the XGM and IMG correlations with Gas ATT and without Gas ATT;
  • SoftMono mirrors is commissioning
  • Gas attenuator commissioned and Raul made look-up table for pressure setting
• 30bunches were operated in SASE3 with Gas attenuator

SPB

• Ready for first commissionig
• AGIPD power and cooling procedures OK
• Feedback problems between Karabo and AGIPD. Being worked out

SQS:

• Hardware installation starting Mid. March after  cleaning of the hutch
• FMB will install the mirror tanks in April. Has to be synchronized with the remaning work in the hutches
• Meetings with AE soon to strat planning the commissioning

FXE (P. Zalden / A. Galler):

• Ready to accept first beam
• Radiation tests done

SCS (J.T. Delitz)

• Removing of wood cover: 05.02.2018

• Cleaning: end of week 11.

• End of march beginning of the installation of the first components (ALAS and TIM)

• Waiting of the exact date for Interlock test to specify more exact the dates for the KB mirror system installation.

PP-Laser (M. Lederer):

• LAS is now commissioning 3-stage NOPAs for both FXE and SPB.
  ​so far all in plan to finish in March

Vacuuum (R. Villanueva):

• SASE1 
  • Nothing remarkable: the whole vacuum system works in a very stable way. We only had one incident on the output of 1-2 ion pumps that trigger once the interlock. For that we are investigating the causes .
• SASE 3.
  • The whole vacuum loop (as required for the first lasing) down to the instrument front-end is under control and no special issues have appeared yet.
  • The SASE3 Gas Attenuator is under manually operation and most of the Photon Commissioning Members are familiar with it. Next steps are aligned with the deployment of the Pressure Feedback Control PID control available at Karabo level and to continue with the software-related tasks.
• SASE2.
  • XTD1------->The complete vacuum string down to the front-end in the shaft building is under vacuum but in local mode operation (including XGM subsystem)
  •          ------->During this almost past week we already initiated the PLC local tests (i.e. US & DS Differential Pumping are already done), and in KW10 we aim to move on with the rest of the system (ion pumps and vacuum sector gate valves).
  •          ------->Absorber, V0 and Fast Valve wiring are in good progress and we think they could be ready for local tests by the beginning of next week.
  •          ------->The interlock tests are planned also for KW10.
  •          ------->Depending of the progresses of AE and CAS, we may even be able to deploy the first control scenes in Karabo.
  •  
  • XTD6 ------->The assembly works of the whole beamline there progress adequately and without remarkable "show-stoppers" that couldn´t be managed.
  •          ------->Vacuum Group and Optics group (N. Kohlstrunk) are having also special attention and dedication to the SDU vacuum chamber conditioning and preparation works for its upcoming in-vacuum mechanics installation.

XPD (T. Maltezopoulos ):

• XGM in XTD1 is connected to DOOCS (together with DESY)
• Synchronization of XGM@XTD1 microTCA next week
• Local tests at XTD1 XGM were finished (together with AE)
• XTD2, XTD9, and XTD10 XGMs are running 24/7
• Change to 300 bunches went without problems
• XTD9 XGM need SRG recalibration ... done meanwhile
• Wajid, Joakim, Tommaso, and Theo will repeat  SASE3 DAQ Imager/XGM correlation today (need single shot operation)

Advanced Electronics (P. Gessler):

• Broken terminal between SASE3 MOV2 PLC Loop and Master Loop were replaced last Friday without any side effects. The Train ID is now also provided from the MOV2 loop.
•  So far, no PLC related export from the SASE2 Tunnel eplans of the vacuum, beam transport and diagnostics PLC crates are possible. Therefore we will build up the XLS file for the start up of the VAC and XGM loops next week. However, the related group plus AE and EETF should meet to evaluate, if an export is possible until end of March in order to use it for further installation work. Otherwise XLS files have to be crates manually again with support from all involved groups. Also consistent documentation is required between diagnostics on one side and vacuum and beam transport eplans.

SASE2 Tunnels (C. Youngman)

• Table with the set priorities and deadlines available in Indico.

ITDM (K. Wrona):

• Run Configuration Groups required for the user experiments for FXE and SPB/SFX have been defined and tested
• Discussion after the meeting: Since the Karabo-DOOCS interface for XGM will be changed and it will affect also how the data arrive at DAQ data aggregators we need to modify and (re)test the BEAM_COND group.

Detectors (J. Sztuk-Dambietz, P. Lang):

• AGIPD will operate with the old firmware due to problems with the new one
• No calibration data available as of today
• AGIPD is in a safe state

CAS (S. Brockhauser):

• Gotthard for HIREX being operated
• Progress on GATT commissioning by Valerii

SPB

- Loops update OK

- Issue with SmarACT controllers. AE is on it (see below).

- Synch motion of mirrors motion being worked out with AE.

FXE

- Loops update OK

- DAQ updated

- CRL in the tunnel have two encoders with problems. Can be disabled and are not a problem for operation.

- A leak before the diamond window was found. Being closed now.

- Preparation for timing tool commissioning ongoing

- The new user chicane will be probably cut next week.

Laser:

- SASE 1 commissioning and upgrade moves according to plan

- SASE 2 and 3:  so far we still have green light concerning access control, laser safety installations, network and sync for installation of laser tables starting at the beginning of April.

Vacuum

- SASE1 und SASE3 vacuum systems are ready for restart of beam operation

- update of SASE1 vacuum loop on Wednesday caused many problems (all ion pumps in error state)

Optics

- Test piezo motors XTD9 today

XPD

- XTD10: K-Mono, XGM, PES, filter chamber, imagers operational except:

- Karabo tests for Exit slit imager.

- Camera cables to be exchanged for 2 imagers, ZZ access 1h.

- PES, issues with digitiser and MPOD scenes.

- XTD2 and 9: All components operational, Hirex also including Gotthard.

- Next steps

- Progress on scenes and projects implementations.

- Compatibility with new Karabo, Karabo GUI.

- Test and improve usage of DAQ.

AE

- Coordinated motion of MC2 Based Motors were implemented and presented to first users in SPB and XPO as well as to CAS

- The issue with the SmarACT motor controllers observed in the past was due to an issue in the controller. It was solved with an firmware update and requird configuration with dip switches. For this the later usage (e.g. rotary, linear, etc stages) has to be defined. The modifications for SPB is ongoing.

- AE observed, that in some cases PLC related electrical work was carried out until the very last day of open tunnel, which require a PLC firmware update and successive test. Since there is also a risk in the firmware update as it could be, that during tests issues might be observed, there is no access anymore to the actual system and thus could not be solved without ZZ or new access time. We had issues due to this, where at least temporary workarounds were possible - but it could also be different. This is a high risk, which should be know to everyone involved and decided on, if we want to continue like this in the future. AE would prefer to stop with any electrical PLC changes at least some days before tunnel closure and allow firmware updates (possible Karabo changes) and full tests by the users of the components with Karabo in order to identify and solve possible issues.

- Another issue took place, where a change request for a PLC change was issued, but instead of waiting for the finally modified designs to then do the related hardware modification, the modification on hardware were done without any new designs. This creates many problems: no CAD design according to hardware exists, the current revision state of the module is not defined, no firmware structure could be exported and generated for the current design, etc. Therefore modifications without new CAD design and consulting AE is not allowed!

ITDM

- Sase3 tunnels: cabling issues solved.

- DAQ so far stable. Has to be tested at full load now with the new Karabo.

- DAQ update for Tunnels in SASE1 and 3 ongoing. Today and next Monday will be test days.

- Gateway update done: small issue with SSH connections, easily solvable==> the procedure will be posted in the logbook

- The Maxwell cluster will need to be shut down for 2 days next week due to upgrades.

CAS

-  2.2.1 has been deployed without real problems on FXE, LAS, SPB, SA1.

-    In SASE3, it is still 2.1.18 and it will be updated early next week

-    2.2.2 release test cycle almost finished. The release will be published early next week - This release will not be mandatory, but recommended based on the new GUI benefits.

-    SASE3 Tunnel most critical tasks were done, including the gas attenuator by Valerii. It was a really complex task that Valerii went deeply and get it done for us all!

-    Sergey keeps working in Digitizer with Hamed (AE) but based on the deadline they decided to go for the old protocol.

-    Andrea (Alessandro and Steffen) has been working on the AGIPD Library to report when AGIPD will stop to acquire data

-    Gabriele started discussions to address the DAQ for LAS with ITDM.

-    XGM's DOOCS integration has been updated to address TrainID issues. We must allocate 2-3 days to test it with Beam.

General

• User run is currently going well. Thanks to everybody!
• Preparations need to be made for the October shutdown. Information will be collected by PSPO and T. Tschentscher. Care must be taken that the relevant people are available. Priority will be given to XTD10!

Short Report from SASE1 Instruments

• ​SPB: 
  • ​First user shift last night
  • No hickup from accelerator
  • Focus is still too large (12 - 15 um) and beam pointing and beam position are unstable
  • Calibrations were done with AGIPD using powder. 
  • DAQ ran stably
• FXE:
  • ​No report as FXE is currently on shift

Tunnels

• Optics:
  • ​Reminder: The last two metal valves in the FXE and SPB beam lines are currently not in the interlock. Hence the beam could hit them if no attention is paid. This has been fixed in the firmware but will only become active after the next reboot. This will not happen before October
  • Switching between instruments is done using a macro by Harald. It worked without hickups so far. Nevertheless, one should attenuate the beam before moving the mirrors
  • It is important to put relevant info into the operations log book. In particular since all the user log books are closed
• Diagnostics:
  • ​XGM data is now available both via DOOCS and Karabo and can be put into the DAQ stream. However, this data needs to be included. This has not been done so far
  • The XGM data should also be stored for the purpose of optimizing the facility. However, if stored with the user data, access will be difficult
  • Parallel storing of XGM and user data is not possilble currently

Laser

• Work is still progressing in line with laser delivery in November. However, the time is getting tight and overtime requests have been filed for the whole group
• Drifts of the laser beam are being observed that do not correlate with temperature and humidity. A feed back is being worked on
• A fix has been prepared by CAS for the motor problem reported earlier. However, due to the high work load from user operation this could not be tested yet. It is scheduled for next week
• DOOCSGATE was updated to allow adjustment of synchronisation

Detectors

• AGIPD:
  • ​DAQ ran stably 
  • Users took calibration data last night that allows to calibrate the different gain stages
  • A. Mancuso showed nice data on his laptop with very clean rings from the glas in the beam
• LPD
  • ​Not used yesterday
  • Should take data today
  • The calibration pipeline on the Maxwell Cluster is now enabled allowing calibrations to be calculated for both detectors

Electronics

• All instruments loops were updated before the user experiments for DAQ and for failsafe operation with train id injection
• Additional signals were prepared for the MPS system (Vacuum ok, end switches, pulse picker beam pattern). This is not yet integrated
• ZZ happened last Tuesday in XTD2 for CRL and MCP. On CRL significant problems with the cabling were identified (wrong connections, sparks in connectors). The sparks are probably the reason for the broken Beckhoff terminals. Fix can only be done in October
• In October the remaining tunnel loops with also be updated

Software

• 90% of personnel is currently working supporting the user operation in SASE1
• A hot fix for the data logger was deployed. 
• Instrument user shifts will be supported by 2 CAS people at least until end of next week. 
• Remaining personnel will work on the backlog, e. g. fixing the motor problems in the laser
• FXE:
  • DAQ for Gotthardt was deployed yesterday
  • Scantool was updated
  • Work continues on the GreatEyes detector. Currently it only works in Win7 and not in LINUX
• SPB
  • Doocs bridge is operational and slow data can be recorded, e. g. e-beam energy, K param
  • Users also made first use of the ZeroMQ tool 

General

• User run is currently going well. Thanks to everybody!
• Preparations need to be made for the October shutdown. Information will be collected by PSPO and T. Tschentscher. Care must be taken that the relevant people are available. Priority will be given to XTD10!

Short Report from SASE1 Instruments

• ​SPB: 
  • ​First user shift last night
  • No hickup from accelerator
  • Focus is still too large (12 - 15 um) and beam pointing and beam position are unstable
  • Calibrations were done with AGIPD using powder. 
  • DAQ ran stably
• FXE:
  • ​No report as FXE is currently on shift

Tunnels

• Optics:
  • ​Reminder: The last two metal valves in the FXE and SPB beam lines are currently not in the interlock. Hence the beam could hit them if no attention is paid. This has been fixed in the firmware but will only become active after the next reboot. This will not happen before October
  • Switching between instruments is done using a macro by Harald. It worked without hickups so far. Nevertheless, one should attenuate the beam before moving the mirrors
  • It is important to put relevant info into the operations log book. In particular since all the user log books are closed
• Diagnostics:
  • ​XGM data is now available both via DOOCS and Karabo and can be put into the DAQ stream. However, this data needs to be included. This has not been done so far
  • The XGM data should also be stored for the purpose of optimizing the facility. However, if stored with the user data, access will be difficult
  • Parallel storing of XGM and user data is not possilble currently

Laser

• Work is still progressing in line with laser delivery in November. However, the time is getting tight and overtime requests have been filed for the whole group
• Drifts of the laser beam are being observed that do not correlate with temperature and humidity. A feed back is being worked on
• A fix has been prepared by CAS for the motor problem reported earlier. However, due to the high work load from user operation this could not be tested yet. It is scheduled for next week
• DOOCSGATE was updated to allow adjustment of synchronisation

Detectors

• AGIPD:
  • ​DAQ ran stably 
  • Users took calibration data last night that allows to calibrate the different gain stages
  • A. Mancuso showed nice data on his laptop with very clean rings from the glas in the beam
• LPD
  • ​Not used yesterday
  • Should take data today
  • The calibration pipeline on the Maxwell Cluster is now enabled allowing calibrations to be calculated for both detectors

Electronics

• All instruments loops were updated before the user experiments for DAQ and for failsafe operation with train id injection
• Additional signals were prepared for the MPS system (Vacuum ok, end switches, pulse picker beam pattern). This is not yet integrated
• ZZ happened last Tuesday in XTD2 for CRL and MCP. On CRL significant problems with the cabling were identified (wrong connections, sparks in connectors). The sparks are probably the reason for the broken Beckhoff terminals. Fix can only be done in October
• In October the remaining tunnel loops with also be updated

Software

• 90% of personnel is currently working supporting the user operation in SASE1
• A hot fix for the data logger was deployed. 
• Instrument user shifts will be supported by 2 CAS people at least until end of next week. 
• Remaining personnel will work on the backlog, e. g. fixing the motor problems in the laser
• FXE:
  • DAQ for Gotthardt was deployed yesterday
  • Scantool was updated
  • Work continues on the GreatEyes detector. Currently it only works in Win7 and not in LINUX
• SPB
  • Doocs bridge is operational and slow data can be recorded, e. g. e-beam energy, K param
  • Users also made first use of the ZeroMQ tool 

General

• First users have done experiments - despite the current problems it appears as if everybody was able to take some data for future publications.

Short Report from SASE1 Instruments

• ​SPB (A. Mancuso): 
  • ​Beam was lost due to incident with electron beam hitting & damaging components downstream of the northern dump.
  • Currently only single bunch operation
  • Lots of work to be done in Oct. shutdown, Adrian will provide a list to Thomas Tschentscher
• FXE (T. Tschentscher):
  • ​Problem with the Tangerine-laser affects PP-experiment ==> to be fixed on Tuesday
  • Consider to switch to oscillator only until then ==> lower repetition rate, lower OL pulse energy
  • Next experiments require Gotthard & LPD in parallel operation ==> merge Gotthard & LPD to same run control next week before user run.
  • Today, test with DAQ and Gotthard was successful, but Gotthard has never seen XFEL X-rays so far. ==> Test with beam could happen potentially Monday night (instead of user experiment)

Tunnels (M. Dommach, J. Grünert):

• SASE1:
  • Few issues with XGM, have been or are being solved.
  • It was suggested to put the Hirex-grating into the beam while FXE is taking beam to measure / commission parasitically. But it appears next week neither Naresh nor Jan will be available.
• SASE3:
  • ​Due to the electron beam incident the first vacuum sections have been vented, there is most probably a hole in the V0, but only Rad. Safety to access the tunnels right now. ZZ/Access by anyone else only after clearance through SRP - working there is probably only possible in Octorb
  • Beam did hit the beam pipe apparently just upstream of the transmissive imager. Imager and further (vacuum) electronics seems to have been damaged in the upstream part of XTD10.
  • Due to this incident, it should be reconsidered if dosimeters are required in XTD10, too. SRP did unfortunately not participate, and thus was not able to comment.

Laser

• Working still on improving the laser
• Issue regarding motors coming back to their position is most probably a firmware/options issue

Detectors

• AGIPD:
  • ​According to Yolanta "operates better than expected"
• LPD
  • ​Also operates better than expected
  • Configuration of LPD is getting messed up by people working in parallel / unclear status of the detectors Gotthard & LPD ==> There should be a responsible for the detector, who is present in the control room and can allow remote access to work with the detectors / run configuration. ==> These steps should be well-documented in the FXE-Logbook.
  • K. Wrona: Next week will be better with just one run controller.

Electronics (P. Gessler)

• There is an issue with some motors & encoder @ SPB.
• AE will finish the work on the last loops in SASE1.
• WP79 needed & got support by AE.

ITDM (K. Wrona)

• DAQ for the tunnels is setup. ==> XGM-run control for the tunnel is to be tried.
• Slow data: It is an issue how and how fast to store slow data (lots of properties which do not change).

Software

• FXE:
  • Still daily meetings, prepare LPD & Gotthard
  • Working on GreatEyes, too
• SPB
  • Configure slow-data cameras
  • Laser project is running in CAS
• SASE3:
  • C. Youngman is the CAS-responsible, should collect requirements to compile master equipment list for SASE3 ==> to be compared against MEL SASE1 to identify additional work required.
• SASE1:
  • Loops & Karabo will be updated together.

October Shutdown:

• SASE1
  • No major items here currently, focus is on SASE3 / XTD10.
• SASE3
  • First priority is to get all items required for first lasing up & running ==> Vacuum system including anything up to the absorber.
  • CAS is starting to set up Karabo and all controls.
  • ITDM will also start to work, but does not require access to the tunnel.
  • C. Youngman to generate MEL from input from the groups ==> invites to a meeting if necessary.
  • Eplan for PES will not be available on time ==> has to be pumped in a provisional fashion.

General

• A recent survey has shown that the experiment hall and the end of the tunnels XTD9 and XTDD10 have settled by as much as 6.5mm. If components need to be re-aligned a completely new network of points would need to be established. This is not foreseen before the end of the year. A plot is attached to these minutes

Instruments

• SPB: 
  • Recabling of many components is currently going on
  • AGIPD has been opened and some electronics are being exchanged
  • Preparations are being made to move AGIPD also to a more downstream position
  • Laser table for bringing the PP laser to the sample has been set up
• FXE:
  • Cables in the hutch have been re-routed and cleaned up
  • Preparations for the PP laser are well advanced as the Tangerine laser was already brought to the sample

Laser:

• Work is going on under pressure as the time available until first laser beam is short
• 3 Days were lost due to unannounced shut downs of the AC systems. 
• Both the SPB and the FXE NOPAs are running
• There are problems with triggering the ADC and hence the timing drift cannot be locked. The problem was reproduced in the lab and fixed. But the fix needs to be transferred to the laser hutch
• Tests with the machine timing are still outstanding and it seems that there are some resource issues on the DESY side
• Currently the optical sync system from DESY provides too little power and hence the laser cannot be synchronized by optical X-correlation. For the upcoming run period sync to the RF system will be used. This means that up to ps drifts are to be expected
• The laser is very sensitive to changes in the humidity. This has been traced to the construction of the laser tables which involves wooden side panels. The problem has been discussed with Newport and the laser tables for SASE2/3 will be modified to avoid this

Tunnels:

• Vacuum XTD10/SASE3:
  • There are currently two leaks in welded beam pipes. The welders are coming on Monday to fix this
  • After the leaks are fixed the MCP can be connected to the vacuum
  • And then finally this entire vacuum section can be closed
  • There are several leaks in the dump area. In particular the V0 valve is leaking and one flange. DESY will exchange V0 next week.
  • The pipe that hit in the vacuum incident in September appears not to be leaking
  • The absorber appears to have been damaged in the September incident and needs to be replaced. The SASE2 absorber could be used
  • Vacuum tests were done for the differential pumping downstream of the gas attenuator
  • Work is still being done one the differential pumping of the Diagnostics group
  • Interlock needs to be set up and tested
  • M. Petrich defined the interlock conditions and sent them to AE. This still needs  to be implemented
  • Cabling was done for the gas attenuator
• Diagnostics
  • Inhibits for XGMs in in XTD2/9/10 was implemented
  • Loops were successfully rebooted in XTD9
  • A lot of work done on HIREX (camera exchanged, Gotthard connected to Beckhoff and tested)
  • Today repairs will be done on the MCP in XTD2
  • For both MCPs in SASE1/3 additional Beckhoff terminals will be installed
  • Next week focus will be to finish in XTD10 and get plc control of installed devices
  • All 10 imagers in SASE3 were tested. 3 needs fixes/repairs

AE

• In order to close the loops in SASE3, focus should be given to the most relevant components
• Component lists were circulated. Please check!
• Work on KMONO is complete
• What should happen to the CRL of FXE in XTD2? Statement from A. Galler: Nothing for now as it can be operated for the upcoming user run
• Work on the instruments is ca 50% done
• In SASE3 most local test are done
• It will be tried to get the Vacuum and Motion 1 loop going next week
• The EPS loop is not so critical

ITDM

• The announced GPFS update was postponed several times but will now happen next Monday
• The DAQ for AGIPD was upgraded (The SW used so far was dating back to May)
• The time allocated was 3h. The upgrade was done after 1h, so the remaining 2h were used for tests.
• Longer runs could be taken without major problems
• People should be aware that during the first three days next week, the detectors cannot be operated due to the GPFS upgrade

Software

• Planning is being aligned between CAS (H. Sanchez) and PSPO (M. Areste)
• Progress was made for the laser. The software however will remain at version 2.13 since they do not need DAQ and the scan tool
• Yesterday version 2.18 was released. It is already deployed for SASE1
• Note that this version is not complatible with previous GUI clients
• Next week: Work with AE for SASE3
• There is no news on the scantool as S. Brockhauser was not present
• FXE should be updated to 2.18 today
• The DOOCS/KARABO interface is not considered a priority on the DESY side. They are working on the undulator controls

General

• SASE1 Instruments are asked to comment on progress with KARABO 2.1.18
• Tunnels XTD9/10 will be closed on 25 Oct at 1700

Instruments

• FXE (A. Galler):
  • KARABO 2.1.18 appears to be working ok. The device instantiation problem reported earlier in the week is no longer there. However, not all equipment could be tested yet
  • Work currently focusses on preparing for the next User Run and in particular on preparing the sample environment
• SPB (A. Mancuso)
  • A situation similar to the on in FXE is observed. Full proof of functionality of KARABO will only be possible once AGIPD is fully assembled again which is next week at the earliest
  • Currently, work is going on inserting the AGIPD cables into drag chains so that the detector can be moved on the CSS. This work should finish today

Tunnels (J. Eidam, R. Villanueva, T. Maltezopoulos)

• Leaky vacuum pipes were successfully welded
• Absorber section is closed and under vacuum
• MCP detector is connected
• DESY finished work on the upstream section in the dump
• The gas attenuator has very good vacuum (10^-9)
• Differential pumping is ready from the HW side
• Cabling for the gas attenuator section is still going on
• Alignment was done in view of the small apertures in the gas section
• Local tests can start now (This is 3 days behind the schedule made at the beginning of the week)
• PES is installed but will only act as a pipe.

Laser (M Lederer)

• NOPA Beam is now going to FXE
• Next week it will be brought to SPB
• The KARABO version used is several versions back with respect to the one in the instruments and there are still a few minor problems
• On Monday work on programming the laser burst patterns together with the respective people at DESY shall start
• Continue testing
• There appears to be a shortage of conversion crystals for the instruments. New ones have been ordered but are not there yet.

Detectors (P. Lang)

• AGIPD HW work is on track and finishes thie week. Software work starts next week
• LPD control system works ok with KARABO 2.1.18. 
• Some small issues with LPD calibrations

ITDM (K. Wrona)

• Two updates were done for the DAQ of LPD
• Some issues were found with the RC
• It was tried to install a new version. This was already done for AGIPD

AE (T. Freyermuth, B. Baranasic, J. Tolkiehn)

• SASE1 Tunnel:
  • Vacuum loop was updated without problems
  • The motion loop was updated a few times because there were problems with some motors. The work continues today
  • EPS loop was updated
  • On Monday/Tuesday the SPB loop will be updated to account for the terminal changes from 24V -> 48V
• SASE1 Instruments:
  • Loop 5 with the redundancy module is now ok
  • Loops 2, 4, 7 were updated
  • Next come loops 1 and 12
  • All modified modules (24V -> 48V) are installed but a few more might need to be changed
• SASE3 Tunnel:
  • Tried to get the vacuum loop up
  • But there are still many mistakes and inconsistencies with names
  • A big amount of work is still needed for the gas attenuator

CAS (S. Brockhauser)

• KARABO 2.1.18 was deployed
• Good progress was made with the Gotthard and GreatEyes Detectors
• For GreatEyes a specific driver on a specific computer will be used
• The problem with extra delays in the GUI client was resolved
• Scantool:
  • A plan of what to implement exists. This needs to be checked with the instruments
  • A first part of this tool is actually already implemented
  • A. Galler points out that he sees the problems predominantly in the integration with the rest of the environment
• DOOCS2KARABO
  • An example for pipelined data was sent to DESY
  • There is no showstopper seen, but the work is progressing according to priorities which are 1 Laser, 2 Undulators, 3 DOOCS2KARABO
  • Diagnostics points out that the will need both the scantool and DOOCS2KARABO for their commissioning period from 1 - 7 Nov. CAS should please provide information so that the tools can be tried

FXE:

• Pump-probe beam is now in the instrument hutch
• Currently working on setting up the remainder of beam steering and compression
• Gotthardt is working. Tests are going on a and some issues with the DAQ still need to be solved
• The sample stack needs to be moved up stream for the next user period

SPB

• Still working on AGIPD. Many cables needed to be disconnected and are being re-connected now
• AE finished all the scheduled modifications on the crates and are doing some more that were originally not scheduled
• A few details with the interlock need fixing
• The newest version of Beckhoff_Devices needs to be deployed
• The previously observed slowness of the GUI is probably fixed but needs more real life exposure. Final test are being done
• The full infrastructure should be back and operational Tuesday/Wednesday

Tunnels

• Work is going on on the gas attenuator
• Interlocks are ready for the WP74 DPS and the XGM
• Detailed steps to completion in XTD10 were re-iterated by C. Youngman. Additional coordination off line after the meeting
• L. Samoylova mentioned that the mirror encoders in XTD2 are currently in error. B. Baranasic will have a look

AE

• In the new version of KARABO, the rights for reading and writing parameters from and to the PLC are now being observed, differently from before.
• This causes problems. P. Gessler mentions that he is very concerned that such a major change is introduced so briefly before the next user period. This puts AE under great pressure and leaves almost no time for testing
• All the SASE1 loops were updated because of the new "feature" in KARABO
• A. Silenzi mentions that he can implement a workaround for this "feature" so that loops need no updating. However, this requires a new KARABO release

CAS

• KARABO2DOOCS: Meeting today with the DESY control people at 1430 in BKR
• ScanTool: Meeting foreseen Tuesday/Wednesday with the instruments
• The GUI slowness necessiated a hotfix which was verified within CAS. Now waiting for feedback from instruments

General

• After discussion between XFEL Vacuum (J. Eidam) and DESY XFEL Run Coor (Matthias Scholz) it is agreed that XTD10 will only be closed on Thursday and that XFEL will search the tunnel between 15:00 and 17:00. (This information became available only after the meeting)
• This gives some extra time to finish activities in XTD10

Instruments

• FXE
  • Currently A. Parenti and FXE are hunting a spurious data loss in Gotthards
  • Otherwise the software is surprisingly stable
• SPB
  • Slow progress towards getting the Detector going again
  • The task to ease the acquistion needs help from the DESY developers

Tunnels

• XTD10:
  • A lot of progress was made by the team working in XTD10: Thank you very much!
  • The complete tunnel is under vacuum
  • Also the missing pieces between XGM/MCP and the rest were inserted 
  • Cleanup is done
  • Denis and Raul are still working on local tests of the gas attenuator system
  • Testing is somewhat slowed down because the GUI client no longer allows active manipulation of components at the moment. So the ikarabo interface is used
  • Work also continues on getting the mov1 loop going
  • Local tests are done on all imgers except on the exit slit imager
• XTD9
  • ​The FXE Mono is not finished. It needs a restart of the motion loop
  • AE will check when this is possible, but there are a lot of constraints from users and from resources
  • Since there has been a change in the interlock system, SPB needs to be in place when the shutter to XTD9 is opened. There is a risk of triggering the interlock and shutting down the machine.

AE

• Almost everything that was planned could be achieved in XTD10
• Some of the remaining work can be finished when the tunnel is closed.
  • Work is going on to finish the controls
• Even the interface to MPS that was originally not foreseen is now available

CAS

• The workaround that allows to modify parameters even though the protection does not allow this, is now in place. This should help with the trouble observed by the Vacuum Group.
• A new bug in the PLC was observed in this context. Changing a parameter whose name has changed leads to a crash. 

ITDM

• Currently the MAXWELL cluster is unavailable.  This is a serious software/hardware issued. The manufacturer is actually on site to fix the issue today

Instruments

• SPB:
  • When PLC loops are restarted, still many device parameters are not properly recovered. This generates a lot of work for the team. Recently an entire day was lost.
    • Answer from AE: The parameters are extracted by CAS using KARABO. It is possible that not all necessary values are returned. AE is also investigating whether there are some bugs in the firmware procedure. It is pointed out that the change from 24V to 48V led to a huge amount of changes. 
  • Injector motion is now working along all three axes
  • AGIPD automatic power up procedure is now working. Tests continue
• FXE:
  • The spurious data losses with the Gotthard Detector were further investigated. They are not resolved but they do not limit operation with 30 bunches

Tunnels

• XTD10/SASE3
  • All vacuum sections are connected, locally tested and ready
  • Part of the interlock system is also ok. A global test of the entire integrated system still needs to be performed
  • The biggest challenge was the gas attenuator section with 7 turbo pumps. All pumps were locally test and integrated into the control system. The vacuum is very good (10-9) Now they are put into a  safe state.
  • R. Villanueva would like to be informed in case whenever the interlock in XTD10 is broken so he can inspect the system
  • XGM is connected to the overall vacuum and tested. The interlock is working and it is also connected to DOOCS
  • The two imagers requested for first lasing (SR/FEL) are available
  • Approximately one shift would be needed to take care of a few remaining tasks before first lasing
• XTD9/SASE1
  • The HiRex is fully operational
  • The Si-Mono needs work but this was de-prioritized as it is not needed for the next users

Laser

• Full specs were reached for both the SPB and FXE beams. The laser is in the respective instrument laser hutches
• The laser was covered in order to reduce effects of any remaining air drafts
• Differential mode compensation is available. Common mode compensation is not
• Work still continues on the 4 agree-upon bunch patterns. Another meeting is set up for today
• Delay control is available through KARABO but requires ca 2 days testing

Detectors

• For AGIPD both  for turning on power and cooling automatic procedures were implemented and tested
• LPD was moved to Karabo 2.18. 
• One charge injection run was done and the system operated for 2 days without interruts

Electronics

• Currently the work that is still needed for SASE3 is being evaluated
• Planning for the next shut down is going on

CAS

• Currently planning next OCD period
• Very good progress was made in SASE3
• Working on DOOCS/KARABO integration together with DESY
• Work on the laser is progressing
• A. Silenzi and S. Hauff worked on the AGIPD procedures
• Work continues on reading and writing data for the GreatEyes detector
• As reported by the detector group, the stability of the DAQ is much improved

ITDM

• A major problem caused the Maxwell cluster to be down for 3 days. IBM Technicians had to be called
• In this context it became clear that the communication with the users must be optimized
• Now the system is ok
• The folders for the next user run are being set up next week. This was delayed due to the technical problems
• For the experiment log books the users will now be automatically set up
• The update of the DAQ for AGIPD is somewhat delayed due to the long time taken for the recabling
• SPB requested 18 new cameras. This effort needs to be carefully coordinated.

General

• The rack layouts and rack placement were finalized during the PRR in August. The result is the basis of the network planning. The network is currently being installed and will most likely be finished by the end of the year. Changes to the layout that affect the network planning are no longer possible. 
• The TC/XHEXP1 meetings shall be combined into one bi-weekly meeting starting on 9 Nov. A separater new invitation shall be sent

Instruments

• SPB
  • Currently preparing for the next user period
  • A lot of work still needs to be done for bringing the laser to the interaction point. This mostly concerns the beam pipe
  • Apparently there was an interlock problem related to SPB components in XTD9. SPB components were taken out of the EPS loop without SPB being informed
    • Comment from T. Freyermuth: The condition of SPB devices was preventing beam operation and noone of SPB could be reached
• FXE
  • A lot of progess was made together with CAS
  • This afternoon a DAQ stress test with several detectors is planned
  • The encoders on the CRL lens stack were disabled as this complicates the operation at the moment

Tunnels

• XTD9 was closed yesterday. Beam could arrive this weekend
• About one shift worth of work remains in XTD10 before readiness for lasing
• KMONO measurements took place on Wed/Thurs from 3PM - 3AM
• Some DAQ and KARABO issues surfaced
• But it was possible to record donut radiation patterns
• Now commissioning of HIREX is being prepared
• XGM requires some minor re-commissioning

Laser

• Beam is in the ILHs. 
• Still problems dealing with the capricious AC system
• The promised specs of the laser beam are met
• Synchronization is still to local timing. Synchronization to the machine timing only comes next week and no time remains for testing before users

Detectors

• LPD:
  • 15/16 super modules are operational
  • This week calibration data from charge injection were taken
  • The detector is ready for users
• AGIPD:
  • Detector is fully working again after a big rearrangement of cables
  • It was found the the clocking-control cable is too long and needs to be re-routed during the winter shutdown. The detector will be fine for the next users
  • Work is ongoing on the firmware in order to allow 30 frames/train rather than only 15

Electronics

• SPB:
  • Updated the tunnel loop for SPB
  • Loop 12 is up-to-date
  • Loop4: The mirror mounts were tested locally
  • The is no feed back on interlock behaviour in loop 5. If there should be problems individual conditions can be disabled
  • For loop 7 zoom and focus and mirror mounts to be tested
  • Same applies to loop 2
• XTD9:
  • want to update EPS loop to include removed SPB components today
  • Vacuum loop to be updated next week
• XTD10
  • Some ion pumps show an error condition in KARABO. J. Tolkiehn is investigating

ITDM

• User handling for log books has changed. Please consult the help log book for details or consult J. Szuba
• The meta data catalogue was updated so that instruments can now annotate with their own credentials
• Folders for the next two user proposals are created, also in the online environment
• A new version of the DAQ software needs to be deployed in order to fix a bug in slow data daq

CAS

• A number of issues need to be resolved with Diagnostics as already mentioned earlier
• The Gotthardt Detector is now working in FXE
• The promised reduced scope for the scantool has been achieved. It will be made available to the users asap, most likely today

General

• User operation is delayed by 48h due to rocky startup of FEL.
• FEL still has not reached acceptable levels of stability and power

Instruments

• SPB:
  • Instrument was brought back to a somewhat running state
  • There are still problems with motors in the sample chamber
  • Alignment with beam is ongoing at this time
• FXE
  • New scan tool was tested
  • Currently working on user-specific alignment tasks
  • Laser was routed to the sample position
  • The beam is still drifting around

Tunnels

• FEL had trouble to find lasing
• There was a controled access on Wednesday to fix various problems. However, further problems appeared
• It was decided to roll back the changes made for a new injector laser, but this did not improve the situation
• At some point 400 uJ were reached and lost again
• There are many problems with the undulator system
  • Incorrect values in lookup tables
  • Difficult operation of the entire system
• FXE should be able to start up tonight
• SPB would follow Saturday morning since time is needed for alignment during the day today
• Shifts for photon diagnostics were largely cancelled due to FEL problems
• KMONO gave some interesting results pointing to the problems in the undulator system
• Next week FEL Imager and KMONO will be installed in SASE2

Laser

• Both SASE1 instruments have laser beam with local timing. 
• Tests with machine timing start today at lunchtime

Detectors

• AGIPD is in best possible state. 
• Firmware and DAQ were upgraded
• Some commissioning time with beam is needed today
• LPD had a cooling leak  and a power problem last night
• At 6 pm yesterday, this was still not fixed

AE

• Continuing small activities for user operation

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on this Monday

CAS

• Dedicated squads for the setups of SPB/FXE, DOOCS integration, Scan Tool closed now
• Tools delivered but with significantly reduced scope
• All necessary things for DOOCS Karabo integration were delivered. Now waiting for DESY

General

• Beam is back with good intensity (up to 1 mJ) 
• Next user period starts on  23 Nov
• Time between now and then should be used to optimize FEL, beam lines and Instruments

Instruments

• FXE:
  • Measurements of pulse intensity were done using an XGM from DESY in the hutch
  • A  pulse intensity of ca 40 uJ was found. This is in rough agreement with the expectations
  • Spectrometer was aligned
  • GreatEyes and Gotthardt detectors are working fine
  • LPD still to be checked
• SPB:
  • Using beam to do alignment
  • Not such a good focus (too big)
  • Plans are to continue with instrument commissioning today and Saturday
  • Work is currently being done by Mahadi on the SmarACT rotary stages
  • FXE points out that most likely SPB operated with the FXE CRLs inserted into the beam
  • Note: The standard "switching script" from Harald should be used to make sure parameters and devices are properly adjusted when beam is switched from one instrument to the other
  • Detector calibration on Sunday

Tunnels

• Last Friday there was a Bender failure on mirror M2
• The mirror was removed from the tunnel during the weekend and the SASE2 spare was inserted 
• In the process one Beckhoff controller was found broken
• In the removal process a further problem was caused in one of the Beckhoff loops
• This problem could be fixed from outside the tunnel 
• After this 20h were spent to redo Beam-Based Alignment in 7 iterations
• The new orbit deviates from the old by up to 80 um.
• After using the new orbit, the FEL lased quickly with good intensity
• The replaced mirror M2 could be re-aligned 
• On Thursday beam was available to FXE
• Remark from J. Grünert: Instruments should make sure that most up-to-date DOOCS panel is loaded to see proper XGM info
• Update of DAQ caused DAQ in tunnels to be broken. This is now fixed according to K. Wrona
• HIREX commissiong is going in parallel 

Detectors

• AGIPD:
  • Before the bender failure a few X-Rays could be taken on AGIPD
  • It looks like 30 X-Ray pulse can be taken now
  • Additional flat field calibration is needed now
  • On Sunday a full shift is foreseen for commissioning with beam
• LPD:
  • One quadrant was disassembled to find the cooling leak.
  • Leak now appears to be fixed
  • No sign of dead spots on detector

Laser

• We are operating in shifts:  early / late, although experiments have been cancelled, performing shift change and other excercises for commissioning.
• Operating the laser with machine timing was so far hampered by the fact that the transmitted laser patterns from the machine were limited to the actual number of x-ray pulses, i. e. mostly non-deterministic 1 or 0. Optical alignment and characterization is not possible under those conditions. Therefore, on the weekend we have switched to local timing to give guys some power.
• Following this experience, we have decided to attempt local generation of laser patterns under machine timing in order to be independent of the machine and not have to switch back and forth between local and machine timing. This has been implemented and is under test today and over night. We still observe spurious long term drifts, which are under investigation.  However, if finally successful, this can only be a make shift solution for the coming weeks. Mid to long term we need proper implementation by DESY, allowing pattern changes on short notice and independent of the machine. This is in preparation (time line ca. mid 2018).
• Currently, beams are sent to FXE and SPB with 30 pulses each, 800nm broadband, should be compressible to 15fs. This is under machine timing but with locally generated patterns.
• There are other curiosities concerning machine timing, which are related to the increased T0 arrival time jitter compared with our local timing. Debugging is ongoing. We are also testing various features under above conditions (machine timing, local pattern):  phase shifter, trigger shifter, shift change. Time line: depending on outcome, daily updates will be provided.

AE

• When the breakdown of the interlock occured in conjunction with the broken Beckhoff terminal it became obvious that crucial emergency phone numbers are not available
• Better emergency procedures are needed
• The special EPS operation mentioned to align the mirrors should be written down so that it can be implemented an no override of the interlock system is needed
• The list of shutdown activities compiled by PSPO contains a lot of work for AE. This is now being discussed. 

ITDM

• Some problems occured this week with DAQ of slow data. This should have been fixed yesterday

CAS

• Doing 24/7 OCD support 
• During switch between instrument CAS comes in to provide support
• The cancellation of user operation last weekend was not properly communicated so that people came in unnecessarily
• Next week additional work for WP74 is being done

General

• User operation ongoing in SPB/SFX and FXE

Instruments

• FXE:
  • User operation started during the night
  • Some concerns over running LPD as the sample chamber is from diamond and direct reflections could cause damage
  • Generally instrument worked very well
  • Synchronization with PP laser is now in a state where it can be tested
• SPB
  • Many data taken for detector calibration
  • Severe issues with KARABO
    • Response is very slow
    • Major crash this morning
    • Detector controls complex and instable
  • A failure of AGIPD related to FPGA not understood
  • Strange interlock issue: EPS interlocks appeared disabled

Tunnels

• L. Samoylova spent many hours on Wednesday to set up the DAQ in the tunnel
• With the help of many support people she managed to get the system to work
• The system is instable and also seems to record many more parameters than are needed
• Use of the system is very clumsy
• Comment from ITDM (L. Maia): The data sources were specified by the users. It is acknowledged that the system needs a lot more work
• Diagnostics is working with CAS in a squad to work down a list of open redmine tickets
• The DOOCS - KARABO bridge is now working. The FEL imager is routinely available in DOOCS to the BKR crew
• DESY set up a statistics server to help collect data about beam pointing instabilities

Laser

• Local pattern generation synchronized to machine timing is working
• Overlap finding together with FXE was done but has to be tested under real condition next week
• A few suddent shutoffs of the front end occured. The origin might be understood
• There are large timing drifts in the ADCs that are currently not understood (up to ms)

ITDM

• Next Wednesday there will be a 1h shutdown of the load balancer in the DESY RZ. This should affect mostly the web services
• It should be noted that the DAQ ran quite stably in SPB and FXE. More than 20 runs were collected in each case without a restart

CAS

• Two hot fixes are out (2.1.18.4/5). These address  very specific issues
• A major new release (2.2) is planned for the winter shutdown. This will be mandatory
• Comment from instrument: It must happen early so that wrinkles can be ironed out!)

General:

• User Operation ongoing until Tuesday 5 Dec 8 AM
• CAS shall present a release plan for KARABO 2.2 next week
• Dispatch meetings shall move to a Mo/Wed 13:00 schedule starting next Wedneday 6 Dec

Instruments

• FXE:
  • Scantool was successfully used to establish overlap with laser beam (Tangerine) 
  • Some problems with slow DAQ in particular also for tunnel components
  • Comment from ITDM: There are known issues with the slow data acquisiton. A new version exists that was intentionally not deployed before the end of UOP
• SPB
  • Just finished first shift of laser user experiment
  • No big problems
  • All goals in the optimistic schedule were achieved

Tunnels

• Currently preparing for the shutdown
• Sticking condition of mirror bender could be reproduced in the lab
• WP74 just installed the last missing pieces in XTD1

Laser

• Running
• temporal and spatial overlap with beam is achieved
• But now instruments do not intend to use the PP laser for the last experiment

Detectors

• Both detectors are working
• The DAQ for slow data is crashing occasionally
• Gotthard is fully operational but there problems getting the data out. Help from PSI is needed

AE

• No problems

ITDM

• No problems

CAS

• CAS-XPD Running:
  • We will finish next Monday
  • We have been addressing issues related to MCP, Karabo-DAQ, Cameras, and K-Mono
  • Deploy K-mono today
  • Review DAQ Groups in Karabo: Cyril (interface to ITDM), Liubov, Koch
  • Gotthard we have to address in a separate and more systematic discussion: trigger, operation, calibration...
• XGM memleak - we already have a fix to get it stable but have to check when we will be able to upgrade (only device).
• Internal discussion about CAS/Karabo priorities for the winter shutdown

General

• Starting from next week shutdown activities will be tracked using M. Aresté's list
• Dispatch meeting on Monday and Wednesday at 13:00
• Next Friday Readiness meeting also at 13:00 because of conflict with XDAC

Instruments

• FXE:
  • Currently focussing of implementing all default parameters into the next firmware release
• SPB:
  • Major detector work with AGIPD disassembly started

Tunnels

• Currently testing firmware and software in SASE3
• Installation of first mirror starts next week. Mirror installation takes about 1 wk/mirror
• CRL in XTD2 will be vented to align internal mask and fix stuck arm
• Drilling takes place at the end of XTD9 for the installation of the FXE diamond grating
• Mirror cooling will not be installed this shutdown. It is not needed for the planned bunch rates and might introduce unwanted effects/vibrations
• Activities of Diagnostics:
  • Photonics Science camera in SASE1 exchange.
  • Tests are being carried out in SASE3 together with J. Tolkiehn
  • Cabling is going on in XTD1 on the XGM
  • There is a noise problem on the Gotthardt detector in the HIREX. Need support from AE to find the origin

AE

• Motion loop in SASE3 will be split into mov1 and mov2. This will allow to sort components into more and less stable sets and keep one of the loops up most of the time

Laser

• Laser activities had to start late due to problems with AC and laser interlock (delays of 6 and 5 months respectively)
• Availability of trigger pattern from the machine was also a problem
• Specs could just be reached for the Nov user period
• Eventually the PP laser was not used for any of the experiments
• Work on the laser is far from finished and will require 3 more months to come to full funtionality
• Currently, the Karabo version is being upgraded and the most up-to-date PLC firmware is being installed
• These updates had intentionally been delayed until the end of the user period.

Detector

• Detector group not present at the meeting
• Information from SPB: major work is ongoing on AGIPD

AE

• Train builder is being tested as it was not used so far. This required re-cabling
• PLC motor loops are also being split in SASE1
• Many activities in SPB
  • Remaining modifications of modules have started
  • Export of input for firmware generation for loop 8 has been provided by EETF
• Waiting for go in FXE
  • Need to work on the interface for the EPS system
• No tasks currently from laser group
• New information available on SmarACT stages: The observed problems arise from a cable problem and can be solve by configuration
• The problems with movers in the sample chamber in SPB (non-functioning encoders) arise from cable cross talk at the vacuum feed through. This require hardware re-design
• In SASE2 there are tests coming up together with the diagnostics group

ITDM

• Train builder tests are scheduled next week
• Recabling of Infiniband network happens on 18/19/20 Dec. No DAQ and controls available on these days
• A summary of DAQ statistics was shown (see slides on INDICO)

CAS

• Release of KARABO 2.2 on 18 Dec
• Freeze of functionality on 21 Jan
• Discussion reveals that a discussion on specific deployment dates and needs of different groups is urgently needed (Instruments, Tunnels, DAQ)

General

• Meeting on Friday will take place at 1300 due to overlap with XDAC
• M. Areste shall ask for updates on task list on Thursday

Instruments

• SPB
  • Work on AGIPD is ahead of schedule. Closing of detector should happen at the beginning of next week
  • Additional installations for roughing pump system is going on and will probably finish this year

Tunnels

• Diagnostics group currently working in the Southern tunnels, cabling components and aligning imagers
• Colleagues from Russia are coming 3 - 20 Jan to work on MCP in SASE3
• KMono local tests in SASE3 should also happen in early Jan
• Diagnostics requests scenes for XGM in XTD1

AE

• Vacuum loop in SASE3 will be modified. Some components shall be moved to a separate loop, e. g. Soft Mono
• SPB tunnel loop will be re-deployed to include the attenuator
• The motion loop in SASE1 will also be split to allow more rapid updates for newly integrated components
• Z-motion crate for detector in SPB will be done until next week or early next year

CAS

• Currently working with AE and ITDM on refining release schedule for KARABO
• Testing the version for release next week. This will then first be used by ITDM and is needed for the firmware upgrade of AGIPD
• Work is going on on the XGM train ID problem
• FXE will be moved to a separate topic (broker) to disentangle some dependencies at least temporarily
• FEL imager performance problem was investigated. This runs on a server that is very busy and should be moved to a different server