FXE

• Still working on getting the laser on the sample
• Commissioning the recently received intensity monitor
• Work is also ongoing on the Gotthardt detector
• LPD is still in its own "Topic" but will be moved to the SASE1 "Topic" today

SPB

• Problem developed with one ELMO motor. It was moved to a test crate where it worked
• All loops were updated to allow slow data integration into DAQ
• AGIPD running improved significantly in stability after implementing the latest nightly build of KARABO
• ITDM team has various testing tools operational on the different channels of the detector
• Charge injection scan was possible for the first time without significant degradation of KARABO response

ITDM

• Slow data is now integrated for SPB
• Request is made for the list of data sources to integrate with FXE

AE

• PG points out that it was always foreseen that the timestamp/train ID integration would be done in software
• A HW solution was implemented during his absence to generate train IDs in a  PLC. He considers this to add significant risks.
• The HW solution will not be implemented in the tunnels to avoid unexpected behaviour during the user run
• Hence a work around solution needs to be found how to uniquely identify slow tunnel data with train IDs

Discussion about time stamps and train ID

• The summary of a longer discussion about the subject leads to the following conclusions:
• Slow tunnel data could be logged via a python script that also stores time stamps and train ids. An integration of this data with the fast data would have to be done offline post mortem
• Different opinions exist whether the association between time stamps and train ids in the KARABO data logger is unique and compatible with the train ids used in DAQ
• It is decided that this will be checked now and the result reported during the Friday S1R meeting

Announcement by ITDM

• The elog will be upgraded this afternoon
• The online cluster servers will be rebooted this afternoon

General

• User operation starts on Thursday 14 September 08:00 in FXE
• Instruments alternate every 12h switching at 0800 and 2000
• A common SASE1 operations log book has been set up here. It is accessible for all inside XFEL
• The PIs of the first 6 user groups have been informed that the scope of their experiments will be rather "user-assisted commissioning". In compensation they will get more beam time in March 2018 if the objectives of their proposals could not be reached

SPB

• An effort was made with ACC to achieve good conditions for focussing. So far this has not been successful
• At 2200 on thursday these efforts were abandoned as there were no more ideas to try
• Also the power level in the beam was rather low
• Negative effects of running the detector on KARABO responsiveness are much reduced now
• This night KARABO froze up completely (no detector running) and no support from CAS was available (No response from calling the OCD mobile)

FXE

• Cross calibration effort for the beam energy was performed with HIREX. This succeeded after some efforts.
• The absolute energy scale could be established to within 70 eV. The fluctuations (averaged over several seconds) are around 20 eV
• Effort to synchronize the tangerine laser beam and to bring it to the sample continues. The quality of the sync signal is not very good. There are big signal losses in the fiber

Laser

• Still two problems with KARABO: Cannot reliably read date from wave guides and the python to doocs interfaces does not implement essential data types
• Currently working on aligning the NOPA
• The fact that the synch signal will not be available until end Sept puts a strain on the planned dates for beam delivery
• The full SYNC hutch will not be available until end Dec

Detectors

• The new GUI version gives good response while detector is running
• SPB was instructed on how to use the offline calibration
• LPD was brought into  the SASE1 broker topic
• Coolant leak in LPD was fixed but 1 super module had to be removed

Electronics

• A failsafe mechanism has been implemented in the firmware framework that allows the PLCs to continue running even if train ids should not be available from the timing system
• This requires an upgrade of all loops. 
• One encoder problem in SPB was identified as a problem with the cable
• One motor problem in SPB (injector y motion) is suspected to be a faulty terminal. The motor was moved to a test crate where it works fine. This can stay like this for the moment

CAS

• KARABO 2.1.16 was released. This release is not mandatory but recommended
• Work is ongoing on the DOOCS bridge
• XGM data is availalble now but requires a check by J. Grünert
• Work continues on the Gotthardt and GreatEyes detectors
• In the case of GreatEyes the vendor has been contacted and the reply is being awaited
• Good progress was made with data analysis on the ONDA cluster

ITDM

• Accounts for users are prepared. 
• Maxwell cluster is ready for offline analysis. Example files and programs are available
• Several PCs in XHQ have been set up that have direct access to the control network
• These PCs are only for monitoring experiment activities. They are not for general use and are not connected to the internet.

General

• ​More or less continuous beam delivery over the weekend
• Beam was lost 2 times. Took almost 2h to recover in the second case
• The stability of the machine was poor. 
• SPB cannot achieve a good focus and FXE reports sync problems as the timining jumps by 220ns (1 bucket) from train to train

SPB

• Liquid jet could be operated with water for an entire shift
• Images were acquired both with powder samples as well as with the water jet
• A lot of flat field data was taken for AGIPD calibration
• One camera had a problem and CAS was called in
• There is unexpected behaviour with limit switches. Suspicion from CAS: A SW flag might be incorrectly set. A. Silenzi and B. Weinhausen will investigate)

FXE

• As stated above, the timing jumps in machine operation render pump-probe experiments impossible. A way out might be 4.5MHz operation which has other issues
• Need a lot more experience with the new scan tool
• Request is made for training on the configurator tool: ITDM reports that they have set up a script to set the many parameters from stored values
• There is a fundamental problem that the scan tools does not see some parameters available in DOOCs. Hence scanning is not really possible right now.

AE

• There are at least two arms in the XTD2 CRL in error. This requires access. Possibly bundle with the ZZ request of W. Freund
• Two loops in SPB still require update (loops5/12)
• It is not clear why many configuration parameters have to be put back by hand after a loop update. N. Jardon and B. Weinhausen will check

ITDM

• Many tests over the weekend for DAQ of slow data with SPB
• Problems were observed because train IDs were zero
• Also time stamps showed values of many hundred hours ago
• CAS suspects that this might be due to wrong configuration of some devices (use_timeserver flag)

General

• User run is currently going well. Thanks to everybody!
• Preparations need to be made for the October shutdown. Information will be collected by PSPO and T. Tschentscher. Care must be taken that the relevant people are available. Priority will be given to XTD10!

Short Report from SASE1 Instruments

• ​SPB: 
  • ​First user shift last night
  • No hickup from accelerator
  • Focus is still too large (12 - 15 um) and beam pointing and beam position are unstable
  • Calibrations were done with AGIPD using powder. 
  • DAQ ran stably
• FXE:
  • ​No report as FXE is currently on shift

Tunnels

• Optics:
  • ​Reminder: The last two metal valves in the FXE and SPB beam lines are currently not in the interlock. Hence the beam could hit them if no attention is paid. This has been fixed in the firmware but will only become active after the next reboot. This will not happen before October
  • Switching between instruments is done using a macro by Harald. It worked without hickups so far. Nevertheless, one should attenuate the beam before moving the mirrors
  • It is important to put relevant info into the operations log book. In particular since all the user log books are closed
• Diagnostics:
  • ​XGM data is now available both via DOOCS and Karabo and can be put into the DAQ stream. However, this data needs to be included. This has not been done so far
  • The XGM data should also be stored for the purpose of optimizing the facility. However, if stored with the user data, access will be difficult
  • Parallel storing of XGM and user data is not possilble currently

Laser

• Work is still progressing in line with laser delivery in November. However, the time is getting tight and overtime requests have been filed for the whole group
• Drifts of the laser beam are being observed that do not correlate with temperature and humidity. A feed back is being worked on
• A fix has been prepared by CAS for the motor problem reported earlier. However, due to the high work load from user operation this could not be tested yet. It is scheduled for next week
• DOOCSGATE was updated to allow adjustment of synchronisation

Detectors

• AGIPD:
  • ​DAQ ran stably 
  • Users took calibration data last night that allows to calibrate the different gain stages
  • A. Mancuso showed nice data on his laptop with very clean rings from the glas in the beam
• LPD
  • ​Not used yesterday
  • Should take data today
  • The calibration pipeline on the Maxwell Cluster is now enabled allowing calibrations to be calculated for both detectors

Electronics

• All instruments loops were updated before the user experiments for DAQ and for failsafe operation with train id injection
• Additional signals were prepared for the MPS system (Vacuum ok, end switches, pulse picker beam pattern). This is not yet integrated
• ZZ happened last Tuesday in XTD2 for CRL and MCP. On CRL significant problems with the cabling were identified (wrong connections, sparks in connectors). The sparks are probably the reason for the broken Beckhoff terminals. Fix can only be done in October
• In October the remaining tunnel loops with also be updated

Software

• 90% of personnel is currently working supporting the user operation in SASE1
• A hot fix for the data logger was deployed. 
• Instrument user shifts will be supported by 2 CAS people at least until end of next week. 
• Remaining personnel will work on the backlog, e. g. fixing the motor problems in the laser
• FXE:
  • DAQ for Gotthardt was deployed yesterday
  • Scantool was updated
  • Work continues on the GreatEyes detector. Currently it only works in Win7 and not in LINUX
• SPB
  • Doocs bridge is operational and slow data can be recorded, e. g. e-beam energy, K param
  • Users also made first use of the ZeroMQ tool 

General

• User run is currently going well. Thanks to everybody!
• Preparations need to be made for the October shutdown. Information will be collected by PSPO and T. Tschentscher. Care must be taken that the relevant people are available. Priority will be given to XTD10!

Short Report from SASE1 Instruments

• ​SPB: 
  • ​First user shift last night
  • No hickup from accelerator
  • Focus is still too large (12 - 15 um) and beam pointing and beam position are unstable
  • Calibrations were done with AGIPD using powder. 
  • DAQ ran stably
• FXE:
  • ​No report as FXE is currently on shift

Tunnels

• Optics:
  • ​Reminder: The last two metal valves in the FXE and SPB beam lines are currently not in the interlock. Hence the beam could hit them if no attention is paid. This has been fixed in the firmware but will only become active after the next reboot. This will not happen before October
  • Switching between instruments is done using a macro by Harald. It worked without hickups so far. Nevertheless, one should attenuate the beam before moving the mirrors
  • It is important to put relevant info into the operations log book. In particular since all the user log books are closed
• Diagnostics:
  • ​XGM data is now available both via DOOCS and Karabo and can be put into the DAQ stream. However, this data needs to be included. This has not been done so far
  • The XGM data should also be stored for the purpose of optimizing the facility. However, if stored with the user data, access will be difficult
  • Parallel storing of XGM and user data is not possilble currently

Laser

• Work is still progressing in line with laser delivery in November. However, the time is getting tight and overtime requests have been filed for the whole group
• Drifts of the laser beam are being observed that do not correlate with temperature and humidity. A feed back is being worked on
• A fix has been prepared by CAS for the motor problem reported earlier. However, due to the high work load from user operation this could not be tested yet. It is scheduled for next week
• DOOCSGATE was updated to allow adjustment of synchronisation

Detectors

• AGIPD:
  • ​DAQ ran stably 
  • Users took calibration data last night that allows to calibrate the different gain stages
  • A. Mancuso showed nice data on his laptop with very clean rings from the glas in the beam
• LPD
  • ​Not used yesterday
  • Should take data today
  • The calibration pipeline on the Maxwell Cluster is now enabled allowing calibrations to be calculated for both detectors

Electronics

• All instruments loops were updated before the user experiments for DAQ and for failsafe operation with train id injection
• Additional signals were prepared for the MPS system (Vacuum ok, end switches, pulse picker beam pattern). This is not yet integrated
• ZZ happened last Tuesday in XTD2 for CRL and MCP. On CRL significant problems with the cabling were identified (wrong connections, sparks in connectors). The sparks are probably the reason for the broken Beckhoff terminals. Fix can only be done in October
• In October the remaining tunnel loops with also be updated

Software

• 90% of personnel is currently working supporting the user operation in SASE1
• A hot fix for the data logger was deployed. 
• Instrument user shifts will be supported by 2 CAS people at least until end of next week. 
• Remaining personnel will work on the backlog, e. g. fixing the motor problems in the laser
• FXE:
  • DAQ for Gotthardt was deployed yesterday
  • Scantool was updated
  • Work continues on the GreatEyes detector. Currently it only works in Win7 and not in LINUX
• SPB
  • Doocs bridge is operational and slow data can be recorded, e. g. e-beam energy, K param
  • Users also made first use of the ZeroMQ tool 

General

• First user experiments ended on Monday evening/Tuesday morning
• They are considered a great success and overall the instruments worked better than expected
• The S1S/S3R meeting on Friday happens at 13:00 this week due to SAC

SPB

• Karabo is freezing up much less frequently than previously
• The latency of data being available for online analysis seems to be increasing
• Todos:
  • Update the laser loop (11) today
• Please keep in mind that updating the PLC loops requires a lot of hand work from the instrument afterwards and should not be done during beam time

FXE

• FXE was not represented
• The parallel operation of Gotthard and LPD was discussed. This is currently not possible. But this restriction was put deliberatly in order not to compromise LPD operation. It can be lifted whenever requested. However, the incoming user group will only need LPD. Hence this should be postponed for one week

AE

• Preparations are ongoing for work on SASE3 tunnel during October shutdown
• Prepartions are on done to connect the EPS in SPB to the DESY MPS system

CAS

• The second stage of the SPB-CAS and SPB-FXE projects has been started
• They include additional functionality that is required for user operation in November and also for the operation with the optical laser
• The next Karabo release is being prepared
• Next thing on the list is to implement the slow data acquisition for the tunnels

General

• A recent survey has shown that the experiment hall and the end of the tunnels XTD9 and XTDD10 have settled by as much as 6.5mm. If components need to be re-aligned a completely new network of points would need to be established. This is not foreseen before the end of the year. A plot is attached to these minutes

Instruments

• SPB: 
  • Recabling of many components is currently going on
  • AGIPD has been opened and some electronics are being exchanged
  • Preparations are being made to move AGIPD also to a more downstream position
  • Laser table for bringing the PP laser to the sample has been set up
• FXE:
  • Cables in the hutch have been re-routed and cleaned up
  • Preparations for the PP laser are well advanced as the Tangerine laser was already brought to the sample

Laser:

• Work is going on under pressure as the time available until first laser beam is short
• 3 Days were lost due to unannounced shut downs of the AC systems. 
• Both the SPB and the FXE NOPAs are running
• There are problems with triggering the ADC and hence the timing drift cannot be locked. The problem was reproduced in the lab and fixed. But the fix needs to be transferred to the laser hutch
• Tests with the machine timing are still outstanding and it seems that there are some resource issues on the DESY side
• Currently the optical sync system from DESY provides too little power and hence the laser cannot be synchronized by optical X-correlation. For the upcoming run period sync to the RF system will be used. This means that up to ps drifts are to be expected
• The laser is very sensitive to changes in the humidity. This has been traced to the construction of the laser tables which involves wooden side panels. The problem has been discussed with Newport and the laser tables for SASE2/3 will be modified to avoid this

Tunnels:

• Vacuum XTD10/SASE3:
  • There are currently two leaks in welded beam pipes. The welders are coming on Monday to fix this
  • After the leaks are fixed the MCP can be connected to the vacuum
  • And then finally this entire vacuum section can be closed
  • There are several leaks in the dump area. In particular the V0 valve is leaking and one flange. DESY will exchange V0 next week.
  • The pipe that hit in the vacuum incident in September appears not to be leaking
  • The absorber appears to have been damaged in the September incident and needs to be replaced. The SASE2 absorber could be used
  • Vacuum tests were done for the differential pumping downstream of the gas attenuator
  • Work is still being done one the differential pumping of the Diagnostics group
  • Interlock needs to be set up and tested
  • M. Petrich defined the interlock conditions and sent them to AE. This still needs  to be implemented
  • Cabling was done for the gas attenuator
• Diagnostics
  • Inhibits for XGMs in in XTD2/9/10 was implemented
  • Loops were successfully rebooted in XTD9
  • A lot of work done on HIREX (camera exchanged, Gotthard connected to Beckhoff and tested)
  • Today repairs will be done on the MCP in XTD2
  • For both MCPs in SASE1/3 additional Beckhoff terminals will be installed
  • Next week focus will be to finish in XTD10 and get plc control of installed devices
  • All 10 imagers in SASE3 were tested. 3 needs fixes/repairs

AE

• In order to close the loops in SASE3, focus should be given to the most relevant components
• Component lists were circulated. Please check!
• Work on KMONO is complete
• What should happen to the CRL of FXE in XTD2? Statement from A. Galler: Nothing for now as it can be operated for the upcoming user run
• Work on the instruments is ca 50% done
• In SASE3 most local test are done
• It will be tried to get the Vacuum and Motion 1 loop going next week
• The EPS loop is not so critical

ITDM

• The announced GPFS update was postponed several times but will now happen next Monday
• The DAQ for AGIPD was upgraded (The SW used so far was dating back to May)
• The time allocated was 3h. The upgrade was done after 1h, so the remaining 2h were used for tests.
• Longer runs could be taken without major problems
• People should be aware that during the first three days next week, the detectors cannot be operated due to the GPFS upgrade

Software

• Planning is being aligned between CAS (H. Sanchez) and PSPO (M. Areste)
• Progress was made for the laser. The software however will remain at version 2.13 since they do not need DAQ and the scan tool
• Yesterday version 2.18 was released. It is already deployed for SASE1
• Note that this version is not complatible with previous GUI clients
• Next week: Work with AE for SASE3
• There is no news on the scantool as S. Brockhauser was not present
• FXE should be updated to 2.18 today
• The DOOCS/KARABO interface is not considered a priority on the DESY side. They are working on the undulator controls

General

• Closing of XTD2/4/9/10 foreseen for Oct 23. Possibly XTD9/10 can be re-opened afterwards and stay open until 28 October

Instruments

• SPB:
  • A lot of cabling work was done. Cable drag chains were installed for AGIPD to move it to another posiiton
  • Roughing pipes were installed
  • Update of KARABO to 2.1.18 was not as smooth as expected. Currently it is not operational. An operational control system is needed soon
  • M. Messerschmidt sent another message with problems early today
• FXE
  • Final testing ongoing of Gotthardt with KARABO
  • Cable management and cable re-routing was done last week

Tunnels

• V0 valve was replaced
• The first vacuum section down stream from V0 was successfully closed and leak tested
• DESY Vacuum group is currently working on the respective section upstream of V0
• Welders are here today to work on the two pipes with leaks. They spent all morning on safety trainings
• The gas attenuator was leak tested on the "good" side (where vacuum section is currently ok/no leaks)
• Pumping has started on the diff pumping section by Vac
• Many small things (mostly cabling and electronics)_ remain to be done

Diagnostics

• Group was not represented by some infos from J. Tolkiehn
• More tests are needed for the Diff Pumping of the Diagnostics group.
• Diff Pumping and PES require more crates to be connected

AE

• SPB loop 5 was cycled.
• The issues with the turbo pumps is being worked on today (Pumps do not turn on after loop is cycled)
• Work is ongoing on the MKS gauge controller
• In the SASE1 tunnel the motion loop will be updated tomorrow and the vacuum loop on Wednesday 
• At this time the SPB loop in XTD9 will also be cycled
• FXE is asked to provide information on what works need to be done
• SASE3 Tunnel:
  • Interlock definitions arre prepared but need to be implemented
  • Many local tests still ongoing
  • There are still problems with inconsistent/wrong device names
  • Two motions loops (mov1/mov2) mov1 contains the most important components upstream
  • EPS not expected for some time

ITDM

• GPFS upgrade today
• redundancy data link might be done today
• Tomorrow: DAQ SW upgrade for LPD (Is still at KARABO 2.1.16.4)

CAS

• Long discussion happened on what the word "mandatory" means in conjunction with the most recent KARABO release
• CAS will look into the specific problems in SPB and FXE that have been brought up
• H. Santos discussed the scan tool with S. Brockhauser on Saturday

Dear Tobias,

Just to clarify that the last information that we got from SPB was the message below. From this information and the technical evaluation that we did the issue is not strong enough to classify Karabo 2.1.18 as "Currently it is not operational. An operational control system is needed soon". This statement is too strong and does not contain technical fundamental. 

Anyway, we already mentioned in the meeting that since Friday we have been working to solve the problem and we already have alternatives. We will publish a patch (Karabo 2.1.18.x) today or tomorrow.

Finally, I talked offline with Adrian and we agreed about some topics. Especially to review the "mandatory" word in the release process.

Hugo

From: "Marc Messerschmidt" <marc.messerschmidt@desy.de>
To: "Hugo Santos" <hugo.santos@xfel.eu>, "chen xu" <chen.xu@xfel.eu>, "cyril danilevski" <cyril.danilevski@xfel.eu>, "Marc Messerschmidt" <marc.messerschmidt@xfel.eu>, "Jolanta Sztuk-Dambietz" <jolanta.sztuk@xfel.eu>, "Britta Weinhausen" <britta.weinhausen@xfel.eu>, "Alessandro Silenzi" <alessandro.silenzi@xfel.eu>
Cc: "John Wiggins" <john.wiggins@xfel.eu>
Sent: Friday, 13 October, 2017 20:47:41
Subject: AW: GUI Responsiveness in SPB

I saw some issues today as well when instantiating devices, but nothing as bad as yesterday. The gui delay also seemed functional and went only up top around 30s.

Marc 

General

• SASE1 Instruments are asked to comment on progress with KARABO 2.1.18
• Tunnels XTD9/10 will be closed on 25 Oct at 1700

Instruments

• FXE (A. Galler):
  • KARABO 2.1.18 appears to be working ok. The device instantiation problem reported earlier in the week is no longer there. However, not all equipment could be tested yet
  • Work currently focusses on preparing for the next User Run and in particular on preparing the sample environment
• SPB (A. Mancuso)
  • A situation similar to the on in FXE is observed. Full proof of functionality of KARABO will only be possible once AGIPD is fully assembled again which is next week at the earliest
  • Currently, work is going on inserting the AGIPD cables into drag chains so that the detector can be moved on the CSS. This work should finish today

Tunnels (J. Eidam, R. Villanueva, T. Maltezopoulos)

• Leaky vacuum pipes were successfully welded
• Absorber section is closed and under vacuum
• MCP detector is connected
• DESY finished work on the upstream section in the dump
• The gas attenuator has very good vacuum (10^-9)
• Differential pumping is ready from the HW side
• Cabling for the gas attenuator section is still going on
• Alignment was done in view of the small apertures in the gas section
• Local tests can start now (This is 3 days behind the schedule made at the beginning of the week)
• PES is installed but will only act as a pipe.

Laser (M Lederer)

• NOPA Beam is now going to FXE
• Next week it will be brought to SPB
• The KARABO version used is several versions back with respect to the one in the instruments and there are still a few minor problems
• On Monday work on programming the laser burst patterns together with the respective people at DESY shall start
• Continue testing
• There appears to be a shortage of conversion crystals for the instruments. New ones have been ordered but are not there yet.

Detectors (P. Lang)

• AGIPD HW work is on track and finishes thie week. Software work starts next week
• LPD control system works ok with KARABO 2.1.18. 
• Some small issues with LPD calibrations

ITDM (K. Wrona)

• Two updates were done for the DAQ of LPD
• Some issues were found with the RC
• It was tried to install a new version. This was already done for AGIPD

AE (T. Freyermuth, B. Baranasic, J. Tolkiehn)

• SASE1 Tunnel:
  • Vacuum loop was updated without problems
  • The motion loop was updated a few times because there were problems with some motors. The work continues today
  • EPS loop was updated
  • On Monday/Tuesday the SPB loop will be updated to account for the terminal changes from 24V -> 48V
• SASE1 Instruments:
  • Loop 5 with the redundancy module is now ok
  • Loops 2, 4, 7 were updated
  • Next come loops 1 and 12
  • All modified modules (24V -> 48V) are installed but a few more might need to be changed
• SASE3 Tunnel:
  • Tried to get the vacuum loop up
  • But there are still many mistakes and inconsistencies with names
  • A big amount of work is still needed for the gas attenuator

CAS (S. Brockhauser)

• KARABO 2.1.18 was deployed
• Good progress was made with the Gotthard and GreatEyes Detectors
• For GreatEyes a specific driver on a specific computer will be used
• The problem with extra delays in the GUI client was resolved
• Scantool:
  • A plan of what to implement exists. This needs to be checked with the instruments
  • A first part of this tool is actually already implemented
  • A. Galler points out that he sees the problems predominantly in the integration with the rest of the environment
• DOOCS2KARABO
  • An example for pipelined data was sent to DESY
  • There is no showstopper seen, but the work is progressing according to priorities which are 1 Laser, 2 Undulators, 3 DOOCS2KARABO
  • Diagnostics points out that the will need both the scantool and DOOCS2KARABO for their commissioning period from 1 - 7 Nov. CAS should please provide information so that the tools can be tried

FXE:

• Pump-probe beam is now in the instrument hutch
• Currently working on setting up the remainder of beam steering and compression
• Gotthardt is working. Tests are going on a and some issues with the DAQ still need to be solved
• The sample stack needs to be moved up stream for the next user period

SPB

• Still working on AGIPD. Many cables needed to be disconnected and are being re-connected now
• AE finished all the scheduled modifications on the crates and are doing some more that were originally not scheduled
• A few details with the interlock need fixing
• The newest version of Beckhoff_Devices needs to be deployed
• The previously observed slowness of the GUI is probably fixed but needs more real life exposure. Final test are being done
• The full infrastructure should be back and operational Tuesday/Wednesday

Tunnels

• Work is going on on the gas attenuator
• Interlocks are ready for the WP74 DPS and the XGM
• Detailed steps to completion in XTD10 were re-iterated by C. Youngman. Additional coordination off line after the meeting
• L. Samoylova mentioned that the mirror encoders in XTD2 are currently in error. B. Baranasic will have a look

AE

• In the new version of KARABO, the rights for reading and writing parameters from and to the PLC are now being observed, differently from before.
• This causes problems. P. Gessler mentions that he is very concerned that such a major change is introduced so briefly before the next user period. This puts AE under great pressure and leaves almost no time for testing
• All the SASE1 loops were updated because of the new "feature" in KARABO
• A. Silenzi mentions that he can implement a workaround for this "feature" so that loops need no updating. However, this requires a new KARABO release

CAS

• KARABO2DOOCS: Meeting today with the DESY control people at 1430 in BKR
• ScanTool: Meeting foreseen Tuesday/Wednesday with the instruments
• The GUI slowness necessiated a hotfix which was verified within CAS. Now waiting for feedback from instruments

General

• After discussion between XFEL Vacuum (J. Eidam) and DESY XFEL Run Coor (Matthias Scholz) it is agreed that XTD10 will only be closed on Thursday and that XFEL will search the tunnel between 15:00 and 17:00. (This information became available only after the meeting)
• This gives some extra time to finish activities in XTD10

Instruments

• FXE
  • Currently A. Parenti and FXE are hunting a spurious data loss in Gotthards
  • Otherwise the software is surprisingly stable
• SPB
  • Slow progress towards getting the Detector going again
  • The task to ease the acquistion needs help from the DESY developers

Tunnels

• XTD10:
  • A lot of progress was made by the team working in XTD10: Thank you very much!
  • The complete tunnel is under vacuum
  • Also the missing pieces between XGM/MCP and the rest were inserted 
  • Cleanup is done
  • Denis and Raul are still working on local tests of the gas attenuator system
  • Testing is somewhat slowed down because the GUI client no longer allows active manipulation of components at the moment. So the ikarabo interface is used
  • Work also continues on getting the mov1 loop going
  • Local tests are done on all imgers except on the exit slit imager
• XTD9
  • ​The FXE Mono is not finished. It needs a restart of the motion loop
  • AE will check when this is possible, but there are a lot of constraints from users and from resources
  • Since there has been a change in the interlock system, SPB needs to be in place when the shutter to XTD9 is opened. There is a risk of triggering the interlock and shutting down the machine.

AE

• Almost everything that was planned could be achieved in XTD10
• Some of the remaining work can be finished when the tunnel is closed.
  • Work is going on to finish the controls
• Even the interface to MPS that was originally not foreseen is now available

CAS

• The workaround that allows to modify parameters even though the protection does not allow this, is now in place. This should help with the trouble observed by the Vacuum Group.
• A new bug in the PLC was observed in this context. Changing a parameter whose name has changed leads to a crash. 

ITDM

• Currently the MAXWELL cluster is unavailable.  This is a serious software/hardware issued. The manufacturer is actually on site to fix the issue today

The SASE3 tunnel XTD10 was searched by Janni and her team and interlocked about one hour ago.

It is now almost ready for first lasing in SASE3:

• The vacuum is completely connected and closed from V0 all the way to the end
• All valves, pumps and gauges are included in the PLC loop and can be controlled from Karabo
• The gas attenuator section was completely tested and is also under PLC control. The turbos are isolated from the rest of the vacuum and put into a safe state
• All three DPS's are operational, tested and included
• All imagers are operational
• The XGM vacuum is part of the general vacuum and controlled and monitored from KARABO
• The interlocks of the DPSs were succesfully tested

In order to allow beam, it is estimated that one access with approximately 8h would be needed to sort out some remaining smaller issues

I would like to thank the entire team from Vacuum, Optics, Diagnostics, AE and CAS for the big effort and in particular C. Youngman for coordinating.

Instruments

• SPB:
  • When PLC loops are restarted, still many device parameters are not properly recovered. This generates a lot of work for the team. Recently an entire day was lost.
    • Answer from AE: The parameters are extracted by CAS using KARABO. It is possible that not all necessary values are returned. AE is also investigating whether there are some bugs in the firmware procedure. It is pointed out that the change from 24V to 48V led to a huge amount of changes. 
  • Injector motion is now working along all three axes
  • AGIPD automatic power up procedure is now working. Tests continue
• FXE:
  • The spurious data losses with the Gotthard Detector were further investigated. They are not resolved but they do not limit operation with 30 bunches

Tunnels

• XTD10/SASE3
  • All vacuum sections are connected, locally tested and ready
  • Part of the interlock system is also ok. A global test of the entire integrated system still needs to be performed
  • The biggest challenge was the gas attenuator section with 7 turbo pumps. All pumps were locally test and integrated into the control system. The vacuum is very good (10-9) Now they are put into a  safe state.
  • R. Villanueva would like to be informed in case whenever the interlock in XTD10 is broken so he can inspect the system
  • XGM is connected to the overall vacuum and tested. The interlock is working and it is also connected to DOOCS
  • The two imagers requested for first lasing (SR/FEL) are available
  • Approximately one shift would be needed to take care of a few remaining tasks before first lasing
• XTD9/SASE1
  • The HiRex is fully operational
  • The Si-Mono needs work but this was de-prioritized as it is not needed for the next users

Laser

• Full specs were reached for both the SPB and FXE beams. The laser is in the respective instrument laser hutches
• The laser was covered in order to reduce effects of any remaining air drafts
• Differential mode compensation is available. Common mode compensation is not
• Work still continues on the 4 agree-upon bunch patterns. Another meeting is set up for today
• Delay control is available through KARABO but requires ca 2 days testing

Detectors

• For AGIPD both  for turning on power and cooling automatic procedures were implemented and tested
• LPD was moved to Karabo 2.18. 
• One charge injection run was done and the system operated for 2 days without interruts

Electronics

• Currently the work that is still needed for SASE3 is being evaluated
• Planning for the next shut down is going on

CAS

• Currently planning next OCD period
• Very good progress was made in SASE3
• Working on DOOCS/KARABO integration together with DESY
• Work on the laser is progressing
• A. Silenzi and S. Hauff worked on the AGIPD procedures
• Work continues on reading and writing data for the GreatEyes detector
• As reported by the detector group, the stability of the DAQ is much improved

ITDM

• A major problem caused the Maxwell cluster to be down for 3 days. IBM Technicians had to be called
• In this context it became clear that the communication with the users must be optimized
• Now the system is ok
• The folders for the next user run are being set up next week. This was delayed due to the technical problems
• For the experiment log books the users will now be automatically set up
• The update of the DAQ for AGIPD is somewhat delayed due to the long time taken for the recabling
• SPB requested 18 new cameras. This effort needs to be carefully coordinated.

General

• The rack layouts and rack placement were finalized during the PRR in August. The result is the basis of the network planning. The network is currently being installed and will most likely be finished by the end of the year. Changes to the layout that affect the network planning are no longer possible. 
• The TC/XHEXP1 meetings shall be combined into one bi-weekly meeting starting on 9 Nov. A separater new invitation shall be sent

Instruments

• SPB
  • Currently preparing for the next user period
  • A lot of work still needs to be done for bringing the laser to the interaction point. This mostly concerns the beam pipe
  • Apparently there was an interlock problem related to SPB components in XTD9. SPB components were taken out of the EPS loop without SPB being informed
    • Comment from T. Freyermuth: The condition of SPB devices was preventing beam operation and noone of SPB could be reached
• FXE
  • A lot of progess was made together with CAS
  • This afternoon a DAQ stress test with several detectors is planned
  • The encoders on the CRL lens stack were disabled as this complicates the operation at the moment

Tunnels

• XTD9 was closed yesterday. Beam could arrive this weekend
• About one shift worth of work remains in XTD10 before readiness for lasing
• KMONO measurements took place on Wed/Thurs from 3PM - 3AM
• Some DAQ and KARABO issues surfaced
• But it was possible to record donut radiation patterns
• Now commissioning of HIREX is being prepared
• XGM requires some minor re-commissioning

Laser

• Beam is in the ILHs. 
• Still problems dealing with the capricious AC system
• The promised specs of the laser beam are met
• Synchronization is still to local timing. Synchronization to the machine timing only comes next week and no time remains for testing before users

Detectors

• LPD:
  • 15/16 super modules are operational
  • This week calibration data from charge injection were taken
  • The detector is ready for users
• AGIPD:
  • Detector is fully working again after a big rearrangement of cables
  • It was found the the clocking-control cable is too long and needs to be re-routed during the winter shutdown. The detector will be fine for the next users
  • Work is ongoing on the firmware in order to allow 30 frames/train rather than only 15

Electronics

• SPB:
  • Updated the tunnel loop for SPB
  • Loop 12 is up-to-date
  • Loop4: The mirror mounts were tested locally
  • The is no feed back on interlock behaviour in loop 5. If there should be problems individual conditions can be disabled
  • For loop 7 zoom and focus and mirror mounts to be tested
  • Same applies to loop 2
• XTD9:
  • want to update EPS loop to include removed SPB components today
  • Vacuum loop to be updated next week
• XTD10
  • Some ion pumps show an error condition in KARABO. J. Tolkiehn is investigating

ITDM

• User handling for log books has changed. Please consult the help log book for details or consult J. Szuba
• The meta data catalogue was updated so that instruments can now annotate with their own credentials
• Folders for the next two user proposals are created, also in the online environment
• A new version of the DAQ software needs to be deployed in order to fix a bug in slow data daq

CAS

• A number of issues need to be resolved with Diagnostics as already mentioned earlier
• The Gotthardt Detector is now working in FXE
• The promised reduced scope for the scantool has been achieved. It will be made available to the users asap, most likely today

General

• Recomissioning of the FEL has been bumpy and up to this point only very weak lasing could be achieved. 
• So far SPB had only little beam and FXE had no beam at all. Both instruments state that the commissioning time for the up coming user run is insufficient

Instruments

• FXE:
  • First SW and DAQ stress test on Friday went well.
  • Some problems appeared with the slow data that should be fixed by now
  • K. Wrona states that these problem originated from missing permissions due to change of userid
• SPB 
  • Entire instruments needed re-alignment with beam since the CSS was moved during the shutdown
  • Very low intensity and not very stable beams were available on the Weekend. The beam could not be properly focussed
  • The bender for M2 cannot reach the flat condition. Is this the temperature problem again?
  • One pump was overheating
  • AGIPD tests with new bunch patterns started this morning
  • Significantly more time needed before user operation
  • There were some problems with the interlock system that trigger beam aborts

Tunnel

• Commissioning program was very limited in the last few days
• KMONO got some first interesting results
  • Rings and Kicks could be seen
  • Some wrong undulator settings were identified

Detectors

• Firmware update was done to record all 30 bunches. However this still has problems
• Need beam for calibrations
• The planned update by ITDM of the DAQ was not yet done

Advanced Electronics

• The mentioned interlock issues in SPB require an update to loop 7
• In loop 4 need to reset the "all valves open" signal by hand if it trips
• The SPB Master loop needs updating both for MPS and timing

Software

• Tests of the scantool took place with hardware. These tests continue today
• A new data model is being worked on for the scan tool but this is not available yet
• SPB states that at this point they will not make use of the scan tool as they first have to consolidate the instrument
• FXE is involved in the CAS tests

ITDM

• As mentioned before the DAQ version is still old and needs an update. This contains many improvements
• IBM discovered a bug in GPFS and published a patch. The bug could lead to data corruption
• Decision: DAQ update and GPFS patch will be installed today in the morning.

General

• There is currently no beam less than 24h before user operation is supposed to start in SPB
• The affects everybody (Instruments, Detector, Laser 

General

• User operation is delayed by 48h due to rocky startup of FEL.
• FEL still has not reached acceptable levels of stability and power

Instruments

• SPB:
  • Instrument was brought back to a somewhat running state
  • There are still problems with motors in the sample chamber
  • Alignment with beam is ongoing at this time
• FXE
  • New scan tool was tested
  • Currently working on user-specific alignment tasks
  • Laser was routed to the sample position
  • The beam is still drifting around

Tunnels

• FEL had trouble to find lasing
• There was a controled access on Wednesday to fix various problems. However, further problems appeared
• It was decided to roll back the changes made for a new injector laser, but this did not improve the situation
• At some point 400 uJ were reached and lost again
• There are many problems with the undulator system
  • Incorrect values in lookup tables
  • Difficult operation of the entire system
• FXE should be able to start up tonight
• SPB would follow Saturday morning since time is needed for alignment during the day today
• Shifts for photon diagnostics were largely cancelled due to FEL problems
• KMONO gave some interesting results pointing to the problems in the undulator system
• Next week FEL Imager and KMONO will be installed in SASE2

Laser

• Both SASE1 instruments have laser beam with local timing. 
• Tests with machine timing start today at lunchtime

Detectors

• AGIPD is in best possible state. 
• Firmware and DAQ were upgraded
• Some commissioning time with beam is needed today
• LPD had a cooling leak  and a power problem last night
• At 6 pm yesterday, this was still not fixed

AE

• Continuing small activities for user operation

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on this Monday

CAS

• Dedicated squads for the setups of SPB/FXE, DOOCS integration, Scan Tool closed now
• Tools delivered but with significantly reduced scope
• All necessary things for DOOCS Karabo integration were delivered. Now waiting for DESY

Instruments

• SPB:
  • Waiting for beam to come in a usable condition (500 uJ and pointing within aperture of optical elements)
• FXE
  • Similar

AE

• There was an interrupt of the vacuum loop in XTD2 due to a faulty pump controller. This was fixed by ZZ
• Rotational SmarACT stage and slits in the sample chamber for SPB are being worked on

CAS

• Due to change of beam schedule, now working on implementing more devices in DOOCS-KARABO for Jan Grünert

ITDM

• Will upgrade DAQ software today. Note: Still based on KARABO 2.1.16

General

• Beam is back with good intensity (up to 1 mJ) 
• Next user period starts on  23 Nov
• Time between now and then should be used to optimize FEL, beam lines and Instruments

Instruments

• FXE:
  • Measurements of pulse intensity were done using an XGM from DESY in the hutch
  • A  pulse intensity of ca 40 uJ was found. This is in rough agreement with the expectations
  • Spectrometer was aligned
  • GreatEyes and Gotthardt detectors are working fine
  • LPD still to be checked
• SPB:
  • Using beam to do alignment
  • Not such a good focus (too big)
  • Plans are to continue with instrument commissioning today and Saturday
  • Work is currently being done by Mahadi on the SmarACT rotary stages
  • FXE points out that most likely SPB operated with the FXE CRLs inserted into the beam
  • Note: The standard "switching script" from Harald should be used to make sure parameters and devices are properly adjusted when beam is switched from one instrument to the other
  • Detector calibration on Sunday

Tunnels

• Last Friday there was a Bender failure on mirror M2
• The mirror was removed from the tunnel during the weekend and the SASE2 spare was inserted 
• In the process one Beckhoff controller was found broken
• In the removal process a further problem was caused in one of the Beckhoff loops
• This problem could be fixed from outside the tunnel 
• After this 20h were spent to redo Beam-Based Alignment in 7 iterations
• The new orbit deviates from the old by up to 80 um.
• After using the new orbit, the FEL lased quickly with good intensity
• The replaced mirror M2 could be re-aligned 
• On Thursday beam was available to FXE
• Remark from J. Grünert: Instruments should make sure that most up-to-date DOOCS panel is loaded to see proper XGM info
• Update of DAQ caused DAQ in tunnels to be broken. This is now fixed according to K. Wrona
• HIREX commissiong is going in parallel 

Detectors

• AGIPD:
  • Before the bender failure a few X-Rays could be taken on AGIPD
  • It looks like 30 X-Ray pulse can be taken now
  • Additional flat field calibration is needed now
  • On Sunday a full shift is foreseen for commissioning with beam
• LPD:
  • One quadrant was disassembled to find the cooling leak.
  • Leak now appears to be fixed
  • No sign of dead spots on detector

Laser

• We are operating in shifts:  early / late, although experiments have been cancelled, performing shift change and other excercises for commissioning.
• Operating the laser with machine timing was so far hampered by the fact that the transmitted laser patterns from the machine were limited to the actual number of x-ray pulses, i. e. mostly non-deterministic 1 or 0. Optical alignment and characterization is not possible under those conditions. Therefore, on the weekend we have switched to local timing to give guys some power.
• Following this experience, we have decided to attempt local generation of laser patterns under machine timing in order to be independent of the machine and not have to switch back and forth between local and machine timing. This has been implemented and is under test today and over night. We still observe spurious long term drifts, which are under investigation.  However, if finally successful, this can only be a make shift solution for the coming weeks. Mid to long term we need proper implementation by DESY, allowing pattern changes on short notice and independent of the machine. This is in preparation (time line ca. mid 2018).
• Currently, beams are sent to FXE and SPB with 30 pulses each, 800nm broadband, should be compressible to 15fs. This is under machine timing but with locally generated patterns.
• There are other curiosities concerning machine timing, which are related to the increased T0 arrival time jitter compared with our local timing. Debugging is ongoing. We are also testing various features under above conditions (machine timing, local pattern):  phase shifter, trigger shifter, shift change. Time line: depending on outcome, daily updates will be provided.

AE

• When the breakdown of the interlock occured in conjunction with the broken Beckhoff terminal it became obvious that crucial emergency phone numbers are not available
• Better emergency procedures are needed
• The special EPS operation mentioned to align the mirrors should be written down so that it can be implemented an no override of the interlock system is needed
• The list of shutdown activities compiled by PSPO contains a lot of work for AE. This is now being discussed. 

ITDM

• Some problems occured this week with DAQ of slow data. This should have been fixed yesterday

CAS

• Doing 24/7 OCD support 
• During switch between instrument CAS comes in to provide support
• The cancellation of user operation last weekend was not properly communicated so that people came in unnecessarily
• Next week additional work for WP74 is being done