Software:

• CAS meets with SPB and FXE on a daily basis to fine tune progression towards user operation
• A list of required deliverables for user operation has been agreed between CAS and SPB/FXE (see slide 5 in presentation by H. Santos attached to these minutes)
• The list has been associated with Karabo releases and dates
• These dates and respective progress needs to be monitored continuously as the schedule is very tight

SPB

• AGIPD installation is progressing according to plan. 
• Next steps are data taking and then the exchange of the detector modules by the consortium
• SmarACT stages in the sample chamber started working yesterday but it is not understood why they are working now while they were not working before

FXE

• Work continues on van Hamos integrations
• Next step is integration with the Tangerine laser

Tunnel Installations

• This week the focus was on SASE3 installation in XTD10
• The vacuum group closed almost all gaps
• A lot of cabling was completed
• Local tests together with AE are being carried out on the vacuum components
• MCP was delivered with too short bellows so this could not be connected
• XGM is connected on one side only
• The Diagnostics group brought PES into the tunnel
• 5/10 imagers are operational. 2 more are almost done

Laser

• Work is back on track after the cooling water incident 10 days ago
• AC controls are ok but the air flow is too high for laser operation

Electronics

• A lot of progress was made in the course of AGIPD integration in SPB (Chiller working, Gate Valve working. interlocks working and tested)
• A smaller effort was done for FXE (Some support for LPD and re-integration of van Hamos)
• Tests of the vacuum components in XTD10 are still ongoing and continue through Monday morning
• A new email with associated ticketing has been set up: ae-support@xfel.eu

Dear all,

I updated the tunnel access schedule with the most recent information provided by M. Dommach and H. Sinn. If you plan any works in the tunnels please check their availability on

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

General

• Access to tunnel XTD10 ends today around midday and beam is expected back for the instruments on Thursday

SPB

• Good progress was made over the weekend with AGIPD. On Saturday first dark images were recorded
• Today the detector hybrids are being exchanged (1.0 -> 1.1)
• The injector motion system was taken into operation
• The SmarACT stages can currently not be tested since some parts are disassembled for final AGIPD installation

FXE

• Some loose ends were tied up for LPD
• JJXray was here last week and did some more installations, e. g. I0 monitors and alignment of the sample stack
• This week Gotthard integration will proceed

Tunnels

• The vacuum loop is now running for XTD10 with the full design firmware
• Approximately 40 ion pumps were locally test and are no under PLC control
• Almost all valves were locally tested and are also under PLC control
• XGM is connected to its own PLC currently (integrating Vacuum and gas supply)

CAS

• A disucssion on the details of Gotthard integration will happen with FXE after this meeting

Dear All,

We have a issue on the control servers and we are already working on it...

Best,

Hugo

Dear All,

We solved the incident and the services are running normally. We are still try to investigate, as far as we know we have some "zumbies" process running in the GUI servers and it was not consuming the HEATBEATs.

Dear All,

I would like to report that we started today to test (internally in CAS) the Karabo 2.1.15 version.

We got delayed to prioritise and solve a showstopper issue related to GUI Client. If our tests succeed, we are foreseeing to finish the tests and release tomorrow.

Best,

Hugo

CAS

• Finishing tests for KARABO 2.1.15. Deployment should start today during the middle of the day for SPB
• SPB asks for additional slow data to be included in the DAQ. This requires tests after 2.1.15 is deployed
• For the moment 2.1.15 will only be deployed for SPB. Once it is installed on all control servers, switch from 2.1.13 can take place
• The switch can be reversed in case of problems
• AGIPD control server will remain on current software version for the moment

FXE

• FXE was unfortunately not represented
• AE needs to perform updates on loops 9 and 7 

Tunnels

• Beam should be coming back today
• XTD9 will be searched today at 1400
• In XTD10 7/10 imagers were succefully tested and integrated. The rest will have to be done in October

AE

• Need to test one motor for SPB/CSS
• Yesterday 4 encoders in SPB were fixed but one additional problem was identified with the microscope stage in the sample chamber`
• N. Jardon needs to do further updates of PLC loops for SPB. This will be done after KARABO has been upgraded, probably tomorrow

CAS

• KARABO 2.1.15 should contain crucial fixes to associate data from tunnel devices with detector images
• Initially this will be possible train-by-train
• Pulse-by-pulse association, e. g. as needed for XGM, will require further work

General

• T. Haas conveys the congratulations of the management
  • for the achievements in installing and commissioning the AGIPD in SPB
  • for the progress made with installations in XTD10/SASE3
• There will be a meeting next week in order to determine whether we are ready for "Early User Experiments" in SASE1.  
• A discussion on what needs to be prepared for this meeting followed with the following conclusions
  • The term "Early User Experiments" needs further qualification in order to establish a well defined scope
  • Instruments and support groups (SPB, FXE, AE, CAS, ITDM, Detectors) should prepare a table with their list of deliverables and the corresponding status, following the model presented by CAS in the talk by H. Santos last week
  • In the wake of the meeting it needs to be communicated to the first scheduled groups what boundary conditions they will most likely encounter

FXE

• The optical signal for laser synchronisation is still missing and not likely to arrive before next week
• Hence a pump-probe experiment cannot happen before the scheduled "Readiness Meeting" as there will also be no beam on Tuesday and Wednesday
• It will be difficult to answer the question whether a pump-probe experiment is possible
• JJ was here this week and installed the intensity monitor

SPB

• Big gate valve between AGIPD and sample chamber was opened for the first time 10 mins ago
• The liquid jet is running and the differential pumping is working so that proper vacuuum can be maintained
• Issues remain with the SmarACT stages. Resets are needed at short invervals
• Availability of "slow" data in the DAQ chain remains the hot topic

Tunnel Installations

• Work in XTD10 continued through Monday morning
• Ion pumps and valves are now connected to the PLC. 
• KARABO control has to wait for CAS resources

Laser

• Issues remain with AC system.
• The flow rates are way to high and it appears that the system was designed for much higher flows than needed
• Weiss Klimatechnik is coming on Monday for a discussion
• Otherwise, installation proceeds and is still on track for laser beam delivery in October

Detectors

• A lot of progress was made with the detectors
• However, time is still needed to take calibration data. This is estimated as one day without beam per detector
• A reduced rates both detectors can run in parallel for tests. This is not possible for the calibration

Electronics

• The encoder bug reported on Wednesday was found and consequently all loops were updated
• Currently work continues on the SPB SmarACT stages
• The interface to the MPS system still needs to be set up. This requires an interrupt and testing

CAS

• Several unexpected issues were found in the most recent KARABO release, e. g. there are problems with the time server that still prevent DAQ integration of the slow data
• This takes a lot of additional time
• A plan review has now been agreed with both instruments once per week
• A survey was made on the need for specific DOOCS devices

Dear all,

I updated the tunnel access schedule - as you can see there will be no access tomorrow. Any access requests regarding next weeks Tuesday as usual to M. Dommach, cc to Harald Sinn and T. Tschentscher.

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Cheers, Gerd

Electronics

• Currently work continues on the SPB SmarACT stages, this issue progress slowly since right now it is not possible to reproduce the error.
• SPB loops 1/2/3/4/12 already updated, they will continue with the update of loops 5/7/13 today.
• Update on FXE loops will be the next.
• It is suggested to update also the loops in XTD9.
• AE report a request from J. Grunert to work in the HYREX. So far first priority is to work in the Experiments, this issue will be on hold.

CAS

• CAS has been working in the last days to solve the problems with the DAQ integration. Main showstopper now is the update of the Beckhoff loops.
• Still working to fix some minor issues for SPB.

Tunnels

• There is a problem in the HYREX photonic camera, CAS has been investigating but is not a KARABO issue.

General:

• The length of the shut down in October is still under discussion. Currently it appears that it is shortenend to 8 working days
• The management reviewed the readiness of the instruments for first users on 14 September:
  • Very good progress was made over the last weeks
  • Data was taken with both large pixel detectors
  • A few "red lights" remain
  • In spite of the remaining difficulties, the first users will be invited
  • However, the Management shall inform the first users that they will get another beam time in March in case the experiment objectives cannot be reached
• The second  call for proposal will only cover 5 weeks in May/June
• The third call will start on 6 September for all 6 instruments

FXE

• van Hamos was aligned for the Cu K_\beta line (required for the 3rd exp)
• XRay emission was observed on the GreatEyes detector
• Work is still urgently needed to get the Gotthardt Detector working 
• The laser is synchronised to 100fs. It is currently up to the shutter in the instrument hutch

SPB

• On Monday AGIPD took first data  with 30 frames 
• The sample jet is working with sufficient vacuum to operate the detector
• Yesterday, everything was ready to take data with liquid jet and AGIPD. Unfortunately, there was a major software/electronics failure that lasted for 6.5h
  • A fuse was blown in one crate needed for the train IDs (single point of failure)
  • This affected all crates and KARABO
  • A fallback is currently being implemented
• The door interlock could not be operated because there was a problem with the operator password

Tunnel

• SASE3 tunnel installation will not be reviewed because there is no work on it
• SASE2 tunnel installations will be reviewed once AE and CAS get involved.
• Network installations may take up to 4 months (info by K. Wrona)
• First signals were seen from HIREX last week
• Currently this integrates over 30 bunches as Gotthardt is not available

Laser

• Differently from what was reported earlier, the AC system works at the required lower flow rates
• Progress on laser installation and commissioning is progressing as planned
• There are a few problems with motor operations from KARABO

Detectors

• Both AGIPD and LPD are taking uncalibrated data
• A cooling leak was detected in LPD. Experts from RAL are going to look at this today
• Calibration may just be achieved for AGIPD. Probably not for LPD due tot the HW problems

AE

• All SPB loops were updated to accommodate the train ids
• Almost all FXE loops were updated too
• The tunnel loops were not updated yet. 
• For the tunnel, the motion loop needs to be updated. EPS and Vacuum loops should remain unchanged
• There remains one open issue with an encoder on the sample injector. X and Y encoders will be exchanged today to isolate the problem

CAS

• Work this week concentrated on GUI issues (achieve faster feedback)
• One patch was released to fix GUI problems
• Good progress was made with data analysis tests together with people from CFEL
• Next important topic on the list is to get the XGM data into the KARABO DAQ stream. This requires the DOOCS interface
• KARABO 2.1.16 needs to be closed
• For FXE there is good progress with Gotthardt. This now needs testing in the hutch. The same applies to the digitizers.
• W. Wrona points out that independent KARABO installations exist for SPB, FXE and tunnels. This needs to be fixed to allow DAQ integration.

02.09. from 08:00 until 03.09. 08:00 - Wolfgang Freund 9-5630  

03.09. from 08:00 until 04.09. 08:00 - Florian Dietrich 9-6968

SPB

• Demo experiment done last week
• Flat fields taken with AGIPD over the weekend
• Focus was re-characterised. This required modifications to several beam parameters and then led to a controversial discussion with the RunCoor
• The sluggish behaviour of the GUI remains an issue and might pose a real risk if the injector freezes up and direct reflections into the detector occur
  • Kerstin and Gero looked at it last week
  • John is back from holidays and will also look
  • One observation is that the same information may update quickly on one server but not on another
• One motor is stuck in XTD9. A temporary access is requested
• On Sunday at beam hand over the CRL in XTD2 was not fully retracted. This was then done unannounced. Nothing bad happened this time but  this might cause real damage. 

FXE

• on shift and not represented at the meeting
• Harald mentions that 2 motors in the XTD2 CRL are stuck/in error. This should be fixed during this week's repair period
• Info by CB after the meeting: Scanned K_\beta and established machine energy scale within 40 eV

Tunnels

• Motion loop and SPB loop to be updated by AE tomorrow
• Tunnels and instruments are in different KARABO "Topics". K. Wrona states that this is a problem for DAQ
• SPB states that they have all information required already in their "Topic"
• It is unclear whether this might be an issue for FXE

CAS

• No specific issue

Just to clarify:

• Tunnel and SPB are in different Karabo Topics. It should not be a problem since we agreed that SPB does not need XTD2 slow data (from devices)
• Tunnel and FXE are in the same Karabo Topic
• We will keep prioritizing the configuration and deploy of Karabo-DAQ in SPB and FXE. Before going for Tunnel updates (we are mitigating risks of broadcast unknown issues just before the 1st User Experiments).  

Dear all,

I have updated the TAS with the most recent information about Tuesday and Wednesday this week:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

FXE

• Still working on getting the laser on the sample
• Commissioning the recently received intensity monitor
• Work is also ongoing on the Gotthardt detector
• LPD is still in its own "Topic" but will be moved to the SASE1 "Topic" today

SPB

• Problem developed with one ELMO motor. It was moved to a test crate where it worked
• All loops were updated to allow slow data integration into DAQ
• AGIPD running improved significantly in stability after implementing the latest nightly build of KARABO
• ITDM team has various testing tools operational on the different channels of the detector
• Charge injection scan was possible for the first time without significant degradation of KARABO response

ITDM

• Slow data is now integrated for SPB
• Request is made for the list of data sources to integrate with FXE

AE

• PG points out that it was always foreseen that the timestamp/train ID integration would be done in software
• A HW solution was implemented during his absence to generate train IDs in a  PLC. He considers this to add significant risks.
• The HW solution will not be implemented in the tunnels to avoid unexpected behaviour during the user run
• Hence a work around solution needs to be found how to uniquely identify slow tunnel data with train IDs

Discussion about time stamps and train ID

• The summary of a longer discussion about the subject leads to the following conclusions:
• Slow tunnel data could be logged via a python script that also stores time stamps and train ids. An integration of this data with the fast data would have to be done offline post mortem
• Different opinions exist whether the association between time stamps and train ids in the KARABO data logger is unique and compatible with the train ids used in DAQ
• It is decided that this will be checked now and the result reported during the Friday S1R meeting

Announcement by ITDM

• The elog will be upgraded this afternoon
• The online cluster servers will be rebooted this afternoon

Dear all,

I updated the TAS to incorporate the newest dates for the October shutdown (currently 4.10.-23.10.):

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

General

• User operation starts on Thursday 14 September 08:00 in FXE
• Instruments alternate every 12h switching at 0800 and 2000
• A common SASE1 operations log book has been set up here. It is accessible for all inside XFEL
• The PIs of the first 6 user groups have been informed that the scope of their experiments will be rather "user-assisted commissioning". In compensation they will get more beam time in March 2018 if the objectives of their proposals could not be reached

SPB

• An effort was made with ACC to achieve good conditions for focussing. So far this has not been successful
• At 2200 on thursday these efforts were abandoned as there were no more ideas to try
• Also the power level in the beam was rather low
• Negative effects of running the detector on KARABO responsiveness are much reduced now
• This night KARABO froze up completely (no detector running) and no support from CAS was available (No response from calling the OCD mobile)

FXE

• Cross calibration effort for the beam energy was performed with HIREX. This succeeded after some efforts.
• The absolute energy scale could be established to within 70 eV. The fluctuations (averaged over several seconds) are around 20 eV
• Effort to synchronize the tangerine laser beam and to bring it to the sample continues. The quality of the sync signal is not very good. There are big signal losses in the fiber

Laser

• Still two problems with KARABO: Cannot reliably read date from wave guides and the python to doocs interfaces does not implement essential data types
• Currently working on aligning the NOPA
• The fact that the synch signal will not be available until end Sept puts a strain on the planned dates for beam delivery
• The full SYNC hutch will not be available until end Dec

Detectors

• The new GUI version gives good response while detector is running
• SPB was instructed on how to use the offline calibration
• LPD was brought into  the SASE1 broker topic
• Coolant leak in LPD was fixed but 1 super module had to be removed

Electronics

• A failsafe mechanism has been implemented in the firmware framework that allows the PLCs to continue running even if train ids should not be available from the timing system
• This requires an upgrade of all loops. 
• One encoder problem in SPB was identified as a problem with the cable
• One motor problem in SPB (injector y motion) is suspected to be a faulty terminal. The motor was moved to a test crate where it works fine. This can stay like this for the moment

CAS

• KARABO 2.1.16 was released. This release is not mandatory but recommended
• Work is ongoing on the DOOCS bridge
• XGM data is availalble now but requires a check by J. Grünert
• Work continues on the Gotthardt and GreatEyes detectors
• In the case of GreatEyes the vendor has been contacted and the reply is being awaited
• Good progress was made with data analysis on the ONDA cluster

ITDM

• Accounts for users are prepared. 
• Maxwell cluster is ready for offline analysis. Example files and programs are available
• Several PCs in XHQ have been set up that have direct access to the control network
• These PCs are only for monitoring experiment activities. They are not for general use and are not connected to the internet.

General

• ​More or less continuous beam delivery over the weekend
• Beam was lost 2 times. Took almost 2h to recover in the second case
• The stability of the machine was poor. 
• SPB cannot achieve a good focus and FXE reports sync problems as the timining jumps by 220ns (1 bucket) from train to train

SPB

• Liquid jet could be operated with water for an entire shift
• Images were acquired both with powder samples as well as with the water jet
• A lot of flat field data was taken for AGIPD calibration
• One camera had a problem and CAS was called in
• There is unexpected behaviour with limit switches. Suspicion from CAS: A SW flag might be incorrectly set. A. Silenzi and B. Weinhausen will investigate)

FXE

• As stated above, the timing jumps in machine operation render pump-probe experiments impossible. A way out might be 4.5MHz operation which has other issues
• Need a lot more experience with the new scan tool
• Request is made for training on the configurator tool: ITDM reports that they have set up a script to set the many parameters from stored values
• There is a fundamental problem that the scan tools does not see some parameters available in DOOCs. Hence scanning is not really possible right now.

AE

• There are at least two arms in the XTD2 CRL in error. This requires access. Possibly bundle with the ZZ request of W. Freund
• Two loops in SPB still require update (loops5/12)
• It is not clear why many configuration parameters have to be put back by hand after a loop update. N. Jardon and B. Weinhausen will check

ITDM

• Many tests over the weekend for DAQ of slow data with SPB
• Problems were observed because train IDs were zero
• Also time stamps showed values of many hundred hours ago
• CAS suspects that this might be due to wrong configuration of some devices (use_timeserver flag)

Dear all,

I updated the tunnel access schedule to reflect the access to the XTDs tomorrow.

Cheers, Gerd