Dear all,

I have updated the TAS with the most recent information about Tuesday and Wednesday this week:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

Just to clarify:

• Tunnel and SPB are in different Karabo Topics. It should not be a problem since we agreed that SPB does not need XTD2 slow data (from devices)
• Tunnel and FXE are in the same Karabo Topic
• We will keep prioritizing the configuration and deploy of Karabo-DAQ in SPB and FXE. Before going for Tunnel updates (we are mitigating risks of broadcast unknown issues just before the 1st User Experiments).  

SPB

• Demo experiment done last week
• Flat fields taken with AGIPD over the weekend
• Focus was re-characterised. This required modifications to several beam parameters and then led to a controversial discussion with the RunCoor
• The sluggish behaviour of the GUI remains an issue and might pose a real risk if the injector freezes up and direct reflections into the detector occur
  • Kerstin and Gero looked at it last week
  • John is back from holidays and will also look
  • One observation is that the same information may update quickly on one server but not on another
• One motor is stuck in XTD9. A temporary access is requested
• On Sunday at beam hand over the CRL in XTD2 was not fully retracted. This was then done unannounced. Nothing bad happened this time but  this might cause real damage. 

FXE

• on shift and not represented at the meeting
• Harald mentions that 2 motors in the XTD2 CRL are stuck/in error. This should be fixed during this week's repair period
• Info by CB after the meeting: Scanned K_\beta and established machine energy scale within 40 eV

Tunnels

• Motion loop and SPB loop to be updated by AE tomorrow
• Tunnels and instruments are in different KARABO "Topics". K. Wrona states that this is a problem for DAQ
• SPB states that they have all information required already in their "Topic"
• It is unclear whether this might be an issue for FXE

CAS

• No specific issue

02.09. from 08:00 until 03.09. 08:00 - Wolfgang Freund 9-5630  

03.09. from 08:00 until 04.09. 08:00 - Florian Dietrich 9-6968

General:

• The length of the shut down in October is still under discussion. Currently it appears that it is shortenend to 8 working days
• The management reviewed the readiness of the instruments for first users on 14 September:
  • Very good progress was made over the last weeks
  • Data was taken with both large pixel detectors
  • A few "red lights" remain
  • In spite of the remaining difficulties, the first users will be invited
  • However, the Management shall inform the first users that they will get another beam time in March in case the experiment objectives cannot be reached
• The second  call for proposal will only cover 5 weeks in May/June
• The third call will start on 6 September for all 6 instruments

FXE

• van Hamos was aligned for the Cu K_\beta line (required for the 3rd exp)
• XRay emission was observed on the GreatEyes detector
• Work is still urgently needed to get the Gotthardt Detector working 
• The laser is synchronised to 100fs. It is currently up to the shutter in the instrument hutch

SPB

• On Monday AGIPD took first data  with 30 frames 
• The sample jet is working with sufficient vacuum to operate the detector
• Yesterday, everything was ready to take data with liquid jet and AGIPD. Unfortunately, there was a major software/electronics failure that lasted for 6.5h
  • A fuse was blown in one crate needed for the train IDs (single point of failure)
  • This affected all crates and KARABO
  • A fallback is currently being implemented
• The door interlock could not be operated because there was a problem with the operator password

Tunnel

• SASE3 tunnel installation will not be reviewed because there is no work on it
• SASE2 tunnel installations will be reviewed once AE and CAS get involved.
• Network installations may take up to 4 months (info by K. Wrona)
• First signals were seen from HIREX last week
• Currently this integrates over 30 bunches as Gotthardt is not available

Laser

• Differently from what was reported earlier, the AC system works at the required lower flow rates
• Progress on laser installation and commissioning is progressing as planned
• There are a few problems with motor operations from KARABO

Detectors

• Both AGIPD and LPD are taking uncalibrated data
• A cooling leak was detected in LPD. Experts from RAL are going to look at this today
• Calibration may just be achieved for AGIPD. Probably not for LPD due tot the HW problems

AE

• All SPB loops were updated to accommodate the train ids
• Almost all FXE loops were updated too
• The tunnel loops were not updated yet. 
• For the tunnel, the motion loop needs to be updated. EPS and Vacuum loops should remain unchanged
• There remains one open issue with an encoder on the sample injector. X and Y encoders will be exchanged today to isolate the problem

CAS

• Work this week concentrated on GUI issues (achieve faster feedback)
• One patch was released to fix GUI problems
• Good progress was made with data analysis tests together with people from CFEL
• Next important topic on the list is to get the XGM data into the KARABO DAQ stream. This requires the DOOCS interface
• KARABO 2.1.16 needs to be closed
• For FXE there is good progress with Gotthardt. This now needs testing in the hutch. The same applies to the digitizers.
• W. Wrona points out that independent KARABO installations exist for SPB, FXE and tunnels. This needs to be fixed to allow DAQ integration.

Electronics

• Currently work continues on the SPB SmarACT stages, this issue progress slowly since right now it is not possible to reproduce the error.
• SPB loops 1/2/3/4/12 already updated, they will continue with the update of loops 5/7/13 today.
• Update on FXE loops will be the next.
• It is suggested to update also the loops in XTD9.
• AE report a request from J. Grunert to work in the HYREX. So far first priority is to work in the Experiments, this issue will be on hold.

CAS

• CAS has been working in the last days to solve the problems with the DAQ integration. Main showstopper now is the update of the Beckhoff loops.
• Still working to fix some minor issues for SPB.

Tunnels

• There is a problem in the HYREX photonic camera, CAS has been investigating but is not a KARABO issue.

Dear all,

I updated the tunnel access schedule - as you can see there will be no access tomorrow. Any access requests regarding next weeks Tuesday as usual to M. Dommach, cc to Harald Sinn and T. Tschentscher.

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Cheers, Gerd

General

• T. Haas conveys the congratulations of the management
  • for the achievements in installing and commissioning the AGIPD in SPB
  • for the progress made with installations in XTD10/SASE3
• There will be a meeting next week in order to determine whether we are ready for "Early User Experiments" in SASE1.  
• A discussion on what needs to be prepared for this meeting followed with the following conclusions
  • The term "Early User Experiments" needs further qualification in order to establish a well defined scope
  • Instruments and support groups (SPB, FXE, AE, CAS, ITDM, Detectors) should prepare a table with their list of deliverables and the corresponding status, following the model presented by CAS in the talk by H. Santos last week
  • In the wake of the meeting it needs to be communicated to the first scheduled groups what boundary conditions they will most likely encounter

FXE

• The optical signal for laser synchronisation is still missing and not likely to arrive before next week
• Hence a pump-probe experiment cannot happen before the scheduled "Readiness Meeting" as there will also be no beam on Tuesday and Wednesday
• It will be difficult to answer the question whether a pump-probe experiment is possible
• JJ was here this week and installed the intensity monitor

SPB

• Big gate valve between AGIPD and sample chamber was opened for the first time 10 mins ago
• The liquid jet is running and the differential pumping is working so that proper vacuuum can be maintained
• Issues remain with the SmarACT stages. Resets are needed at short invervals
• Availability of "slow" data in the DAQ chain remains the hot topic

Tunnel Installations

• Work in XTD10 continued through Monday morning
• Ion pumps and valves are now connected to the PLC. 
• KARABO control has to wait for CAS resources

Laser

• Issues remain with AC system.
• The flow rates are way to high and it appears that the system was designed for much higher flows than needed
• Weiss Klimatechnik is coming on Monday for a discussion
• Otherwise, installation proceeds and is still on track for laser beam delivery in October

Detectors

• A lot of progress was made with the detectors
• However, time is still needed to take calibration data. This is estimated as one day without beam per detector
• A reduced rates both detectors can run in parallel for tests. This is not possible for the calibration

Electronics

• The encoder bug reported on Wednesday was found and consequently all loops were updated
• Currently work continues on the SPB SmarACT stages
• The interface to the MPS system still needs to be set up. This requires an interrupt and testing

CAS

• Several unexpected issues were found in the most recent KARABO release, e. g. there are problems with the time server that still prevent DAQ integration of the slow data
• This takes a lot of additional time
• A plan review has now been agreed with both instruments once per week
• A survey was made on the need for specific DOOCS devices

CAS

• Finishing tests for KARABO 2.1.15. Deployment should start today during the middle of the day for SPB
• SPB asks for additional slow data to be included in the DAQ. This requires tests after 2.1.15 is deployed
• For the moment 2.1.15 will only be deployed for SPB. Once it is installed on all control servers, switch from 2.1.13 can take place
• The switch can be reversed in case of problems
• AGIPD control server will remain on current software version for the moment

FXE

• FXE was unfortunately not represented
• AE needs to perform updates on loops 9 and 7 

Tunnels

• Beam should be coming back today
• XTD9 will be searched today at 1400
• In XTD10 7/10 imagers were succefully tested and integrated. The rest will have to be done in October

AE

• Need to test one motor for SPB/CSS
• Yesterday 4 encoders in SPB were fixed but one additional problem was identified with the microscope stage in the sample chamber`
• N. Jardon needs to do further updates of PLC loops for SPB. This will be done after KARABO has been upgraded, probably tomorrow

CAS

• KARABO 2.1.15 should contain crucial fixes to associate data from tunnel devices with detector images
• Initially this will be possible train-by-train
• Pulse-by-pulse association, e. g. as needed for XGM, will require further work

Dear All,

I would like to report that we started today to test (internally in CAS) the Karabo 2.1.15 version.

We got delayed to prioritise and solve a showstopper issue related to GUI Client. If our tests succeed, we are foreseeing to finish the tests and release tomorrow.

Best,

Hugo

Dear All,

We solved the incident and the services are running normally. We are still try to investigate, as far as we know we have some "zumbies" process running in the GUI servers and it was not consuming the HEATBEATs.

Dear All,

We have a issue on the control servers and we are already working on it...

Best,

Hugo

General

• Access to tunnel XTD10 ends today around midday and beam is expected back for the instruments on Thursday

SPB

• Good progress was made over the weekend with AGIPD. On Saturday first dark images were recorded
• Today the detector hybrids are being exchanged (1.0 -> 1.1)
• The injector motion system was taken into operation
• The SmarACT stages can currently not be tested since some parts are disassembled for final AGIPD installation

FXE

• Some loose ends were tied up for LPD
• JJXray was here last week and did some more installations, e. g. I0 monitors and alignment of the sample stack
• This week Gotthard integration will proceed

Tunnels

• The vacuum loop is now running for XTD10 with the full design firmware
• Approximately 40 ion pumps were locally test and are no under PLC control
• Almost all valves were locally tested and are also under PLC control
• XGM is connected to its own PLC currently (integrating Vacuum and gas supply)

CAS

• A disucssion on the details of Gotthard integration will happen with FXE after this meeting

Dear all,

I updated the tunnel access schedule with the most recent information provided by M. Dommach and H. Sinn. If you plan any works in the tunnels please check their availability on

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

Software:

• CAS meets with SPB and FXE on a daily basis to fine tune progression towards user operation
• A list of required deliverables for user operation has been agreed between CAS and SPB/FXE (see slide 5 in presentation by H. Santos attached to these minutes)
• The list has been associated with Karabo releases and dates
• These dates and respective progress needs to be monitored continuously as the schedule is very tight

SPB

• AGIPD installation is progressing according to plan. 
• Next steps are data taking and then the exchange of the detector modules by the consortium
• SmarACT stages in the sample chamber started working yesterday but it is not understood why they are working now while they were not working before

FXE

• Work continues on van Hamos integrations
• Next step is integration with the Tangerine laser

Tunnel Installations

• This week the focus was on SASE3 installation in XTD10
• The vacuum group closed almost all gaps
• A lot of cabling was completed
• Local tests together with AE are being carried out on the vacuum components
• MCP was delivered with too short bellows so this could not be connected
• XGM is connected on one side only
• The Diagnostics group brought PES into the tunnel
• 5/10 imagers are operational. 2 more are almost done

Laser

• Work is back on track after the cooling water incident 10 days ago
• AC controls are ok but the air flow is too high for laser operation

Electronics

• A lot of progress was made in the course of AGIPD integration in SPB (Chiller working, Gate Valve working. interlocks working and tested)
• A smaller effort was done for FXE (Some support for LPD and re-integration of van Hamos)
• Tests of the vacuum components in XTD10 are still ongoing and continue through Monday morning
• A new email with associated ticketing has been set up: ae-support@xfel.eu

Dear All,

In XTD10 there are now office network ports available in each rack. These can be used to connect your regular laptops with a network cable without special configuration and registration of your laptop.

The list of racks and corresponding port numbers is listed below. Please, note that in lead racks (3.4, 3.7, and 3.13) you need to plug the cable directly to the office switch port 24. This port is located in the bottom row, most to the right.

General

• The focus of activities next week for the AE and vacuum groups is on getting as much done in XTD10 as possible. There is a conflict with the aim to bring AGIPD to SPB and start commissioning. A lot of preparatory work was done for the latter this week already. Nevertheless this conflict needs to be monitored carefully. There will be a daily stand up meeting at 11:30 in front of the MCR to monitor the ongoing work. The SASE1 displatch meetings on Monday and Wednesday at 09:30 are correspondingly cancelled

FXE

• Philipp Lang reported on the status of the LPD commissioning. The detector is essentially ready to take data with 14 out of 16 super modules on the weekend. A few software issues remain but should not stop the activities.
• FXE is working on commissioning their greateyes camera 
• Loop 7 needed to be re-worked for the van Hamos spectrometer
• Work is stil required for the Alio stage in the monochromater in XTD9. This will probably not be ready for first user operation.
• During the shutdown week work will focus on getting the instrument laser hutch going with the Tangerine laser. After the XFEL beam is back, first pump-probe experiments should be possible
• This afternoon radiation safety tests with 30 bunches and different materials shall take place

SPB

• One SmarACT motor is now running. The problem was identified as a combination of a connection and a Karabo issue
• The next step is to reconnect the remaining motors correctly and then all SmarACT motors should be operational (The firmware is set up for all)
• Updates need to be done for loop 5 (AGIPD chiller) and loop 7 (interlock). Since this can only be done after 1500 once the sample chamber is vented, this work was postponed to Monday
• SPB also provided an updated priority list to AE in order to fine tune the remaining work until user operation 

Detectors

• Moving AGIPD will start early on Monday and the detector shall arrive in SPB around lunch time
• Immediate works shall be mechanical, connection of vacuum, chillers, power and cooling. 
• A. Sztuk points out that the integration plan from SPB is very optimistic

Tunnels

• The goal of next week is to get the vacuum and the imagers into the PLC and eventually KARABO controls
• Detailed schedule planning was unfortunately stopped yesterday due to the fire next door but will be done today
• There is a scheduled power outage next week in the tunnels. Here is the respective email with information.
• The problem with the fast valves in SASE1 (spurious trips of the interlock) has not re-occured after the controllers were exchanged. They have been sent to the manufacturer

Laser

• Progress was set back last week due to un-announced work on the 11 C cooling circuit. This necessitated a re-alignment of the lasers
• Otherwise commissioning work in SASE1 is progressing

Electronics

• The points discussed are included with the affected groups

Software

• The storage for images from the cameras in the tunnels was changed due to a disk space crisis. Currently 150TB are available. This should be enough for some time
• CAS support for XTD10 work is need only after Thursday next week once the PLC work is done
• Karabo 2.1.14 was released. Currently 2.1.13 is deployed.
• Currently the data from LPD cannot be archived. Work on this topic was stopped yesterday due to the fire. Should happend today

General

• Main focus for Vacuum Group and AE next week will be to get as much done in XTD10/SASE3 as possible. Conflicts with activities in SASE1 need to be carefully monitored

SPB

• SMARAGD motors are still not working even though one of them worked once. Mahadi/AE is focussing on this problem. It should be solved asap

• Focus should be on loops 11 and 13. Loop 8 has lower priority at the moment. Loop 11 is urgently needed for the laser

• A vacuum event occured last night. Currently some parts of the instrument vacuum are at 10-3 mbar.

​Tunnels

• There is an issue with disk quotas for storing camera images. This is due to the fact that the DAQ system is not implemented for the tunnel components and an interim solution is being used.
• The focus of the DAQ activities is on AGIPD and LPD

​DAQ Tests

• Tests with LPD achieved recording 10000 trains. Performance is still limited when many images (>300) are being stored per train. 
• Currently data is not yet stored on gpfs but recorded on disks locally

The system is up and running.

Regards,

Wajid 

As previously announced we will deploy KARABO 2.1.13 in tunnel under topic SA1. The system will go down in 10 mins. I will announce it again as soon the system is up again.

Regards,

Wajid Ehsan