SASE1 / SASE3 readiness --- WP74

2017-07-14

SASE1: 

• commissioning of Kmono started yesterday at 10h for some hours
• Kmono will be used for undulator commissioning as soon as Kmono is commissioned itself, and machine is ready for this
• Support to SASE tuning given to machine (ongoing)
• Maybe today: support during SASE gain studies
• During last week (service week) : GLT-Anbindung XTD2 established

SASE3:

• During last week shutdown : 
  • progress on DP installation, XGM cabling almost done, much imager and timing cabling done
• When XTD10 is closed (as now) we currently cannot do much because there is no PLC loop in SASE3.
• Non-Beckhoff technical commissioning started (e.g. some progress on MPOD HV-power supply)
• WP74 must do a lot of technical commissioning in XTD10. Our work is by far not done when all our vacuum screws are tightened. There are different WP74 components in XTD10 than in SASE1 (not just a simple copy). WP74 requires the Beckhoff loops for tech. commissioning.
• Alignment of DP „islands“ to be done in next shutdown.
• Planned for next shutdown: installation of PES in XTD10, finishing of cabling

Dear all,

the Tunnel Access Schedule has been update - potential access has been shifted from Wednesday the 19.7. to Tuesday the 18.7. Should you need this access, please inform Harald Sinn!

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

General

• From now on progress with the SASE3 tunnel will be monitored in this meeting

FXE (A. Galler)

• Work is going on to be ready for the start of instrument commissioning of Sunday
• Currently the LPD movement is being tested
• Today the firmware for the vacuum interlock system is being deployed

SPB (M. Messerschmidt)

• All inputs needed for the vacuum interlocks can now be read out
• Karabo was migrated to version 2.1.12. This appears to have some fixes for reading out cameras, but the GUI becomes unusable after ca. 10 min. Hence the version had to be rolled back to 2.1.11. This generated a lot of extra work
• The test of the emergency off system was done this week. In preparation the PLCs were shut down to avoid damage. Bringing the system back was a significant effort. This should be getting less as more and more parameters are migrated to the firmware
• A partial test of the MPOD power supplies was done in preparation for the AGIPD commissioning. Unfortunately some Wiener power supplies still have an older firmware that appears to produce voltage spikes when ramping up. This could damage the detector. Hence a firmware upgrade is needed. But no Windows PC was available on the controls network. The respective card was dismounted and taken to the lab for upgrading by K.-E. Ballack

Tunnel Installations SASE3 (M. Dommach)

• A lot of installation work was carried out during the service period last week
  • All Day 1 devices are in place
  • A lot of missing connections were made (water, pressurized air)
  • Calbing was done for the imagers
  • The Solid Attenuator was assembled
  • There are still two vacuum leaks, one of them is on top of the dump. This may have to wait for the next service period in August
• Focus during the next service period in August:
  • Install missing connection pieces and bellows in order to close the entire vacuum
  • Start bringing up Beckhoff loops with support from AE
  • Bring up as many components as possible 
• For the comments made by WP74/J. Grünert, see the respective log book entry today

Laser

• After a number of fixes and adjustments the AC parameters (Humidity/Temperature) are now according to specs

LPD

• LPD is now fully installed in FXE
• Work on the DAQ system starts next week
• Next step for AGIPD in HERA-S is to get the DAQ system working. After this the detector will be brought to SPB
• K. Wrona would like to cross check the configuration in HERA-S before this happens

AE

• Work focussed this week on getting the remaining SPB loops up, in particular loops 12 and 5
• For loop 12 a number of modules had to be taken out, modified and brought back. This relates to the operation of so-called SMARAGD motors
• Originally it was planned to test the vacuum interlock system in SPB first. This requires inter-loop communication and there are still unused loops in SPB
• Due to the problems with loop 12, it was decided to do the tests in FXE.
• The firmware for the FXE interlock should be deplyed today
• A meeting was held with MPS to clarify the interface between XFEL and MPS sind the MPS system cannot distinguish between the branch beam lines. This can now be handled by logic inside the PLC
• AE is preparing for the support of SASE3 during the next service period

CAS

• For more details see the detailed slides presented by S. Brockhauser during the meeting

AE on call service:

    Contact person:

        Nicola Coppola

    Phone

        9 - 8040

Dear all,

an updated TAS is provided: https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

There is potential access to the XTDs on next Tuesday + Wednesday (25+26.7.), as well as on Monday - Wednesday (31.7.-2.8.). Access will be organized by M. Dommach, since H. Sinn will be on vacation - so please send Martin your access request in time.

Cheers, Gerd

General

• The latency observed by SPB in the GUI client could be reproduced by CAS. Also a  potential reason was identified. As soon as this has been sufficiently tested it will be released
• DAQ tests in FXE with LPD worked for the first time yesterday for 1/16 of the detector
• FXE and SPB will show 1-2 slides about progress and ongoing work during the TC meeting on Friday morning

XTDs

• Fast valve in XTD9 still has issues with noise and will probably need to remain disabled

FXE

• Interlock for loop 1 was deployed. Still need to do loop 2. Work per loop requires ca. 3-4 h.

SPB

• Firmware for the AGIPD hv power supply was updated. The spikes during ramping are gone
• Next items to get going are the SMARAGD motors
• Loop 7 needs to be brought back. Crate 51 has been fixed.
• Work on in vacuum interlock in SPB will need to be carefully timed and coordinated after finishing FXE. Due to impending beam probably not before next week

Dear all,

according to the current planning on next Tuesday (25.7.) both XTD9 and XTD10 will be accessible from 7-14:00:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Cheers, Gerd

Dear all,

access to XTD9 and XTD10 will be available from tomorrow Tuesday the 25.7. 7:00 (Vacuum group will make sure the interlock is broken), until Wednesday 26.7. 14:00:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Please send any additional access requests to M. Dommach and T. Tschentscher.

Cheers, Gerd

General (T. Tschentscher/G. Wellenreuther)

• Further tuning of the machine might require some time which was initially dedicated to photon/experiment commissioning.
• Since Harald Sinn will be on vacation: If you need any kind of access please address T. Tschentscher or M. Dommach. TT will organize coordination of groups requiring access to the tunnels vs. requirements of the machine vs. the demand of the instruments for commissioning.
• Lasing with 30 bunches was achieved this week. Consequently the radiation protection measurements have to be undertaken as soon as possible.

FXE (C. Bressler)

• yesterday no beam at all
• attenuators are being tested
• all hardware installed

SPB (R. Bean/A. Mancuso)

• Implementation of loops required for focal characterization — mainly loops 12 and 13. Also loops required for vacuum interlock.
• Still more loops required to be brought up—working with AE down a clear priority list.
• Clear plan for AGIPD to come to SPB/SFX instrument. Requires DAQ and more.
• CAS reacted quickly to undesirable behavior (freezing) in Karabo, which is now largely solved.

Tunnel Installations SASE3 (J. Eidam, L. Samoylova, N. Kujala)

• SASE3: No access – no progress
• SASE2: work as planned is ongoing
• SASE1: There are still issues with the fast valve.
• During this and last week maintenance days, the Newton PICO motors of FXE monochromator were successfully tested in the tunnel together with Nicola. All four motors for crystal roll and pitch can be moved in vacuum. However a slow motion, necessary for fine tuning, seems still not available. Further motor parameters adjustment is needed, which will be done in the lab with test environment.
• To fulfill the technical commissioning and make the FXE mono available for photon commissioning and operation, the firmware for Alio linear stage should be created. The work is on-going. May be available for tests before mid-August, but the schedule and priorities should be clarified.
• Status of HIREX spectrometer in XTD9 tunnel: The HIREX saw FEL beam for the first time yesterday. All motors were commissioned with Karabo software and ready for commissioning with FEL beam. It is in discussion with SPB and FXE to have parasitic beam for commissioning in the coming week.

Laser (Daniel Kane)

• The pump-probe laser group reported that, with the stabilization of the AC system, we are now continuing with production of the laser system. We are still missing the functionality some components of our control system, but we have enough at the moment that it is not an impediment to us building the laser for now, however it is something that needs keeping an eye on.

Detectors (J. Sztuk-Dambietz)

• LPD: K. Wrona reported that they could observe on all channels but they are still working on storing the data. Loss of device settings induced by glitches in Karabo cost a lot of time to overcome.
• AGIPD: ten out of 16 modules have been successfully tested but there is still a lot of work to do; still testing procedures and electronics, etc. before AGIPD is moving  in to SPB in August

AE (P. Gessler)

• Focus is on loop 12 this week, continued to get remaining parts for loop 12 running
• The firmware based interlock for FXE was implemented, certain conditions are disabled but otherwise it is running.
• Plans for shut down week of SASE3 in tunnel in August are currently being made

CAS (S. Brockhauser)

• As reported previously, a release candidate for 2.1.13 has been prepared last Friday
• During the last week-end CAS could systematically reproduce the GUI delay problem SPB was reporting
• In the beginning of the week CAS could identify the source of the GUI delay issue (which was in an old code of the GUI client and got triggered only under special circumstances) and fix it.
• Solution was patched into the new 2.1.13 release resulting in a new release candidate which CAS finished testing now.
• Note that the new release will also fix most of the annoying issues (especially the configuration persistence problems) experienced recently during LDP integration to FXE
• As soon as the new release becomes finalized its deployment in the different production systems will be done under CAS coordination after consulting with all relevant WPs involved.
• The newly introduced quick release cycles with the attached deployment process (as shown on the meeting last week) will help provide access to improved releases while guaranteeing that no surprises and irreversible regressions apply.
• On the ScanTool thread opened by Christian: CAS has started to design a ScanTool around the SRD from SPB which is now being crosschecked against requirements coming from others, too, like FXE, before starting its implementation.
• Thomas, Gerd, and Harald suggested to disseminate its design, so others can also give feedback. Hence, we agreed, that Sandor Brockhauser provide this information and organize a meeting next week.

General

• The latency observed by SPB in the GUI client has been almost completely fixed. Still some "freezing" from time to time remains.
• DAQ tests in FXE with LPD worked for 12/16 of the detector. Trigger's signal stability issue being addressed by AE.
• XTD 9 and 10 will remain open from 25.07. at 7:00 to 26.07. at 14:00  
• On Wednesday afternoon the machine will be prepare to lase with 30 bunches in order to allow the radiation measurements in SASE1 Exp. Hall on Thursday and Friday.

XTDs

• Fast valve in XTD9 still has issues with noise. Issue will be addressed tomorrow with AE support.

SPB

• Integration of the turbo pumps in  the VAC loop ongoing.
• Chiller control being included in Loop 5
• Firmware for the AGIPD hv power supply was updated. The spikes during ramping are gone
• SMARACT motors being implemented. (depend on Loop 13)
• Work on in vacuum interlock in SPB will probably start this week.
• Some problems with Karabo (While measuring profiles the level scaling does not work, Region of interest values wrong) have been reported to CAS.

Dear all,

I had some minor updates of the TAS during the last days, please check again.

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Be reminded that there will be virtually no access to XTD2, XTD4 and XTD10 before the 14.8. shutdown, so plan your work accordingly. Access requests please to M. Dommach, cc T. Tschentscher.

Cheers, Gerd

General

• The radiation test in SASE1 could not take place yesterday since the machine was not able to deliver proper light intensity with 30 bunches. Another tentative is happening now.

• Instrument commissioning during the weekend

• Monday - Wednesday next week will be machine development time. XTD 9 will be accessible on request. Please contact M. Dommach. XTD 10 is foreseen to be close.

   

  SPB

• After tests Smaragd motors are not working. Under investigation with AE/CAS

• Cameras trigger for the sample environment is not working. AE has been made aware of that during the meeting. Ongoing checks.

• Vacuum interlock for Loop 5 (AGIPD) is being implemented. This has to be done before the detector installation (foreseen beginning of August).

• Freezing and delays of the GUI are being experienced. CAS ticket has been sent. This has to be addressed as soon as possible in view of the instrument commissioning time during the week end.

   

  Tunnels

• Fast valve fake triggering has not been solved. After tests the suspect was on the controller side. Controller has been replaced but the issue has not been completely fixed. Investigation ongoing. Valves are ACTIVE regardless.

• In case of fake triggering of the fast valves, there is now a procedure to open them again remotely. Please check the related entry posted by M. Dommach.

• XTD10 – progress in the last 2 days have been achieved concerning closing of vacuum gaps and leaks testing

• XTD6 – Vacuum connections ongoing.

   

  AE

• FXE Interlock deployed and under tests.

• Working on Alio Stages integration for Mono in the tunnel. The goal is to have it ready by the end of August.

• SPB Loop 12 – Smaragd motors and ELMO controllers implementation ongoing

• SPB Loop 5 is up including the AGIPD related systems. AGIPD interlock will be added once low level tests done

• SPB Loop 13 - Phase 2 cables for some modules are at different positions compared to eplans. SPB wants to leave it and rather swap module positions and hand in a change request.

• Cables for interface to Person Interlock laid. Connections to the MPS system to be done next week.

• SASE3 Tunnel - Work started on PLC projects. There is the need for accurate schematics and a list of priorities in order to plan and prepare the shutdown week.

• SASE3 Instruments – AE will start the installation of components ion the racks soon (Mid Aug. after network cables are finished in the PLC related racks).

• General remark  - Do not leave cables in tunnels disconnected after having worked!

   

  DET

• LPD – Trigger problems solved. DAQ tests to continue

• AGIPD – Detector being tested in Hera south. As soon as the important issues concerning DAQ have been solved in Hera south (time allocated next week for that). The decision to move it to the SPB hutch can be taken. Schedule may be slightly affected, depending of the test results in HERA.

   

  CAS

• 2.1.13 Karabo version has been presented yesterday

• Scan tool measurement presentation today at the Mock-up control room

Next week NO SASE1/3 Readiness meeting!

Dear all,

on Tuesday, XTD9 will be accessible from 7:00 - 14:00, due to requests by Naresh and Konrad. I updated the TAS:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

General

• SRP was not able to measure any radiation even inside D.09 and D.02. This Thursday the radiation test in D.05 (FXE) will be performed. 

• Tomorrow XTD9 open 7:00 ==>14:00

FXE

• Gotthard detectors to be tested in Karabo

SPB

• Problem with Smaract motors has been identified. Loop to be updated. 

• Cameras trigger for the sample environment is now working. Software issues.  CAS is addressing that. 

• Vacuum interlock for Loop 5 (AGIPD) is being implemented. There will be a meeting soon (today?) with AE to check the interlock definitions for AGIPD and continue work on this loop.

• Focus characterization with CRLs ongoing. Beam size reached: <10 um

• Tunnels

• Fast valve fake triggering has not been reported in the last week end. 

• In case of fake triggering of the fast valves, there is now a procedure to open them again remotely. Please check the related entry posted by M. Dommach.

• Some problems with the cameras that stopped working have been reported by A. Koch. That could be just a matter of restarting the GUI. To be checked.

​AE

• FXE Interlock definitions have been changed and this reflects in the update of loop 1 and 2. Ongoing.

• Working on Alio Stages integration for Mono in the tunnel. To start this week. 

• SPB Loop 12 – Smaract motors and ELMO controllers implementation ongoing

• SPB Loop 5 is up including the AGIPD related systems. AGIPD interlock will be added once low level tests done

• SPB Loop 13 - Phase 2 cables for some modules are at different positions compared to eplans. SPB wants to leave it and rather swap module positions and hand in a change request.

• Cables for interface to Person Interlock laid. Connections to the MPS system to be done this week.

​

• DET

• LPD – Time server was set up. Images collected from 1 quadrant. Firmware upgrade for the other quadrants ongoing. 

• AGIPD – Detector being tested in Hera south. KWrona will start the tests of DAQ in Hera today or tomorrow.

   

  CAS

• 2.1.13 Karabo version update is being agreed with the interested parties. It takes approx. 1-3 hrs. 

General

• This Thursday the radiation test in D.05 (FXE) will be performed (from approximately 0900 to 1500), access to all areas adjacent to the hutch is only allowed by those that are assisting with the measurements.

• Meeting on Friday is cancelled. We will meet again on Monday 7-08-2017.

FXE

• Gotthard detectors to be tested in Karabo. On going.
• Finished commissioning of OGT2_CRL and OGT1_ATT. The laser to be used in September installed and connected to electronics.

SPB

• Problem with Smaract motors has been identified. Loop 12 updated, full test still on going.

• Cameras trigger for the sample environment is now working. Software issues, CAS is addressing that. 

• Vacuum interlock for Loop 5 (AGIPD) is being implemented. Interlock definitions for AGIPD under discussion with AE to continue work on this loop.

• Working in the commissioning of Loop13.

Tunnels

• In case of fake triggering of the fast valves, there is now a procedure to open them again remotely. Please check the related entry posted by M. Dommach.

• Two issues with photonic cameras already solved by CAS.

• Images are displayed later, this issue will be discussed off-line with CAS.

​AE

• FXE Interlock definitions have been changed and this reflects in the update of loop 1 and 2. Done.

• Working on Alio Stages integration for Mono in the tunnel. Developing. 

• SPB Loop 12 – Smaract motors and ELMO controllers implementation done.

• SPB Loop 5 thre are some issues with Julabo Chiller, AE contacted the manufacturer for information.

• SPB Loop 13 - Phase 2 cables for some modules are at different positions compared to eplans. Change request submitted  to swap module postions. 

• Cables for interface to Person Interlock laid. Connections to the MPS system to be done this week.

• Loop 8 should be the next, EET  to clarify if the export is done and provide to AE.

DET

• LPD – Synchronization is still showing problems, this is delaying the tests.

• AGIPD – Detector being tested in Hera south. KWrona started the tests of DAQ, some minor issues arised but in general it works.

CAS

• 2.1.13 Karabo version update is being agreed with the interested parties. Implementation will be next Tuesday 8.08.2017 in the afernoon. It takes approx. 3 hrs. 

• CAS will send around a list with the tasks foreseen to be solved during the next weeks.

General

• The Radiation measurement test will be postponed until the beam reach the 30 bunches.

SPB

• Problem with Smaract motors has been identified. Loop 12 updated and working fine.

• SPB working to focus the CRL, it works but beam is not stable.

• Vacuum interlock for Loop 5 (AGIPD) is being implemented. Interlock definitions for AGIPD under discussion with AE, this is still on going.

• Export for Loop13 updated due to a reorganizattion of the modules into the rack. Firmware will follow.

• Export for Loop 8 still missing.

Tunnels

• Images are displayed later, this issue was discussed off-line with CAS. This issue has been fixed for one camera and it works, now will be deployed for the rest. Will be necessary 15min. of downtime to deploy, CAS to coordinate with Control room.

​AE

• SPB Loop 5 thre are some issues with Julabo Chiller, AE contacted the manufacturer for information, still working on this.

• SPB Loop 13 - Phase 2 cables for some modules are at different positions compared to eplans. Change request submitted  to swap module positions. 

• Loop 8 should be the next, EET  to clarify if the export is done and provide to AE.

• AE ask to define priorities for next week since they will have to work in SASE3 tunnels but threre are still many issues form SASE1 on going (Firmware, AGIPD..).

CAS

• 2.1.13 Karabo version update is being agreed with the interested parties. Implementation will be next Tuesday 8.08.2017 in the afernoon. It takes approx. 3 hrs. 

As previously announced we will deploy KARABO 2.1.13 in tunnel under topic SA1. The system will go down in 10 mins. I will announce it again as soon the system is up again.

Regards,

Wajid Ehsan

The system is up and running.

Regards,

Wajid 

General

• Main focus for Vacuum Group and AE next week will be to get as much done in XTD10/SASE3 as possible. Conflicts with activities in SASE1 need to be carefully monitored

SPB

• SMARAGD motors are still not working even though one of them worked once. Mahadi/AE is focussing on this problem. It should be solved asap

• Focus should be on loops 11 and 13. Loop 8 has lower priority at the moment. Loop 11 is urgently needed for the laser

• A vacuum event occured last night. Currently some parts of the instrument vacuum are at 10-3 mbar.

​Tunnels

• There is an issue with disk quotas for storing camera images. This is due to the fact that the DAQ system is not implemented for the tunnel components and an interim solution is being used.
• The focus of the DAQ activities is on AGIPD and LPD

​DAQ Tests

• Tests with LPD achieved recording 10000 trains. Performance is still limited when many images (>300) are being stored per train. 
• Currently data is not yet stored on gpfs but recorded on disks locally

General

• The focus of activities next week for the AE and vacuum groups is on getting as much done in XTD10 as possible. There is a conflict with the aim to bring AGIPD to SPB and start commissioning. A lot of preparatory work was done for the latter this week already. Nevertheless this conflict needs to be monitored carefully. There will be a daily stand up meeting at 11:30 in front of the MCR to monitor the ongoing work. The SASE1 displatch meetings on Monday and Wednesday at 09:30 are correspondingly cancelled

FXE

• Philipp Lang reported on the status of the LPD commissioning. The detector is essentially ready to take data with 14 out of 16 super modules on the weekend. A few software issues remain but should not stop the activities.
• FXE is working on commissioning their greateyes camera 
• Loop 7 needed to be re-worked for the van Hamos spectrometer
• Work is stil required for the Alio stage in the monochromater in XTD9. This will probably not be ready for first user operation.
• During the shutdown week work will focus on getting the instrument laser hutch going with the Tangerine laser. After the XFEL beam is back, first pump-probe experiments should be possible
• This afternoon radiation safety tests with 30 bunches and different materials shall take place

SPB

• One SmarACT motor is now running. The problem was identified as a combination of a connection and a Karabo issue
• The next step is to reconnect the remaining motors correctly and then all SmarACT motors should be operational (The firmware is set up for all)
• Updates need to be done for loop 5 (AGIPD chiller) and loop 7 (interlock). Since this can only be done after 1500 once the sample chamber is vented, this work was postponed to Monday
• SPB also provided an updated priority list to AE in order to fine tune the remaining work until user operation 

Detectors

• Moving AGIPD will start early on Monday and the detector shall arrive in SPB around lunch time
• Immediate works shall be mechanical, connection of vacuum, chillers, power and cooling. 
• A. Sztuk points out that the integration plan from SPB is very optimistic

Tunnels

• The goal of next week is to get the vacuum and the imagers into the PLC and eventually KARABO controls
• Detailed schedule planning was unfortunately stopped yesterday due to the fire next door but will be done today
• There is a scheduled power outage next week in the tunnels. Here is the respective email with information.
• The problem with the fast valves in SASE1 (spurious trips of the interlock) has not re-occured after the controllers were exchanged. They have been sent to the manufacturer

Laser

• Progress was set back last week due to un-announced work on the 11 C cooling circuit. This necessitated a re-alignment of the lasers
• Otherwise commissioning work in SASE1 is progressing

Electronics

• The points discussed are included with the affected groups

Software

• The storage for images from the cameras in the tunnels was changed due to a disk space crisis. Currently 150TB are available. This should be enough for some time
• CAS support for XTD10 work is need only after Thursday next week once the PLC work is done
• Karabo 2.1.14 was released. Currently 2.1.13 is deployed.
• Currently the data from LPD cannot be archived. Work on this topic was stopped yesterday due to the fire. Should happend today