XTD Tunnels open until 15:00. Later XTD2 and XTD4 are closed, XTD9/10 remain open

Vacuum Loop:

Status:

• One broken Beckhoff terminal was exchanged before the weekend. This was the first failure of this type in 2 years
• After this the system ran without hickup over the Easter weekend
• The fix applied for ion pumps that avoids triggering an interlock condition when there is a communication failure on the serial output should be also applied now to other vacuum device, e. g. the turbo pumps

EPS Loop:

Status:

• Requires an update to be done asap so that N. Gerassimova can continue tests with the PBLMs afterwards

Motion Loop:

Status:

• Movement of the PBLM motors (ELMO controller) is not satisfactory yet. Motors start and stop but the movement is uneven and stops not always requested
• There is no test set up in the lab. So tests have to be done in the tunnels (XTD9)

PBLMs:

• All systems in XTD2 were tested and cameras calibrated including some at the beginning of XTD9
• Work continues on remaining systems in XTD9
• System with motor movement is still unsatisfactory (see above)

Imagers

Status:

• 2 Imagers in XTD2 have no trigger signal. Suspicion is that this might be configuration problem. P. Gessler will investigate
• One pop-in (245) in XTD9 cannot be reached with IP. Suspicion is incorrect network configuration -> K. Fillipakopoulos

SPB Tunnel components

• Signal pins of the Attenuator crate were tested. DC voltage is seen there at different levels on different pins. This looks problematic -> P. Gessler

Karabo:

Status:

• Ran stably over the weekend 

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Take equipment and junk out of tunnels
• Check trigger cabling for imagers in XTD2
• Check PBLM motor movement and continue testing in XTD9

Other topics

• Controlled access to tunnels is possible after 1500. Coordination with T. Haas and the DESY run coordination

Dear all,

The EPS loop is running again with the last modifications concerning devices' names.

Thank you.

Dear all,

The Motion loop would be updated now in order to fix names problems with PSLITs motors. In few minutes would be reachable again.

Than you!

The first EtherCAT coupler in 1.7 VAC was broken and is replaced now. The loop is up and running again.

Cheers,

    Tobias

All northern tunnels XTD2/4/9/10 are fully accessible again! 

Radiation Interlock Tests are finished. Tunnels XTD9 and XTD10 up to the fence at the dump can be accessed right now. Magnet current interlock tests still takes a bit of time (ca. 1h). Once this is also done, I shall post a note 

 Tunnel Closed until approx 1300, Open over the Easter Weekend until Tuesday 1500.

Vacuum:

• All valves and pumps in XTD2 are working and can be operated remotely
• The vacuum system including DPS and XGM is interlocked. Interlocks have been tested and are working properly
• The fast valve and the absorber at the interface to the electron beam vacuum are working and interlocked. The communication with DESY has been successfully tested
• Some issue exist when switching turbo pumps on/off in Karabo. Then erratic behaviour is observed. Doing the same remotely communicating via TCP directly works ok
• More testing needs to be done in XTD9

Mirrors, PBLMs, SRA, Attenuator

• M1 and M2 are installed and under vacuum. M3 will be installed after easter
• 3 out of 5 motors can be operated in M1 and M2. The mirrors have been moved into a safe position
• PBLM motors (ELMO driver) can be operated (started and stopped) but operation is still problematic (motors stop in wrong position, movement not linear)
• On the SRA the connection to two motors is interchanged
• On the attenuator certain motors occasionally get stuck
• Slits have been moved out so the beam line is safe for first operation

Imagers:

•  
• All imagers tested, motors and cameras, motion loop, EPS loop, for 3 imagers XTD2, and 3 imagers XTD9.
• All blocking issues concerning tunnel closure solved.
• Some remaining points that can be treated remotely: Trigger, software delays, others.
• The Photonics Camera driver is now working in Karabo.
• Cameras can currently be read out at 3 Hz which is too slow but ok for first beam commissioning

XGM:

•  
• XGM inserted into vacuum loop and interlock (in XTD2 and XTD9)
•  
• Grounding completed in XTD2+XTD9 (power cycle of rack was needed)
• Gases ready to be used in XTD2
• XGM @ XTD2 ready for beam         (need DOOCS middle-layer servers activated) 

FXE

• Testing finished for all devices in loop1
• Now waiting for loops 2 and beyond
•  

SPB

• SPB tunnel loop deployed. Testing can start next week
• For the Optics hutch some Beckhoff crates are still missing 
•  

PLC

• Fixes to MC2 library (ELMO and TECHNOSOFT motors) will deployed next week and then tested in XTD9
• FXE loop 2 was modified and loops 3&4 are deployed
• SPB loop 1 to be deployed today. Working on loop 2

KARABO

• Slow response appears to be solved
• There are other not so severe bugs on the backlog
• Currently all high priority components are supported
• No requires cleanup and day-one component list
• Monitor stability and robustness

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Fix the issue with the disconnected vacuum loop after tunnel reopens

Tasks for the weekend:

• Relax and look for Easter Eggs
• AE and CAS will provide contacts on a good will/best effort basis in case an intervention should be necessary

The EtherCAT loop for the vacuum PLC is interrupted at crate 1.6 (XTD2).

This triggered all Beckhoff Terminals  default behaviour from this point on (i.e. Valves Closed, Digital Outputs low, etc.).

 Tunnel Closed.

Mirrors:

Status:

• No information on status of mirrors and PBLM since Optics group was not at the meeting

Vacuum Loop:

Status:

• All vacuum sections were successfully tested in XTD2 yesterday (work ended very late)
• DPS was tested successfully and all interlocks and remote controls were working
• The protection switch on the ion pumps can still not be actuated remotely 
• XGMs in XTD2 and XTD9 fully in the interlock 
• V0 valve and absorber were moved and interlock tested. Communication with DESY was also tested and is ok.

EPS Loop:

Status:

• No explicit information today (see remark above)

Motion Loop:

Status:

• No explicit information today (see remark above)

Imagers

Status:

• All imagers in XTD2 are ok
• Trigger is not working on one of three imagers. 
• Testing proceeded in XTD9 yesterday
• One camera in one pop-in can not be reached (Network problem?)

Karabo:

Status:

• Running stably
• Work required now:
• Photonics Camera driver
• Clean up the project database
• Clean up master equipment list

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Proceed with remote testing
• Check out adequacy of scenes
• Monitor stability and responsiveness of GUI Server and Log Servers 
• Set up monitoring of temperatures and pressures overnight (A. Silenzi)
• Cleanup Karabo

Tasks for tomorrow:

• Tunnels should re-open tomorrow
• Continue tests from remote and check stability and robustness of systems

Other topics

• Tunnels open over the Easter weekend and also during the day on Tuesday 18 April
• There will be a SASE Readiness Meeting at 09:00 on Thursday in XHQ E1.173

Imager Status, 12.4.17

-          All imagers tested, motors and cameras, motion loop, EPS loop, for 3 imagers XTD2, and 3 imagers XTD9.

-          All blocking issues concerning tunnel closure solved.

-          Remaining issues:

o   Camera trigger, tbd. for 4 imagers.

o   Camera driver for SR imager, Photonic Science, tbd.

o   Camera of sa1-xtd9-imgpii45-cam-beamview not reachable via IP address, tbd..

o   Camera display is now at around 2 Hz, not yet full speed.

o   Store image function, tbd.

o   Firmware update to operate braking switches for homing procedure, tbd.

Both SASE1 XGMs (in XTD2 and XTD9) are now again in the Vacuum Control Loop, their vacuum being controlled by Beckhoff/karabo. This works fine.

Therefore: DO NOT FOOL AROUND WITH THE VACUUM LOOP !

Since we all cannot enter the tunnel XTD2 in the next two days, it's agreed that the Vacuum Loop is NOT going to be touched at all, since  we cannot intervene in the tunnel if something goes wrong.

 The Interlock conditions for these two components has been positively tested but not exhaustively done:

The triggering of both via the actuation on the Fast Valve----------------->DONE and OK

Their triggering via HV down of each of the 6 ion pumps------------------>Partially DONE and OK: Partially means that only one of the pumps randomly chosen has been deactivated. The agreement is that the other remaining 5 can be done later, also from the office since the correspondence between the devices and the physical components has been confirmed. 

Also, the individual actuation of both, Beam Stopper and V0 valve has been proof showing a consistent behaviour regarding the interlock definition.

Finally the XFEL Vacuum OK signal is available but AE is currently improving it. By the moment is not automatically sent to DESY, but instead it needs to be activated manually. According to AE feedback this should be corrected in the following days.

 The Fast Valve test has been positively tested. 

It has been consider that the signal on the EPS to open again, should be also available in the vacuum loop. Still this need to be further discussed.

 The interlock definition for complete isolation of the whole section based on high pressure signal has been successfully tested. 

DPS (Upstream and Downstream) are now fully "controlled" by Karabo.

XGM has been reported by J. Grünert that it will be re-integrated during the evening.

Still some problems to call the remote operation of the turbopumps from Karabo, but it doesn´t compromise any of the interlock definitions (only is troublesome if the one needs to work with the pump, for instance after the interlock has been triggered).

Dear all,

The EPS loop would be stopped, in order to correct some issues with the hardware in Rack 1.1 and Rack 1.5 in XTD2. We will upload it as soon as possible and I will post a new entry here when it is running again.

Thank you,

Both XGMs were and are on manual control (but the remote cables into the turbo pump controllers remained plugged in as usual) = both XGMs are "out of the VAC loop". 

In XTD2, I saw around 13h that both turbos were off (0 Hz) and the pressure was at 10E-4 mbar !

The green light above the ON-button was flickering. 
I switched parameter 023 (Motor TMP) to ON and restarted both pumps. The green light above the ON-button 
 stops flickering and goes steady green. 
 
---going to XTD9 
 
In XTD9, I saw that the upstream turbo was off (0 Hz) and the pressure was in the lower 10E-6mbar. 
The downstream turbo in XTD9 was still operating. 
Parameter 023 to ON, and restart pump. 
 
---going to XTD2 
 
Error codes 6, 6, 98, 2, 37 on one pump, error codes 6,6,6 on the other (XTD2).
I unplugged the REMOTE cables in XTD2 and XTD9 to avoid further trouble as long as we are on "manual". 
This info was also sent by email to the involved experts.

Please find in attachment the dowload bash script for the latest Karabo.

if your system has a "karabo" directory, this script will not overwrite it, but the update will fail.
In this case, remove or rename the karabo directory.

To use it:
1. download the script in a directory of your choice.

2. open a terminal and execute it with
   $ bash PATH_TO/download.sh

Where PATH_TO is the path where the download script was saved.

Dear all,

Sorry for disturbing again, I will do a small update to put one missing coupler in the hardware configuration in rack 1.18, the loop would be running again in few minutes.

Thank you!

Dear all,

Motion loop is running again with the last update (names for PSLIT and LVDT have been changed).

Thank you,

Tunnel Open.

Mirrors:

Status:

• Vacuum on M2 should be good enough to switch on ion pumps
• Surveying work going on 

Vacuum Loop:

Status:

• Several vacuum sections successfully tested in XTD9 yesterday
• Tests need to proceed in XTD2 today
• The protection switch on the ion pumps cannot be actuated remotely 
• Loop needs an update for the XGM
• XGM was in the vacuum interlock overnight. Will be taken out during the day but should be put back at the end of the day
• Interlock of High Voltage with vacuum in XGM not working yet
• V0 valve needs to be exercised in collaboration with DESY 

EPS Loop:

Status:

• The terminal exchanged for the FEL imager was not the broken one. Another terminal needs to be exchanged today
• The MCP needs to be moved out of the beam manually. This will be checked using the EPS switch. For this the EPS loop is needed

Motion Loop:

Status:

• An improved fix for the motors not stopping was deployed yesterday
• The loop needs to be reconfigured to replace the SRA motors 
• One broken motor was exchanged on the mirror chambers and needs to be tested

Imagers

Status:

• All motion tests in XTD2 done. Now continue in XTD9
• Photonics camera driver still not working but not critical for tunnel closure (there is also a workaround by the name of ebus player)
• EPS switch on FEL imager needs testing  after terminal replacement

Karabo:

Status:

• New version with TCP was deployed. Experience overnight is good

Tasks for today:

Note: Loop reconfiguration and updates of Karabo to be done in the morning. Testing can resume after lunch. Status updates should please be recorded in the log

• Reconfigure vacuum, motion loop (updates for XGM interlock, SRA motors)
• Replace broken terminal in EPS loop for FEL imager 
• Move MCP out of the beam manually and check using EPS
• Continue systematic test of the vacuum components (J. Eidam, V. Bondar)
• Test EPS switch on FEL imager after terminal exchange (A. Koch and W. Ehsan)
• Monitor stability and responsiveness of GUI Server and Log Servers (A. Silenzi)
• Set up monitoring of temperatures and pressures overnight (A. Silenzi)
• Test V0 and absorber

Tasks for tomorrow:

• Tunnels closed
• All loops should run stable
• No or only minimal interference with Karabo
• Continue tests from remote and check stability and robustness of systems

Other topics

• Tunnels open over the Easter weekend and also during the day on Tuesday 18 April
• There will be one more standup 09:30 meeting tomorrow (Wednesday)
• There will be a SASE Readiness Meeting at 09:00 on Thursday