As soon we received the request we worked on that one.  After analysing the issue proper response was sent.

​​​----- Forwarded Message -----

​​​​From: "Wajid Ehsan" <wajid.ehsan@xfel.eu>

To: "Raul Villanueva" <raul.villanueva@xfel.eu>

Cc: "Alessandro Silenzi" <alessandro.silenzi@xfel.eu>, "Nerea Jardon" <nerea.jardon@xfel.eu>

Sent: Friday, 21 April, 2017 12:56:14

Subject: Re: Karabo problems in XTD9 - Urgent!

​​​​​Hi Rahul,

we checked all the devices that are in error. The issue is realted with hardware.

Error description is mentioned below:

0x1001 FlagCtrl Error flag from controller

Regards,

Wajid Ehsan 

 Due to problems and lack of support of CAS for accessing the devices in XTD9 tunnel through KARABO we couldn´t run the test. The system has been returned to manual operation and will remain like that until next notice.

XGM status 21.4.2017

During Easter break, the two XGMs in the control system (XTD2+9) had no issues.
XGM@XTD10 however had an issue: tuesday morning it was found that all pumps had stopped. The scroll pump showed an error. UPS apparently had excess load sunday evening 19h30 but couldn't send email (network error). Possibly vented with tunnel air up to 10mbar since that time. Investigation is ongoing.

Thursday enabled DOOCS pressure reading of XGM in XTD10, UPS rechecked

System surveillance on coming weekend: Jan and Theo 1 day

Other remarks:
- WP74 staff have received their dosimeters for ZZ access if required
- some automatic monitoring and info system is needed soon, e.g. karabo to NAGIOS connection

 In KW17 there will be NO maintenance day on Tuesday (25.04.) 

 
On Monday (24.04.) morning up to about 11:00 we will have the TÜV approval of CS8. During that time 
 NO access of "other people" to the XTL tunnel is possible. After the TÜV approval 
 up to 13:00 there will be the possibility for URGENT access to the XTL. Access requests 
 have to written to the logbook/access requests entry. 
 
The tunnel XTL/XTIN will stay in interlock, access will be possible by ZZ only. 
 
The northern branch XTD2->XS3->XTD4->XDU2 will stay closed, in order not to interrupt the 
 magnet commissioning. 
 
This elogbook entry was sent to following experts: 
xfel-opl@desy.de RC mailing list  

 Tunnels XTD2 and XTD4 are closed, XTD9/10 remain open

Vacuum:

• PLC/Karabo Running stably
• Local tests of ion pumps in XTD9 being done by J. Eidam
• Work continues on M3 in XTD9 and on the vacuum in XTD10 
• DPS in XTD9 should be integrated into the vacuum loop. This requires a cycling of the loop. Timing should be coordinated with WP74 because of XGM in XTD2. Unfortunately nobody from WP74 attended the meeting. Hence it was decided to perform the work tomorrow morning.

EPS Loop:

• Checks for the signal from the undulators and for the fast valve will be done with DESY MPS today

PBLM/Motion Loop:

• Work on the PBLM motors (ELMO controller) continues.
• Yesterday it was possible to align the screens 
• There is a resource problem when trying to read out the cameras. For this reason an additional server needs to be set up dedicated to acquiring the images from the cameras.

SPB Tunnel components

• Crate that was checked in the lab yesterday was moved back into the tunnel
• The unstable support of the imager was fixed
• B. Weinhausen and M. Messerschmidt are testing
• All modules necessary for loops 1 & 2 were received from Poland
• Waiting for the "Freigabe" of the cables for SPB Optics

FXE

• Loops 1,2 and 4 are waiting for input from EETF
• Loops 3&5 are running
• No visible testing activity from FXE team

Karabo:

• Cleanup continues

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Prepare mirror M3 installation
• Continue testing of PBLMs in XTD9
• Continue local tests of ion pumps in XTD9
• Continue testing of SPB tunnel loop

Tasks for tomorrow:

• Cycle vacuum loop for inclusion of DPS in XTD9

Other topics

• Weekly SASE1 Readiness meeting tomorrow at 09:00 in XHQ 1.173
• L. Samoylova requests access to XTD2 to install another webcam
• Magnet current tests will likely continue through Wednesday in XTD2. Maintenance day to be shifted accordingly probably to Thursday/Friday. More information later

Tunnels XTD2 and XTD4 are closed, XTD9/10 remain open

Vacuum:

• Running stably
• Work continues on M3 in XTD9 and on the vacuum in XTD10 

EPS Loop:

• Input from undulator still missing. This requires  fixing a cable in balcony rooms and then testing

PBLM/Motion Loop:

• Work on the PBLM motors (ELMO controller) continues.
• It appears that the uneven movement is due to improper setting of motor parameters

Imagers

• Trigger problem reported yesterday was due to a cabling issue. Apparently during work on the relevant rack cables were unplugged and then wrongly put back
• Network issue with pop-in (245) in XTD9 solved

SPB Tunnel components

• Crate with "problem on signal pins" was brought to the lab and checked. Everything was ok. Problem was due to spurious measurement.

Karabo:

• PSLIT was tested successfully yesterday
• Behaviour of Karabo when a loop goes down needs to be made more graceful. 
• Cleanup continues

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Prepare mirror M3 installation
• Continue testing of PBLMs in XTD9
• Continue testing of SPB tunnel loop

Other topics

• None
   

Dear all,

Today from 13:30h  Mahadi and Natalia will be doing local tests in the PBLM in Rack 1.14 of XTD9, therefore, if someone will be working in the following racks of XTD9, will loose connection to the Motion loop.

Thank you and best regards,

All trigger problems in XTD2 and XTD9 solved except for one: spb_xtd9_imgpi, probably a cabling issue on the camera side.

IP address problem for camera of sa1_xd2_imgpii45 solved.

 Author: Decking

Goals:
- Operation with 10 GeV, 500 pC, 5 kA, 30 bunch
- Increase gradient limits in RF stations
- Technical comissioning of north branch 

XTIN closed all week
XTL open till Tue, 11:00
XTD2 open till Tue, 15:00
XTD4 open till Tue, 15:00
XTD9&XTD10 open all week


|17/04 Mo M |  Easter shutdown
|17/04 Mo A |  Easter shutdown
|17/04 Mo N |  Easter shutdown
|18/04 Tu M |  Interlock pre-test CS8, set XTL interlock at 11:00, Injector beam operation
|18/04 Tu A |  A2 calibration after phase shifter change, Beam operation  to TLD 10 GeV, set interlock XTD2-XTD4
|18/04 Tu N |  Beam operation  to TLD 10 GeV, 30 bunches, 0.5 nC
|19/04 We M |  Slice emittance in BC2; Magnet PS test XTD2-XTD4 
|19/04 We A |  BCM, BAM EO, commissioning; Magnet PS test XTD2-XTD4 
|19/04 We N |  BCM, BAM EO, commissioning; Magnet PS test XTD2-XTD4 
|20/04 Th M |  Collimation set-up; Magnet PS test XTD2-XTD4 
|20/04 Th A |  BCM, BAM EO, commissioning; Magnet PS test XTD2-XTD4 
|20/04 Th N |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC; Magnet PS test XTD2-XTD4 
|21/04 Fr M |  Collimation set-up with 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|21/04 Fr A |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|21/04 Fr N |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|22/04 Sa M |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|22/04 Sa A |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|22/04 Sa N |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|23/04 Su M |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|23/04 Su A |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|23/04 Su N |  Beam operation  to TLD 12 GeV, 30 bunches, 0.5 nC, 5 kA; Magnet PS test XTD2-XTD4 
|24/04 Mo M |  Beam set-up 12 GeV, 1 bunches, 0.1 nC, 1 Hz
|24/04 Mo A |  Beam operation  to TLD 12 GeV, 1 bunches, 0.1 nC, 1 Hz
|24/04 Mo N |  Beam operation  to TLD 12 GeV, 1 bunches, 0.1 nC, 1 Hz

XTD Tunnels open until 15:00. Later XTD2 and XTD4 are closed, XTD9/10 remain open

Vacuum Loop:

Status:

• One broken Beckhoff terminal was exchanged before the weekend. This was the first failure of this type in 2 years
• After this the system ran without hickup over the Easter weekend
• The fix applied for ion pumps that avoids triggering an interlock condition when there is a communication failure on the serial output should be also applied now to other vacuum device, e. g. the turbo pumps

EPS Loop:

Status:

• Requires an update to be done asap so that N. Gerassimova can continue tests with the PBLMs afterwards

Motion Loop:

Status:

• Movement of the PBLM motors (ELMO controller) is not satisfactory yet. Motors start and stop but the movement is uneven and stops not always requested
• There is no test set up in the lab. So tests have to be done in the tunnels (XTD9)

PBLMs:

• All systems in XTD2 were tested and cameras calibrated including some at the beginning of XTD9
• Work continues on remaining systems in XTD9
• System with motor movement is still unsatisfactory (see above)

Imagers

Status:

• 2 Imagers in XTD2 have no trigger signal. Suspicion is that this might be configuration problem. P. Gessler will investigate
• One pop-in (245) in XTD9 cannot be reached with IP. Suspicion is incorrect network configuration -> K. Fillipakopoulos

SPB Tunnel components

• Signal pins of the Attenuator crate were tested. DC voltage is seen there at different levels on different pins. This looks problematic -> P. Gessler

Karabo:

Status:

• Ran stably over the weekend 

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Take equipment and junk out of tunnels
• Check trigger cabling for imagers in XTD2
• Check PBLM motor movement and continue testing in XTD9

Other topics

• Controlled access to tunnels is possible after 1500. Coordination with T. Haas and the DESY run coordination

Dear all,

The EPS loop is running again with the last modifications concerning devices' names.

Thank you.

Dear all,

The Motion loop would be updated now in order to fix names problems with PSLITs motors. In few minutes would be reachable again.

Than you!

The first EtherCAT coupler in 1.7 VAC was broken and is replaced now. The loop is up and running again.

Cheers,

    Tobias

All northern tunnels XTD2/4/9/10 are fully accessible again! 

Radiation Interlock Tests are finished. Tunnels XTD9 and XTD10 up to the fence at the dump can be accessed right now. Magnet current interlock tests still takes a bit of time (ca. 1h). Once this is also done, I shall post a note 

 Tunnel Closed until approx 1300, Open over the Easter Weekend until Tuesday 1500.

Vacuum:

• All valves and pumps in XTD2 are working and can be operated remotely
• The vacuum system including DPS and XGM is interlocked. Interlocks have been tested and are working properly
• The fast valve and the absorber at the interface to the electron beam vacuum are working and interlocked. The communication with DESY has been successfully tested
• Some issue exist when switching turbo pumps on/off in Karabo. Then erratic behaviour is observed. Doing the same remotely communicating via TCP directly works ok
• More testing needs to be done in XTD9

Mirrors, PBLMs, SRA, Attenuator

• M1 and M2 are installed and under vacuum. M3 will be installed after easter
• 3 out of 5 motors can be operated in M1 and M2. The mirrors have been moved into a safe position
• PBLM motors (ELMO driver) can be operated (started and stopped) but operation is still problematic (motors stop in wrong position, movement not linear)
• On the SRA the connection to two motors is interchanged
• On the attenuator certain motors occasionally get stuck
• Slits have been moved out so the beam line is safe for first operation

Imagers:

•  
• All imagers tested, motors and cameras, motion loop, EPS loop, for 3 imagers XTD2, and 3 imagers XTD9.
• All blocking issues concerning tunnel closure solved.
• Some remaining points that can be treated remotely: Trigger, software delays, others.
• The Photonics Camera driver is now working in Karabo.
• Cameras can currently be read out at 3 Hz which is too slow but ok for first beam commissioning

XGM:

•  
• XGM inserted into vacuum loop and interlock (in XTD2 and XTD9)
•  
• Grounding completed in XTD2+XTD9 (power cycle of rack was needed)
• Gases ready to be used in XTD2
• XGM @ XTD2 ready for beam         (need DOOCS middle-layer servers activated) 

FXE

• Testing finished for all devices in loop1
• Now waiting for loops 2 and beyond
•  

SPB

• SPB tunnel loop deployed. Testing can start next week
• For the Optics hutch some Beckhoff crates are still missing 
•  

PLC

• Fixes to MC2 library (ELMO and TECHNOSOFT motors) will deployed next week and then tested in XTD9
• FXE loop 2 was modified and loops 3&4 are deployed
• SPB loop 1 to be deployed today. Working on loop 2

KARABO

• Slow response appears to be solved
• There are other not so severe bugs on the backlog
• Currently all high priority components are supported
• No requires cleanup and day-one component list
• Monitor stability and robustness

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Fix the issue with the disconnected vacuum loop after tunnel reopens

Tasks for the weekend:

• Relax and look for Easter Eggs
• AE and CAS will provide contacts on a good will/best effort basis in case an intervention should be necessary

The EtherCAT loop for the vacuum PLC is interrupted at crate 1.6 (XTD2).

This triggered all Beckhoff Terminals  default behaviour from this point on (i.e. Valves Closed, Digital Outputs low, etc.).

 Tunnel Closed.

Mirrors:

Status:

• No information on status of mirrors and PBLM since Optics group was not at the meeting

Vacuum Loop:

Status:

• All vacuum sections were successfully tested in XTD2 yesterday (work ended very late)
• DPS was tested successfully and all interlocks and remote controls were working
• The protection switch on the ion pumps can still not be actuated remotely 
• XGMs in XTD2 and XTD9 fully in the interlock 
• V0 valve and absorber were moved and interlock tested. Communication with DESY was also tested and is ok.

EPS Loop:

Status:

• No explicit information today (see remark above)

Motion Loop:

Status:

• No explicit information today (see remark above)

Imagers

Status:

• All imagers in XTD2 are ok
• Trigger is not working on one of three imagers. 
• Testing proceeded in XTD9 yesterday
• One camera in one pop-in can not be reached (Network problem?)

Karabo:

Status:

• Running stably
• Work required now:
• Photonics Camera driver
• Clean up the project database
• Clean up master equipment list

Tasks for today:

Note: All loops should be left running and no changes to Karabo.

• Proceed with remote testing
• Check out adequacy of scenes
• Monitor stability and responsiveness of GUI Server and Log Servers 
• Set up monitoring of temperatures and pressures overnight (A. Silenzi)
• Cleanup Karabo

Tasks for tomorrow:

• Tunnels should re-open tomorrow
• Continue tests from remote and check stability and robustness of systems

Other topics

• Tunnels open over the Easter weekend and also during the day on Tuesday 18 April
• There will be a SASE Readiness Meeting at 09:00 on Thursday in XHQ E1.173

Imager Status, 12.4.17

-          All imagers tested, motors and cameras, motion loop, EPS loop, for 3 imagers XTD2, and 3 imagers XTD9.

-          All blocking issues concerning tunnel closure solved.

-          Remaining issues:

o   Camera trigger, tbd. for 4 imagers.

o   Camera driver for SR imager, Photonic Science, tbd.

o   Camera of sa1-xtd9-imgpii45-cam-beamview not reachable via IP address, tbd..

o   Camera display is now at around 2 Hz, not yet full speed.

o   Store image function, tbd.

o   Firmware update to operate braking switches for homing procedure, tbd.

Both SASE1 XGMs (in XTD2 and XTD9) are now again in the Vacuum Control Loop, their vacuum being controlled by Beckhoff/karabo. This works fine.

Therefore: DO NOT FOOL AROUND WITH THE VACUUM LOOP !

Since we all cannot enter the tunnel XTD2 in the next two days, it's agreed that the Vacuum Loop is NOT going to be touched at all, since  we cannot intervene in the tunnel if something goes wrong.