WP74   OCD  for the coming weekend 29.4.-1.5.2017:

Saturday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Sunday: Jan, jan.gruenert@xfel.eu, 9-5777

Monday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Note: Theo has Torbens work phone this weekend since Theo has not yet his own mobile work phone

 Dear All, 

I would like to share the CAS members on the OCD from 29/04 to 01/05:

• Saturday 08:00 to Sunday 08:00 - Valerii - Phone: 95471
• Sunday 08:00 to Monday 08:00 - Gabriele - Phone: 96894
• Monday 08:00 to Tuesday 08:00 - Wajid - Phone: 95630

Best,

Hugo

AE on call service:

    Contact person:

        Tobias Freyermuth

    Phone:

        9 - 6822

Note that plans for next week (points 2& 3) should refer to XTD9 and *NOT* XTD2

 Plans for next week:

• Consolidate vacuum in XTD2 so that V0 can be reliably opened when first beam should be transported
• Finish work in XTD9, in particular PBLMs and instrument components
• Finish systematic tests of ion pumps in XTD9
• Deploy fix for Renishaw encoders in firmware
•  

 Tunnel Status (T. Haas):

• Tunnels XTD2/4/10 were interlocked yesterday afternoon
• Beam is already going through the T2 and T4 beam lines to T4D
• XTD9 remains open until approximately 15 May

Vacuum (J. Eidam, R. Villanueva):

• The interlock in the gas section was tested successfully in XTD9. The interlock does currently not include the gas injection. This still needs setting up and testing
• Systematic testing of ion pumps and interlocks is ongoing in XTD9 but needs to continue next week
• The system generally works reliably
• However, there was an incident this week in which the gas sections in XTD2 and XTD9 went completely off. The reason is still not understood

Mirrors and PBLMs (H. Sinn):

• All three mirrors in the SASE1 beam line in XTD2/9 are installed and pumped
• N. Gerassimova installed PMTs in XTD2 yesterday and set up the cameras of the PBLMs
• The problem with injected properties disappearing was identified on the Karabo side and fixed. The fix still needs to be deployed
• The motors of the PBLMs are still not functioning completely. 
• The read out rate of the cameras is ok (of the order of 10Hz or better)

Imagers (A. Koch):

• Imagers in XTD2 and XTD9 are ready for beam
• Work continues on improvements
• Read out rate is ok, but the latency is sometimes high and needs to be unterstood

XGM (J. Grünert):

• The gas on the XGM in XTD2 was changed at the request of K. Tiedtke from Ar to Ne
• Tests were done with the remote control of the gas system. This is now fully functional
• The HV/Vacuum interlock is not operational yet
• Otherwise the XGM is ready accept beam. Next week software tests should be performed with the readout

FXE (A. Galler):

• Currently commissioning loop 1. There is still a problem reading out encoders. It appears that this is a firmware issue. TF is working on this.

SPB (B Weinhausen):

• Currently commissioning the SPB tunnel components. The encoders have the same problem as the one reported by FXE
• I broken terminal required changing

PLC (T. Freyermuth):

• CRL in XTD2 was integrated into the PLC motion loop yesterday during the access.
• A first version of the EPS logic was agreed upon and implemented. Details to be discussed again with DESY next week

Karabo (A. Silenzi):

• ITDM provided an additional control server
• A redeployment of Karabo is necessary and will be done later today (users will need to re-connect afterwards) 
• The problems previously observed with the GUI server have not re-occured. 

Plans for next week:

• Consolidate vacuum in XTD2 so that V0 can be reliably opened when first beam should be transported
• Finish work in XTD2, in particular PBLMs and instrument components
• Finish systematic tests of ion pumps in XTD2
• Deploy fix for Renishaw encoders in firmware

Other topics:

• We continue the 09:30 meetings next week until further notice

 ACTEMIUM has delivered the handover document for the phase II cables installed in D,02 and D.13. The beckhoff crates can be connected to the cables in these areas of SASE1.

Dear all,
motion loop is up and all Karabo devices in it are running.

Thank you.

27/04	Th A 
Beam operation to TLD 10 GeV, 1 bunches, 0.1 nC, 1 Hz; interlock 
set XTD2-XTD4
27/04	Th N 
Beam operation to TLD 10 GeV, 1 bunches, 0.1 nC, 1 Hz, EOD shift
28/04	Fr M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
28/04	Fr A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
28/04	Fr N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz 
29/04	Sa M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
29/04	Sa A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
29/04	Sa N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz 
01/05	Mo M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
01/05	Mo A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
01/05	Mo N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
 
Access to TLD:  

 XTDs Info:

Tunnel XTD2/4 will be accessible today until 1500. Dosimeter and General radiation training required. 

XTD10 will be closed starting this afternoon

Vacuum:

• DPS with interlocks was tested successfully yesterday evening but...
• ... Complete failure of gas sections in both XTD and XTD9 some time tonight. Valves were closed and turbo pumps were off. Reason is still being investigated
•  
• Systematic tests of ion pumps in XTD9 done for 3 vacuum sections. Currently only the pumps are tested. Tests of the interlock will follow
   

Karabo:

• The problem with the GUI server has unfortunately not re-occured. Hence it can currently not be debugged.
• Second GUI server is available for fall back
• Remote reset of TP controllers has been implemented and will be deployed in the next release

PLC

• A problem was found in the MC2 library affecting the ELMO controller.  This only affected the systems deployed in the tunnel. 
• The non-linear behaviour of the PBLM motors was reproduced independently of PLC and KARABO. Suggestion: Could this happen when the motor has to work against vacuum pressure? Evidence: It does not occur in the lab (no vacuum!)
• A video session with the manufacturer is schedule for next week to look at the problem

PBLM/Motion Loop: 

• There is a resource problem when trying to read out the cameras. This problem will be reconsidered once the general issue with the GUI server is solved.
• ASilenzi will discuss offilen with Natalia the acceptance test definition for PBLM
• List of errors and relative solution concerning ELMO absolute encoder implementation being worked out by AE/CAS

EPS

• Logic of EPS was agreed between AE and H. Sinn:
  • Currently only covers devices up to XS3
  • Vacuum ok requires all valves to be out of the way
  • Any screen down  forces single bunch mode
  • All imagers in safe position allows full beam mode
• Meeting next week @ DESY to discuss technical details with MPS group

SPB Tunnel components

• Renishaw encoder can be read out reliably after fix for error signal line
• SPB would like to work on the pulse picker next and is waiting for it to be integrated

FXE

• FXE guys would like to help with tests of Hard X-Ray Mono 

Tasks for today:

• Perform various maintenance tasks in XTD2, e. g. integrate CRLs, exchange photo tubes, exchange gases
• Investigate why gas sections switched off yesterday
• Continue test of vacuum components in XTD9

Hi all,

in order to integrate the CRL in XTD2 I am going to update the motion loop.

I will let you know as soon as this is done.

Cheers,

    Tobias

 Dear all,

I just checked the Karabo Scene for both XTD9 and XTD9, finding out the not-so-nice situation where all the valves and pumps were respectively closed and stopped. What really worries me is again the fact that these situations are happening without aparent reason. I´m right now trying to start the system remotely but obviously one needs again to access the tunnel for that is the only reason to actually be sure that everything is up an running normally again.

 UPDATE:

XTD2

The Upstream sector is back into operation.

The Downstream one is still out of order since the turbopumps are whether in error or in an unknown/un-instantiated status.

XTD9

Upstream sector is back to full operation

Downstream sector: only one of the turbos is running. The other one P11280T cannot be reset

Right now that is the only thing I can do.

 Dear all,

Finally the tests were run with the expected results: Pressure thresholds for the 3 involved gauges were forced on the system and the interlock on the two gate valves that limit the whole gas section were automatically closed in every case. Also the release switch was tested and the valves were able to be normally operated again.

However the interlock definition also involves the closing of the so-called "shut-off" valve in the gas injection line at the XGM. Unfortunately its behaviour couldn´t be completely observed since the Karabo device is not fully implemented. On the other hand, J. Grünert reported that after the interlock was triggered the valve was shown as also "interlocked" (despite the actual status, open or closed was not clear). This situation may require a second test, combined for instance with the gas injection test to be run by WP74 in the upcoming date). 

In any case the final results were rather positive. (despite there still some permanent uncertainty about devices being reachable or not through Karabo, as it has been previously reported).

Thanks to everyone involved.

Raúl Villanueva on behalf of B. Rio, N. Jardon and V. Bondar.

Since around 17h30 all XTD9 VAC loop is in error. It came back in karabo around 17h50. XTD2 not affected.

At that point, we had to reset every single karabo device of the XGM, then restart pumps etc.

Did the same also for the DPS upstream and downstream of XGM@XTD9.

XGM vatterfly valves will be opened by me only when I see that no further errors occur.

Currently, all but one pumps were restarted, also in Rauls DPS, only one of these not

Dear all,

I will do an update into the Motion loop. Would take only few minutes and the PLC will be running again.

Thank you and sorry

I added the requests that I was made aware of to the list of access requests for tomorrow. While the interlock in XTD2 will be broken and hence the tunnel is accessible without special procedures, please note the followoing

• Please make sure your request in in the DESY table of access requests 
• You have valid access permissions and two (!) dosimeters, one from XFEL one from DESY

 The XSDU2  area will be open (as XTD2) tomorrow between 9 ans 12. To register the access, please send also in this case a mail to T.Haas.

Dear all,

I will do an update of the motion loop so the PBLM could be tested. I will do it know and the PLC will be again available in few minutes.

Thank you!

Problems described in detail below (this entire procedure takes way too much of our time, effectively still nearly 2 hours!):

Nerea/AE requested yesterday afternoon to shutdown and restart the SASE1 VAC PLC loop once again. We start today morning (wednesday) at 8h30. 
8h30
karabo is not showing status of all devices, see three screenshots (attached). karabo-GUI issue.
These devices have to be shutdown and reinstantiated to show a status and to operate them. 
Several valves and gauges and scroll pumps affected both in XGM of XTD2 and XTD9. 
 
9h00 finally I can close all four vatterfly valves 
 
9h15 Nerea shuts down and restarts the SASE1 VAC PLC loop 
 
9h25 still waiting for karabo to reconnect to Beckhoff PLC (I cannot do anything in karabo, and 
 no current status updates of anything) 
 
9h40 SPB_XTD9_XGM scene has loaded (seems like at least). I start the reactivation procedure (switch 
 on scroll, open valves etc.). Need to reinstantiate several devices. 
 
10h00 SPB_XTD9_XGM: reinstantiate both turbo pumps, CRemotePriority ON, COn, CPump (if CRemotePriority 
 not visible, again reinstantiate) 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h05 SA1_XTD2_XGM: reinstantiate both turbo pumps, StartPumpingStation 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h10 READ the actual value of the gauges in configurator (not to trust value shown in scene) 
10h15 observe DOOCS RVC300 UHV pressures, open vatterfly valves 
 
10h20 end of intervention 

Vacuum is running again and Motion also, yesterday I forgot to update you about the Motion PLC.

Sorry and best regards,