All devices went into unknown state in the karabo XGM scenes in XTD2 and XTD9 at 19h52.

Pressures INCREASE since then (as measured/shown in DOOCS):

XTD2:  2,5E-7mbar and rising

XTD9:  7,7E-7mbar and rising

Please respond now !! We are still in but want to go home !

 This Thursday the XTDs will be accessible with ZZ procedure. Access can be requested via T. Haas. 

XTD2: All gate valves are now OK (not interlocked) after re-instantiating and re-setting the XGM_DP_OK. 

The interlock errors of some valves in XTD9 are shown in the screen shots below (Janni and Valerii starting now in XTD9 to debug some more controllers)

Hi all,

due to ongoing tests with the PBLM motor drivers I have to set the Elmo drivers in PreOp mode.

That should not affect any other terminal in the loop.

Cheers,

    Tobias

 Dear All,

Still in time, I would like to correct out schedule.

The scheduled previous agreed for this weekend/holiday was from 10:00 to 17:00. Therefore,

• Saturday 10:00 to 17:00 - Valerii - Phone: 95471
• Sunday 10:00 to 17:00 - Gabriele - Phone: 96894
• Monday 10:00 to 17:00 - Wajid - Phone: 95630

Best,

Hugo

Not really official, but in emergency case you can call 96993 for vacuum group. Next week Martin is back and I think we will have official on call service.

Have a nice weekend with hopefully no problems :-)

WP74   OCD  for the coming weekend 29.4.-1.5.2017:

Saturday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Sunday: Jan, jan.gruenert@xfel.eu, 9-5777

Monday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Note: Theo has Torbens work phone this weekend since Theo has not yet his own mobile work phone

 Dear All, 

I would like to share the CAS members on the OCD from 29/04 to 01/05:

• Saturday 08:00 to Sunday 08:00 - Valerii - Phone: 95471
• Sunday 08:00 to Monday 08:00 - Gabriele - Phone: 96894
• Monday 08:00 to Tuesday 08:00 - Wajid - Phone: 95630

Best,

Hugo

AE on call service:

    Contact person:

        Tobias Freyermuth

    Phone:

        9 - 6822

Note that plans for next week (points 2& 3) should refer to XTD9 and *NOT* XTD2

 Plans for next week:

• Consolidate vacuum in XTD2 so that V0 can be reliably opened when first beam should be transported
• Finish work in XTD9, in particular PBLMs and instrument components
• Finish systematic tests of ion pumps in XTD9
• Deploy fix for Renishaw encoders in firmware
•  

 Tunnel Status (T. Haas):

• Tunnels XTD2/4/10 were interlocked yesterday afternoon
• Beam is already going through the T2 and T4 beam lines to T4D
• XTD9 remains open until approximately 15 May

Vacuum (J. Eidam, R. Villanueva):

• The interlock in the gas section was tested successfully in XTD9. The interlock does currently not include the gas injection. This still needs setting up and testing
• Systematic testing of ion pumps and interlocks is ongoing in XTD9 but needs to continue next week
• The system generally works reliably
• However, there was an incident this week in which the gas sections in XTD2 and XTD9 went completely off. The reason is still not understood

Mirrors and PBLMs (H. Sinn):

• All three mirrors in the SASE1 beam line in XTD2/9 are installed and pumped
• N. Gerassimova installed PMTs in XTD2 yesterday and set up the cameras of the PBLMs
• The problem with injected properties disappearing was identified on the Karabo side and fixed. The fix still needs to be deployed
• The motors of the PBLMs are still not functioning completely. 
• The read out rate of the cameras is ok (of the order of 10Hz or better)

Imagers (A. Koch):

• Imagers in XTD2 and XTD9 are ready for beam
• Work continues on improvements
• Read out rate is ok, but the latency is sometimes high and needs to be unterstood

XGM (J. Grünert):

• The gas on the XGM in XTD2 was changed at the request of K. Tiedtke from Ar to Ne
• Tests were done with the remote control of the gas system. This is now fully functional
• The HV/Vacuum interlock is not operational yet
• Otherwise the XGM is ready accept beam. Next week software tests should be performed with the readout

FXE (A. Galler):

• Currently commissioning loop 1. There is still a problem reading out encoders. It appears that this is a firmware issue. TF is working on this.

SPB (B Weinhausen):

• Currently commissioning the SPB tunnel components. The encoders have the same problem as the one reported by FXE
• I broken terminal required changing

PLC (T. Freyermuth):

• CRL in XTD2 was integrated into the PLC motion loop yesterday during the access.
• A first version of the EPS logic was agreed upon and implemented. Details to be discussed again with DESY next week

Karabo (A. Silenzi):

• ITDM provided an additional control server
• A redeployment of Karabo is necessary and will be done later today (users will need to re-connect afterwards) 
• The problems previously observed with the GUI server have not re-occured. 

Plans for next week:

• Consolidate vacuum in XTD2 so that V0 can be reliably opened when first beam should be transported
• Finish work in XTD2, in particular PBLMs and instrument components
• Finish systematic tests of ion pumps in XTD2
• Deploy fix for Renishaw encoders in firmware

Other topics:

• We continue the 09:30 meetings next week until further notice

 ACTEMIUM has delivered the handover document for the phase II cables installed in D,02 and D.13. The beckhoff crates can be connected to the cables in these areas of SASE1.

Dear all,
motion loop is up and all Karabo devices in it are running.

Thank you.

27/04	Th A 
Beam operation to TLD 10 GeV, 1 bunches, 0.1 nC, 1 Hz; interlock 
set XTD2-XTD4
27/04	Th N 
Beam operation to TLD 10 GeV, 1 bunches, 0.1 nC, 1 Hz, EOD shift
28/04	Fr M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
28/04	Fr A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
28/04	Fr N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz 
29/04	Sa M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
29/04	Sa A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
29/04	Sa N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
30/04	Su N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz 
01/05	Mo M 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
01/05	Mo A 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
01/05	Mo N 
Beam operation to T4D 10 GeV, 1 bunches, 0.1 nC, 1 Hz
 
Access to TLD:  

 XTDs Info:

Tunnel XTD2/4 will be accessible today until 1500. Dosimeter and General radiation training required. 

XTD10 will be closed starting this afternoon

Vacuum:

• DPS with interlocks was tested successfully yesterday evening but...
• ... Complete failure of gas sections in both XTD and XTD9 some time tonight. Valves were closed and turbo pumps were off. Reason is still being investigated
•  
• Systematic tests of ion pumps in XTD9 done for 3 vacuum sections. Currently only the pumps are tested. Tests of the interlock will follow
   

Karabo:

• The problem with the GUI server has unfortunately not re-occured. Hence it can currently not be debugged.
• Second GUI server is available for fall back
• Remote reset of TP controllers has been implemented and will be deployed in the next release

PLC

• A problem was found in the MC2 library affecting the ELMO controller.  This only affected the systems deployed in the tunnel. 
• The non-linear behaviour of the PBLM motors was reproduced independently of PLC and KARABO. Suggestion: Could this happen when the motor has to work against vacuum pressure? Evidence: It does not occur in the lab (no vacuum!)
• A video session with the manufacturer is schedule for next week to look at the problem

PBLM/Motion Loop: 

• There is a resource problem when trying to read out the cameras. This problem will be reconsidered once the general issue with the GUI server is solved.
• ASilenzi will discuss offilen with Natalia the acceptance test definition for PBLM
• List of errors and relative solution concerning ELMO absolute encoder implementation being worked out by AE/CAS

EPS

• Logic of EPS was agreed between AE and H. Sinn:
  • Currently only covers devices up to XS3
  • Vacuum ok requires all valves to be out of the way
  • Any screen down  forces single bunch mode
  • All imagers in safe position allows full beam mode
• Meeting next week @ DESY to discuss technical details with MPS group

SPB Tunnel components

• Renishaw encoder can be read out reliably after fix for error signal line
• SPB would like to work on the pulse picker next and is waiting for it to be integrated

FXE

• FXE guys would like to help with tests of Hard X-Ray Mono 

Tasks for today:

• Perform various maintenance tasks in XTD2, e. g. integrate CRLs, exchange photo tubes, exchange gases
• Investigate why gas sections switched off yesterday
• Continue test of vacuum components in XTD9

Hi all,

in order to integrate the CRL in XTD2 I am going to update the motion loop.

I will let you know as soon as this is done.

Cheers,

    Tobias

 UPDATE:

XTD2

The Upstream sector is back into operation.

The Downstream one is still out of order since the turbopumps are whether in error or in an unknown/un-instantiated status.

XTD9

Upstream sector is back to full operation

Downstream sector: only one of the turbos is running. The other one P11280T cannot be reset

Right now that is the only thing I can do.

 Dear all,

I just checked the Karabo Scene for both XTD9 and XTD9, finding out the not-so-nice situation where all the valves and pumps were respectively closed and stopped. What really worries me is again the fact that these situations are happening without aparent reason. I´m right now trying to start the system remotely but obviously one needs again to access the tunnel for that is the only reason to actually be sure that everything is up an running normally again.

 Dear all,

Finally the tests were run with the expected results: Pressure thresholds for the 3 involved gauges were forced on the system and the interlock on the two gate valves that limit the whole gas section were automatically closed in every case. Also the release switch was tested and the valves were able to be normally operated again.

However the interlock definition also involves the closing of the so-called "shut-off" valve in the gas injection line at the XGM. Unfortunately its behaviour couldn´t be completely observed since the Karabo device is not fully implemented. On the other hand, J. Grünert reported that after the interlock was triggered the valve was shown as also "interlocked" (despite the actual status, open or closed was not clear). This situation may require a second test, combined for instance with the gas injection test to be run by WP74 in the upcoming date). 

In any case the final results were rather positive. (despite there still some permanent uncertainty about devices being reachable or not through Karabo, as it has been previously reported).

Thanks to everyone involved.

Raúl Villanueva on behalf of B. Rio, N. Jardon and V. Bondar.

Since around 17h30 all XTD9 VAC loop is in error. It came back in karabo around 17h50. XTD2 not affected.

At that point, we had to reset every single karabo device of the XGM, then restart pumps etc.

Did the same also for the DPS upstream and downstream of XGM@XTD9.

XGM vatterfly valves will be opened by me only when I see that no further errors occur.

Currently, all but one pumps were restarted, also in Rauls DPS, only one of these not