Dear all,

I will do an update into the Motion loop. Would take only few minutes and the PLC will be running again.

Thank you and sorry

I added the requests that I was made aware of to the list of access requests for tomorrow. While the interlock in XTD2 will be broken and hence the tunnel is accessible without special procedures, please note the followoing

• Please make sure your request in in the DESY table of access requests 
• You have valid access permissions and two (!) dosimeters, one from XFEL one from DESY

 The XSDU2  area will be open (as XTD2) tomorrow between 9 ans 12. To register the access, please send also in this case a mail to T.Haas.

Dear all,

I will do an update of the motion loop so the PBLM could be tested. I will do it know and the PLC will be again available in few minutes.

Thank you!

Problems described in detail below (this entire procedure takes way too much of our time, effectively still nearly 2 hours!):

Nerea/AE requested yesterday afternoon to shutdown and restart the SASE1 VAC PLC loop once again. We start today morning (wednesday) at 8h30. 
8h30
karabo is not showing status of all devices, see three screenshots (attached). karabo-GUI issue.
These devices have to be shutdown and reinstantiated to show a status and to operate them. 
Several valves and gauges and scroll pumps affected both in XGM of XTD2 and XTD9. 
 
9h00 finally I can close all four vatterfly valves 
 
9h15 Nerea shuts down and restarts the SASE1 VAC PLC loop 
 
9h25 still waiting for karabo to reconnect to Beckhoff PLC (I cannot do anything in karabo, and 
 no current status updates of anything) 
 
9h40 SPB_XTD9_XGM scene has loaded (seems like at least). I start the reactivation procedure (switch 
 on scroll, open valves etc.). Need to reinstantiate several devices. 
 
10h00 SPB_XTD9_XGM: reinstantiate both turbo pumps, CRemotePriority ON, COn, CPump (if CRemotePriority 
 not visible, again reinstantiate) 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h05 SA1_XTD2_XGM: reinstantiate both turbo pumps, StartPumpingStation 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h10 READ the actual value of the gauges in configurator (not to trust value shown in scene) 
10h15 observe DOOCS RVC300 UHV pressures, open vatterfly valves 
 
10h20 end of intervention 

Vacuum is running again and Motion also, yesterday I forgot to update you about the Motion PLC.

Sorry and best regards,

Dear all,

We need to update the Vacuum loop in order to correct and add some configuration to the interlock conditions.

It would be running in few minutes again.

Thank you!

Dear all,

We have to make some changes again into the Motion loop. Therefore I will do an update at 14.45h.

I will keep you inform when it is running again.

Thanks!

 After some minor issues, the first test has been done today and the interlock wasn´t triggered despite the conditions were met (also visible in Karabo).

This requires intervention and revision from AE

The next try-out should be done as soon as possible during this week.

Regards.

Raúl, Benoit and Valerii

Dear all.

Motion loop is running again.

Thank you

XTDs Info:

Tunnel XTD2 will be accessible this Thursday, 27.04. between 9 and 12. Dosimeter and General radiation training required. Access requests should go to T.Haas.

XTD10 will be closed starting from this Thursday, 27.04. at 12:00

Provided that the authorities give the "go ahead", DESY plans to put beam in XTD2 on Thursday's evening.

Karabo:

• System running  but performance of the GUI still sluggish
• There appears to be a resource leak. Second server will be set up tomorrow as workaround. Experts are working on it.
• In parallel experts are working on a definitive solution
• Deployment today on a new beckhoff device for the decodification of PLC error messages.

Vacuum:

• Systematic tests of ion pumps started again

PBLM/Motion Loop: 

• There is a resource problem when trying to read out the cameras. This problem will be reconsidered once the general issue with the GUI server is solved.
• ASilenzi will ask Natalia for acceptance test definition for PBLM
• Motion loop will be down until 10:30 for update. An entry will be written when the loop is again up. 

SPB Tunnel components

• The Renishaw encoders can be read and positions are reliable. An error message is coming from the encoder and is probably due to a wrong voltage level delivered to the encoder. The encoders can be operated in this condition.
• T. Freyermuth is in contact with the encoder company tu understand the problem and work on a fix
• The same problem also affects FXE components
• The cameras operated at 1Hz deliver delayed images. 1/3 Hz operation (almost live mode) is OK - probably related to GUI memory issues

FXE

• Encoders of Loop1 have the same error message as SPB - the encoder positions are reliable - local components test can go on (e.g. OGT1)
• Encoders of loop2 are not powered - AE to investigate

Tasks for today:

• Identify a fix/workaround for the GUI server resource problem
• Deployment of a beckhoff device for the decodification of the error messages.
• Find a solution for the Renishaw encoder problem SPB Tunnel/ FXE Loop1
• Investigation of the FXE Loop2 encoders power
• Continue testing of ion pumps in XTD9
• Test of gas section interlock in XTD9

Tasks for tomorrow:

• Setup of the second server for karabo GUI

In case of any problems, please, call the CAS personnel on 9-6443, 9-5630, or 9-6840.

Have a nice week-end, Sandor

The EPS-loop is now complete again.

The interface from EPS to MPS is working now. One broken wire in the cable between EPS and MPS was found. We are using now other pairs for the signals "SA1_SYS_EPS2MPS/DCTRL/MODE1", "SA1_SYS_EPS2MPS/DCTRL/MODES" and "SA1_SYS_EPS2MPS/DCTRL/MODEM".

Cheers,

    Tobias

Together with Nerea and Wajid and Alessandro

10:00 closed vatterflies at XTD9 and XTD2 from Karabo
Florian is at XTD9 next to the XGM 
XTD9: 
LOOP re-cycle
All valve closed and scroll off (turbos in unknown state; Florian in tunnel sees 990 Hz)
Scroll and turbos switched ON
Angle valve opened over Karabo
Problem to open gate-valve
Set Karabo offline
Nerea switched turbos at XTD2 ON (XTD9 were on) by PLC
Karabo is reconnected to PLC (brought online again)
XTD2: (with Alessandro) 
Scene optimized: Turbo speed, new prevac display, and preliminary scroll icon
Open KF angle and gate valve (follow "XGM vacuum incident procedure")
XTD9: (with Alssandro) 
Scene optimized: Turbo speed, new prevac display, and scroll icon
Open angle and gate valve (follow "XGM vacuum incident procedure")
New project created "SASE1-XGM" which contains both XTD2 and XTD9 XGM vacuum projects, thus scenes from both can be opened from one GUI simultaneously.

11:55 XTD9 vatterflies OPEN 
12:00 XTD2 vatterflies OPEN  
      (V10120V could only be opened after shutdown of device and re initialization) 
 
Maybe upper pressure probes need new calibration -> Check formula 
Prevacuum pressures: 
XTD9: 6.4E-4 mbar (G11081P)
XTD2: 2.5E-2 mbar (G10100P) and 1.2E-3 mbar (G10101P)

Summary : both XGMs in XTD2 and XTD9 are now back in normal pumping operation, and they were never exposed to more than a few 10E-6mbar. Good. But it took 2 hours and 4 experts.

Dear all,

The service of On-call Duty will be available from AE group during weekend in this number: 6442

Please make a post with the OCD numbers from CAS, WP74 and WP73.

Thank you and bests.

We have to disconnect the EPS crate in RACK 1.14 in order to check the interface to the MPS.

I will keep you updated

exflplcgwSA1 seems not reachable. This inhibits support for the PLC systems.

Here is the list of devices that are in error:
 

SA1_XTD9_VAC/VALVE/V11090A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve) (the reset command is sent to the PLC and it cannot clear the error)
SA1_XTD9_VAC/VALVE/V11100A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve)
SA1_XTD9_VAC/VALVE/V11110A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve)
SA1_XTD9_VAC/TPUMP/P11220T Error 0x1001 -> err_FlagCtrl -> Error flag from controller (need manual reset on the controller) after reset error is now cleared.
SA1_XTD9_VAC/TPUMP/P11280T Error 0x1001 -> err_FlagCtrl -> Error flag from controller (need manual reset on the controller) after reset error is now cleared.
SA1_XTD9_VAC/VALVE/V11210G Error 0x10001 ->  _err_InterlockDevNotFound  -> Interlock device not found. (the valves depends from a SA1_XTD9_VAC/SWITCH/DP_XGM_DP_ERROR which does not exist. a SA1_XTD9_VAC/SWITCH/DP_XGM_DP_OK is present in the list though, likely a PLC configuration error).
SA1_XTD9_VAC/VALVE/V11220G Error 0x10001 -> _err_InterlockDevNotFound -> Interlock device not found.
 

For the last 2 errors, which are the only one that have a clear PLC problem, I created a bug report in the redmine project of AE.
https://in.xfel.eu/redmine/issues/16510

Cheers

Alessandro Silenzi