Post on behalf of Jan Grünert:

This is the current (w5) progress at DESY concerning the analysis of the serious XGM resistor issue (summary; a detailed report by DESY is available):

• The new “pulser test” was verified at an operational XGMD at FLASH. The test method is (unfortuately) OK. This means that the test results with issues from last week on all our XGMs are REAL.
• 25% of all resistors in XBPMs affected, 0% of resistors in HAMPs affected, no capacitors affected.
• Since no HAMP resistors are damaged, damage by environmental conditions (venting, humidity etc.) can be excluded. The main differences to the affected resistors are the HV-conditioning and the experienced HV-values of the resistors (typically XBPM resisotors are tested up to 12kV, HAMP resistors each see much lower voltages).

• An XGMD chamber at FLASH was tested-to-destruction-of-resistor by applying a large differential voltage (>3kV) between neighboring electrodes, e.g. on mesh_e and FC_e. These resistors (Caddock MS220) have a nominal voltage breakthrough rating of ~500V but never had issues. They are used in all our XGMs.
• New, other resistors (Ohmite MOX-1N-13) with higher voltage breakthrough rating of 10kV were procured and arrived tuesday
• The new resistors were built into the tested-to-destruction XGMD at FLASH, unit is now pumping, will be retested with applying pulsed HV (on both Caddoc and Ohmite resistors) when at suitable pressure.

Further plan (w5end + w6) concerning the XGM resistor issue:

• If the new resistors survive the test, then next monday the XGM01 at HERA-S will be opened, the tested-defect resistors will be checked and replaced with the new resistors.
• If all tests on XGM01 are then ok including an acceptable RGA, all other XGMs (XBPM chambers) will be opened and resistors exchanged. Even ok-tested resitors would be exchanged to avoid future unexpected failures. HAMP chamber resistors will not be exchanged (large number >20 of resistors, no failures).
• If the RGA is not acceptable, the resistor Ohmite MOX-1N-16 is another alternative, but has increased delivery time.

 Vacuum loop:

- Interface between PLC and Karabo is not functioning.

The fault currently resides in the PLC behaviour, the self-description sent out is badly formatted (due to a buffer overrun)

The misbehaviour is confirmed, the PLC TCP Client from AE is also ailed by the same problem.

Motion Loop:

- As of Friday, the motor parameters were not completely sent to AE. In attachment the data collected by N. Jardon (AE). Input needed for PSLIT motors.

2017.02.13 19:30 Status:

- Self-Description back in shape.

- Bad SelfDescription reproduced in BeckhoffSim (Karabo implemented PLC communication) and solved in BeckhoffCom (Karabo "PLC Client").

- The build status of the Firmware is not ok. roughly 90% of the PLC soft devices are in error. It is a fault in the automatic Firmware build procedure.

Nerea and Jan will work on this until it is fixed.

End switch testing: the in-position end switch stopped devices working, also stopped gateway connection. Stroke of motor is not correct, parameter list to be checked. More details in "Improvements / bug list".

Both XGMs were and are on manual control (but the remote cables into the turbo pump controllers remained plugged in as usual) = both XGMs are "out of the VAC loop". 

In XTD2, I saw around 13h that both turbos were off (0 Hz) and the pressure was at 10E-4 mbar !

The green light above the ON-button was flickering. 
I switched parameter 023 (Motor TMP) to ON and restarted both pumps. The green light above the ON-button 
 stops flickering and goes steady green. 
 
---going to XTD9 
 
In XTD9, I saw that the upstream turbo was off (0 Hz) and the pressure was in the lower 10E-6mbar. 
The downstream turbo in XTD9 was still operating. 
Parameter 023 to ON, and restart pump. 
 
---going to XTD2 
 
Error codes 6, 6, 98, 2, 37 on one pump, error codes 6,6,6 on the other (XTD2).
I unplugged the REMOTE cables in XTD2 and XTD9 to avoid further trouble as long as we are on "manual". 
This info was also sent by email to the involved experts.

The EtherCAT loop for the vacuum PLC is interrupted at crate 1.6 (XTD2).

This triggered all Beckhoff Terminals  default behaviour from this point on (i.e. Valves Closed, Digital Outputs low, etc.).

 Due to problems and lack of support of CAS for accessing the devices in XTD9 tunnel through KARABO we couldn´t run the test. The system has been returned to manual operation and will remain like that until next notice.

As soon we received the request we worked on that one.  After analysing the issue proper response was sent.

​​​----- Forwarded Message -----

​​​​From: "Wajid Ehsan" <wajid.ehsan@xfel.eu>

To: "Raul Villanueva" <raul.villanueva@xfel.eu>

Cc: "Alessandro Silenzi" <alessandro.silenzi@xfel.eu>, "Nerea Jardon" <nerea.jardon@xfel.eu>

Sent: Friday, 21 April, 2017 12:56:14

Subject: Re: Karabo problems in XTD9 - Urgent!

​​​​​Hi Rahul,

we checked all the devices that are in error. The issue is realted with hardware.

Error description is mentioned below:

0x1001 FlagCtrl Error flag from controller

Regards,

Wajid Ehsan 

Here is the list of devices that are in error:
 

SA1_XTD9_VAC/VALVE/V11090A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve) (the reset command is sent to the PLC and it cannot clear the error)
SA1_XTD9_VAC/VALVE/V11100A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve)
SA1_XTD9_VAC/VALVE/V11110A  Error 3 -> valve stayed between open and closed for too long time (3s set on the valve)
SA1_XTD9_VAC/TPUMP/P11220T Error 0x1001 -> err_FlagCtrl -> Error flag from controller (need manual reset on the controller) after reset error is now cleared.
SA1_XTD9_VAC/TPUMP/P11280T Error 0x1001 -> err_FlagCtrl -> Error flag from controller (need manual reset on the controller) after reset error is now cleared.
SA1_XTD9_VAC/VALVE/V11210G Error 0x10001 ->  _err_InterlockDevNotFound  -> Interlock device not found. (the valves depends from a SA1_XTD9_VAC/SWITCH/DP_XGM_DP_ERROR which does not exist. a SA1_XTD9_VAC/SWITCH/DP_XGM_DP_OK is present in the list though, likely a PLC configuration error).
SA1_XTD9_VAC/VALVE/V11220G Error 0x10001 -> _err_InterlockDevNotFound -> Interlock device not found.
 

For the last 2 errors, which are the only one that have a clear PLC problem, I created a bug report in the redmine project of AE.
https://in.xfel.eu/redmine/issues/16510

Cheers

Alessandro Silenzi

exflplcgwSA1 seems not reachable. This inhibits support for the PLC systems.

Problems described in detail below (this entire procedure takes way too much of our time, effectively still nearly 2 hours!):

Nerea/AE requested yesterday afternoon to shutdown and restart the SASE1 VAC PLC loop once again. We start today morning (wednesday) at 8h30. 
8h30
karabo is not showing status of all devices, see three screenshots (attached). karabo-GUI issue.
These devices have to be shutdown and reinstantiated to show a status and to operate them. 
Several valves and gauges and scroll pumps affected both in XGM of XTD2 and XTD9. 
 
9h00 finally I can close all four vatterfly valves 
 
9h15 Nerea shuts down and restarts the SASE1 VAC PLC loop 
 
9h25 still waiting for karabo to reconnect to Beckhoff PLC (I cannot do anything in karabo, and 
 no current status updates of anything) 
 
9h40 SPB_XTD9_XGM scene has loaded (seems like at least). I start the reactivation procedure (switch 
 on scroll, open valves etc.). Need to reinstantiate several devices. 
 
10h00 SPB_XTD9_XGM: reinstantiate both turbo pumps, CRemotePriority ON, COn, CPump (if CRemotePriority 
 not visible, again reinstantiate) 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h05 SA1_XTD2_XGM: reinstantiate both turbo pumps, StartPumpingStation 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h10 READ the actual value of the gauges in configurator (not to trust value shown in scene) 
10h15 observe DOOCS RVC300 UHV pressures, open vatterfly valves 
 
10h20 end of intervention 

Since around 17h30 all XTD9 VAC loop is in error. It came back in karabo around 17h50. XTD2 not affected.

At that point, we had to reset every single karabo device of the XGM, then restart pumps etc.

Did the same also for the DPS upstream and downstream of XGM@XTD9.

XGM vatterfly valves will be opened by me only when I see that no further errors occur.

Currently, all but one pumps were restarted, also in Rauls DPS, only one of these not

 Dear all,

I just checked the Karabo Scene for both XTD9 and XTD9, finding out the not-so-nice situation where all the valves and pumps were respectively closed and stopped. What really worries me is again the fact that these situations are happening without aparent reason. I´m right now trying to start the system remotely but obviously one needs again to access the tunnel for that is the only reason to actually be sure that everything is up an running normally again.

 UPDATE:

XTD2

The Upstream sector is back into operation.

The Downstream one is still out of order since the turbopumps are whether in error or in an unknown/un-instantiated status.

XTD9

Upstream sector is back to full operation

Downstream sector: only one of the turbos is running. The other one P11280T cannot be reset

Right now that is the only thing I can do.

All devices went into unknown state in the karabo XGM scenes in XTD2 and XTD9 at 19h52.

Pressures INCREASE since then (as measured/shown in DOOCS):

XTD2:  2,5E-7mbar and rising

XTD9:  7,7E-7mbar and rising

Please respond now !! We are still in but want to go home !

Wajid and Leonce from CAS have managed to get us the karabo controls back.
We have now put the XGM@XTD9 back into normal vacuum operation, and opened finally the vatterfly valves, this one is ok, and we checked also the entire XTD9 gas section to be ok.

However, in XGM@XTD2, both XGM turbo pumps are still in ERROR and we cannot get them out - it is an error on the PLC level ? It's something we have not seen before.
The vacuum is at a controlled level of 5E-7mbar thanks to interlock-closed vatterflys.

Request to AE: tomorrow morning, we must remotely get the two turbos out of error (PLC client ?), otherwise we have to request a ZZ access to the tunnel XTD2 for local intervention on the turbo pump controllers.

Request to the AE+CAS group leaders: apparently we need to organize OCD also during nights on weekdays.

The current slow response times in the karabo GUI are the following (some measured times for reference):

- opening a project from the project manager : 45sec (just loading, not yet touching any device or scene)
- opening a scene and receiving the status for all devices e.g. XGM@XTD2 : between 90 and 120 sec
- beyond that, it's never clear if the displayed gauge values are true/actual and must be separately read.
- reading explicitly a gauge value (with command READ in configurator) gives currently an updated value after about 10 sec (should be almost immediate when requested)
-----> working with this (getting actual work done) is practically almost impossible

- this is using the "new" gui server "sa1-br-sys-con-gui2"
- this is after 19h30 (shouldn't less traffic help the performance ?)

Could somebody please re-establish the karabo controls of the SASE1 tunnel VAC loop (connection with Beckhoff server) ?

The karabo devices for XGM vacuum XTD2 and XTD9 are still all in UNKNOWN STATE.

We urgently would like to check the XGM vacuum status and restart the pumps before they are vented.

Thanks,

Jan

 Jan,

We are working on it.