During the usual recovery procedure after the planned and announced cycling of the SASE1 VAC loop yesterday, we observed a serious mistake by somebody:

The gatevalve(s) of the DP section in XTD9 were opened before the related turbo pumps were (fully) operating.
This caused a pressure increase on the UHV-side to briefly 1E-2mbar (!) and for about 30min to 1E-3mbar before this was cured.
Essentially the prevacuum side gases entered the UHV-side through the DP-turbos, that's not good at all.
Especially it would be tragic, if we had the XGM high voltages on during such a mistake (which was luckily not the case)
Below is the pressure record of the XGM UHV vacuum gauge.

This must not happen again ! Therefore please observe the following points:
1) after PLC reboots, these valves shall only be opened by the experts who know the correct sequence
2) the pressures and TP operation must be verified before opening these valves.
3) WP74 (e.g. Jan or Theo) and VAC (e.g. Raul) must be informed about a planned recycling of the VAC loop (this is an established scheme and always worked very well, also yesterday)

Explanations to graph:
13h30 the preparation of the cycling started with closing the XGM vatterfly valves by WP74
14h00 apparently the gate valves of the entire gas section closed at the PLC reboot which increases slightly the pressure at the XGM
around 14h18 the pressure shoots up although nothing changed at the XGM, and it is found that DP gatevalves are open without full operation of the associated turbos
around 14h50 (after some cross-checking and phone calls) the concerned gate valves were closed and the vatterfly valves at the XGM were opened to bring the XGM (and the entire gas section) down to about 1E-7mbar

Dear all,

we are experiencing constantly crashes with the photonic science camera in the SA1 imager FR. We found now out

- If the pixel format is set to Mono16 the camera will most likely crash, go in error.

- Before instantiation the camera may show Mono16. During connection Karabo but gets a fresh update and the field in the scene switches to Mono8.

Please don't apply Mono16 again.

Best wishes, Dennis

A pile up in the BrokerQueue was detected by CAS monitoring system and

a response action from the On-Call personnel was triggered.

Regrettably, the following device servers have been shutdown and will be re-instantiated.

- Dataloggers
- alarm server
- camera servers
- Motion Loop and Vacuum Loop beckhoff servers

Sorry for the inconvenience

Called at 21.18 for SASE1_XTD2_IMGSR/MOTOR/Y being in error.

Solved by pressing 'Reset' button in Karabo GUI

Dear all,

the fast valves in XTD9 were causing some trouble. From time to time they collect some electrical noise what triggers the valve. The pressure reading on the valve controller then shows oscillations of up to 10^-5 mbar, while the pressure around the fast valve gauges in the beamline is in the 10^-9 mbar range.
To operate the beamline in the meantime I have opened and disabled both fast valves. 

This needs further investigation on the service day next wednesday.

Best regards,
Martin

Dear All,

We have a issue on the control servers and we are already working on it...

Best,

Hugo

Dear All,

We solved the incident and the services are running normally. We are still try to investigate, as far as we know we have some "zumbies" process running in the GUI servers and it was not consuming the HEATBEATs.

Dear all,

we got the information from MKK, that they have to fix a leak in the main compressed air line. Therefore they will close the valves to the tunnels for approximately one hour starting today at 1 p.m.

Please do not use the compresed air system in the tunnels during that time with air driven tools or air guns, as the volume of the distribution systems acts as a buffer.

Regards,

Martin

Dear all,
apparently the Emergency Power-Off at FXE was triggered at around 10:00 - 10:30 (my estimate). As far as I know power has been switched on again. BKR / PRC have been informed. More information will be provided by TS / Torsten Schön soon.

Edit: T. Schoen is on site and can be contacted under 0160 977 287 97

Regard, Gerd

Dear Colleagues,

in the future we would like to use this mechanism to record and exchange information that concerns the work done within the context of this task force. Please use this generously. We shall also put a link to the minutes after each weekly meeting.

regards,

Tobias 

 http://tinyurl.com/gvvv392

Tobias Freyermuth and me worked on the system and as of last night, we found two new issues between Karabo and the PLC interlock:
 - one is a Karabo issue from the Interlock PLC interaction, the Karabo device waits for a PLC response that does not come when the PLC is interlocked (i.e. Karabo wants to open a valve but the PLC correctly refuses and ignores the command)

 - one is a PLC issue, the interlock is not triggered in all cases after testing (e.g. a ion pump out of set point does not closes a gate valve closure as specified).

The first one was reproduced with the BeckhoffSim simulation. It can therefore be fixed by CAS without dependencies from AE (except for the next integration test).

just for reference ... here are pictures and schematics of the racks which are involved in the current commissioning.

 I reproduced the behaviour of the PLC on beckhoffSim. This reproduces the same behaviour seen in the tunnel devices (devices hang).

Implemented the fix and verified the correct behaviour with beckhoffSim.

Fixes will be verified next Monday with the tunnel system.

Have a nice weekend

Alessandro

ps: For future reference:

beckhoffSim  tags/1.1.0-1.4  (implements the recent plc behaviour (framework tag N/A Could not find tobias))

beckhoffDevices tags/2.0.1-1.4 hangs

 Karabo to PLC communication has been tested successfully in the SA1 Vacuum installation.

in attachment:

OK.png -> Initial Status, all OK

I have included the status columns provided by Alessandro and Sandor in the component list. Please find it in the usual location here: http://tinyurl.com/zn7otsa

The Vacuum Controls Commissioning Task Force initially helped to get the vacuum components extending from the gate valve at 2678 m to the gate valve at 2730 m in XTD2. This comprised the valves and pumps and the associated vacuum interlock logic. The necessary components were all installed and locally tested. As the next step the PLC firmware for the control of these devices was generated including the interlock logic. Finally the system was connected to version 1.47 of Karabo, which was able to monitor and control all components. A live demonstration is currently still outstanding.

As the next step the vacuum section considered should be extended upstream to include all components up to the gate valve at 2630m. In addition to vacuum components this should include the installed photon beam line components. The goal is to include all these systems into one continuous vacuum, commission all systems locally, generate the PLC firmware, implement the required vacuum interlocks, monitor the state of the system remotely via the PLC command line interface and eventually connect to Karabo and monitor and control the system via Karabo remotely using the consolidated version 2.0. While this commissioning effort is going on, development should occur on three days during the week (Mo-Wed) while the system should be operated continuously during the remaining time, thus also testing long-term stability. This should be achieved by the end of 2016.