Both SASE1 XGMs (in XTD2 and XTD9) are now again in the Vacuum Control Loop, their vacuum being controlled by Beckhoff/karabo. This works fine.

Therefore: DO NOT FOOL AROUND WITH THE VACUUM LOOP !

Since we all cannot enter the tunnel XTD2 in the next two days, it's agreed that the Vacuum Loop is NOT going to be touched at all, since  we cannot intervene in the tunnel if something goes wrong.

XGM status 21.4.2017

During Easter break, the two XGMs in the control system (XTD2+9) had no issues.
XGM@XTD10 however had an issue: tuesday morning it was found that all pumps had stopped. The scroll pump showed an error. UPS apparently had excess load sunday evening 19h30 but couldn't send email (network error). Possibly vented with tunnel air up to 10mbar since that time. Investigation is ongoing.

Thursday enabled DOOCS pressure reading of XGM in XTD10, UPS rechecked

System surveillance on coming weekend: Jan and Theo 1 day

Other remarks:
- WP74 staff have received their dosimeters for ZZ access if required
- some automatic monitoring and info system is needed soon, e.g. karabo to NAGIOS connection

Together with Nerea and Wajid and Alessandro

10:00 closed vatterflies at XTD9 and XTD2 from Karabo
Florian is at XTD9 next to the XGM 
XTD9: 
LOOP re-cycle
All valve closed and scroll off (turbos in unknown state; Florian in tunnel sees 990 Hz)
Scroll and turbos switched ON
Angle valve opened over Karabo
Problem to open gate-valve
Set Karabo offline
Nerea switched turbos at XTD2 ON (XTD9 were on) by PLC
Karabo is reconnected to PLC (brought online again)
XTD2: (with Alessandro) 
Scene optimized: Turbo speed, new prevac display, and preliminary scroll icon
Open KF angle and gate valve (follow "XGM vacuum incident procedure")
XTD9: (with Alssandro) 
Scene optimized: Turbo speed, new prevac display, and scroll icon
Open angle and gate valve (follow "XGM vacuum incident procedure")
New project created "SASE1-XGM" which contains both XTD2 and XTD9 XGM vacuum projects, thus scenes from both can be opened from one GUI simultaneously.

11:55 XTD9 vatterflies OPEN 
12:00 XTD2 vatterflies OPEN  
      (V10120V could only be opened after shutdown of device and re initialization) 
 
Maybe upper pressure probes need new calibration -> Check formula 
Prevacuum pressures: 
XTD9: 6.4E-4 mbar (G11081P)
XTD2: 2.5E-2 mbar (G10100P) and 1.2E-3 mbar (G10101P)

Summary : both XGMs in XTD2 and XTD9 are now back in normal pumping operation, and they were never exposed to more than a few 10E-6mbar. Good. But it took 2 hours and 4 experts.

Problems described in detail below (this entire procedure takes way too much of our time, effectively still nearly 2 hours!):

Nerea/AE requested yesterday afternoon to shutdown and restart the SASE1 VAC PLC loop once again. We start today morning (wednesday) at 8h30. 
8h30
karabo is not showing status of all devices, see three screenshots (attached). karabo-GUI issue.
These devices have to be shutdown and reinstantiated to show a status and to operate them. 
Several valves and gauges and scroll pumps affected both in XGM of XTD2 and XTD9. 
 
9h00 finally I can close all four vatterfly valves 
 
9h15 Nerea shuts down and restarts the SASE1 VAC PLC loop 
 
9h25 still waiting for karabo to reconnect to Beckhoff PLC (I cannot do anything in karabo, and 
 no current status updates of anything) 
 
9h40 SPB_XTD9_XGM scene has loaded (seems like at least). I start the reactivation procedure (switch 
 on scroll, open valves etc.). Need to reinstantiate several devices. 
 
10h00 SPB_XTD9_XGM: reinstantiate both turbo pumps, CRemotePriority ON, COn, CPump (if CRemotePriority 
 not visible, again reinstantiate) 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h05 SA1_XTD2_XGM: reinstantiate both turbo pumps, StartPumpingStation 
      READ ADriveCurrent, ARPMIn several times, both should change from time to time, don't 
 worry about "0" every second read time 
10h10 READ the actual value of the gauges in configurator (not to trust value shown in scene) 
10h15 observe DOOCS RVC300 UHV pressures, open vatterfly valves 
 
10h20 end of intervention 

Since around 17h30 all XTD9 VAC loop is in error. It came back in karabo around 17h50. XTD2 not affected.

At that point, we had to reset every single karabo device of the XGM, then restart pumps etc.

Did the same also for the DPS upstream and downstream of XGM@XTD9.

XGM vatterfly valves will be opened by me only when I see that no further errors occur.

Currently, all but one pumps were restarted, also in Rauls DPS, only one of these not

WP74   OCD  for the coming weekend 29.4.-1.5.2017:

Saturday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Sunday: Jan, jan.gruenert@xfel.eu, 9-5777

Monday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Note: Theo has Torbens work phone this weekend since Theo has not yet his own mobile work phone

All devices went into unknown state in the karabo XGM scenes in XTD2 and XTD9 at 19h52.

Pressures INCREASE since then (as measured/shown in DOOCS):

XTD2:  2,5E-7mbar and rising

XTD9:  7,7E-7mbar and rising

Please respond now !! We are still in but want to go home !

Wajid and Leonce from CAS have managed to get us the karabo controls back.
We have now put the XGM@XTD9 back into normal vacuum operation, and opened finally the vatterfly valves, this one is ok, and we checked also the entire XTD9 gas section to be ok.

However, in XGM@XTD2, both XGM turbo pumps are still in ERROR and we cannot get them out - it is an error on the PLC level ? It's something we have not seen before.
The vacuum is at a controlled level of 5E-7mbar thanks to interlock-closed vatterflys.

Request to AE: tomorrow morning, we must remotely get the two turbos out of error (PLC client ?), otherwise we have to request a ZZ access to the tunnel XTD2 for local intervention on the turbo pump controllers.

Request to the AE+CAS group leaders: apparently we need to organize OCD also during nights on weekdays.

The current slow response times in the karabo GUI are the following (some measured times for reference):

- opening a project from the project manager : 45sec (just loading, not yet touching any device or scene)
- opening a scene and receiving the status for all devices e.g. XGM@XTD2 : between 90 and 120 sec
- beyond that, it's never clear if the displayed gauge values are true/actual and must be separately read.
- reading explicitly a gauge value (with command READ in configurator) gives currently an updated value after about 10 sec (should be almost immediate when requested)
-----> working with this (getting actual work done) is practically almost impossible

- this is using the "new" gui server "sa1-br-sys-con-gui2"
- this is after 19h30 (shouldn't less traffic help the performance ?)

WP74   OCD  for the  weekend 6.5.-7.5.2017:

Saturday: Jan, jan.gruenert@xfel.eu, 9-5777

Sunday: Theo, theophilos.maltezopoulos@xfel.eu, 9-6792

Could somebody please re-establish the karabo controls of the SASE1 tunnel VAC loop (connection with Beckhoff server) ?

The karabo devices for XGM vacuum XTD2 and XTD9 are still all in UNKNOWN STATE.

We urgently would like to check the XGM vacuum status and restart the pumps before they are vented.

Thanks,

Jan

 Jan,

We are working on it. 

Plan of WP74 OCD contacts for May.
Assignments are subject to change.

Initially availability is 10h-18h on the given days.
This might be adapted as well for later weeks.

If the recycling of the VAC loop for XTD2+9 is required as I understand from AE,
tomorrow would be a good time because there is no beam (in SASE1 until at least 15h).

After about 10h30 would be ok for me.

@ AE : Please call me on my mobile when you want to start - I might be in XTD10.

@ CAS : please have someone ready to "reconnect karabo to Beckhoff"

During the usual recovery procedure after the planned and announced cycling of the SASE1 VAC loop yesterday, we observed a serious mistake by somebody:

The gatevalve(s) of the DP section in XTD9 were opened before the related turbo pumps were (fully) operating.
This caused a pressure increase on the UHV-side to briefly 1E-2mbar (!) and for about 30min to 1E-3mbar before this was cured.
Essentially the prevacuum side gases entered the UHV-side through the DP-turbos, that's not good at all.
Especially it would be tragic, if we had the XGM high voltages on during such a mistake (which was luckily not the case)
Below is the pressure record of the XGM UHV vacuum gauge.

This must not happen again ! Therefore please observe the following points:
1) after PLC reboots, these valves shall only be opened by the experts who know the correct sequence
2) the pressures and TP operation must be verified before opening these valves.
3) WP74 (e.g. Jan or Theo) and VAC (e.g. Raul) must be informed about a planned recycling of the VAC loop (this is an established scheme and always worked very well, also yesterday)

Explanations to graph:
13h30 the preparation of the cycling started with closing the XGM vatterfly valves by WP74
14h00 apparently the gate valves of the entire gas section closed at the PLC reboot which increases slightly the pressure at the XGM
around 14h18 the pressure shoots up although nothing changed at the XGM, and it is found that DP gatevalves are open without full operation of the associated turbos
around 14h50 (after some cross-checking and phone calls) the concerned gate valves were closed and the vatterfly valves at the XGM were opened to bring the XGM (and the entire gas section) down to about 1E-7mbar

SASE1 / SASE3 readiness --- WP74

2017-07-14

SASE1: 

• commissioning of Kmono started yesterday at 10h for some hours
• Kmono will be used for undulator commissioning as soon as Kmono is commissioned itself, and machine is ready for this
• Support to SASE tuning given to machine (ongoing)
• Maybe today: support during SASE gain studies
• During last week (service week) : GLT-Anbindung XTD2 established

SASE3:

• During last week shutdown : 
  • progress on DP installation, XGM cabling almost done, much imager and timing cabling done
• When XTD10 is closed (as now) we currently cannot do much because there is no PLC loop in SASE3.
• Non-Beckhoff technical commissioning started (e.g. some progress on MPOD HV-power supply)
• WP74 must do a lot of technical commissioning in XTD10. Our work is by far not done when all our vacuum screws are tightened. There are different WP74 components in XTD10 than in SASE1 (not just a simple copy). WP74 requires the Beckhoff loops for tech. commissioning.
• Alignment of DP „islands“ to be done in next shutdown.
• Planned for next shutdown: installation of PES in XTD10, finishing of cabling

02.09. from 08:00 until 03.09. 08:00 - Wolfgang Freund 9-5630  

03.09. from 08:00 until 04.09. 08:00 - Florian Dietrich 9-6968

4.11. from 08:00 until 5.11. 08:00 - Jan Gruenert by email (work mobile defect)

5.11. from 08:00 until 6.11. 08:00 - Naresh Kujala 9-6840