V0 is open now!

Janni opened this morning at 11:00 the V0 valve towards the accelerator vacuum.

We removed the B4C absorber in front of the valve and now the spontaneous radiation can enter the Schenefeld vacuum. 

The interlock conditions for Vacuum loop and EPS loops towards the DESY MPS were tested and they are now enabled. 

This means that rebooting SA1 VACUUM  or EPS loop will stop the beam during the reboot (therefore call 4500 before you do it).

Dear all,

The OCD numbers for this weekend and today will be:

PLC Support: 9-8040 / 0151-68822383

DAQ Electronics (incl. timing, MicroTCA) Support: 9-8041 / 0151-68822386

Thank you.

Dear all,

CAS phone numbers for OCD:

Friday: Alessandro (98054),

Saturday: Valerii (98053),

Sunday: Gabriele (98060),

thank you.

 Issues observed by PCT over the weekend (from H. Sinn entry in the PCT log)

- The interlock of the vacuum valves worked as expected. However, we had two times the gas section around the XGM tripping the beam, because the gate valves around the XGM were closing. The reason for these trips will be investigated (probably noise on one of the involved pressure gauges). In the future it would be good to change the Vacuum-to-MPS interlock such that V0 always closes, if any of the other gate vales closes. Then the accelerator permission would come automatically come back after the trip. However re-booting of the vacuum loop needs to be planned (trips the DESY beam permission and potentially shuts down the turbo pumps). 

- The SR Imager may loose steps when moving larger distances (speed too high?)

- Several general problems of Karabo were observed, which make live difficult: For example, the 'V0 closed' and 'B4C in' indication on our main screen may just lock, even when the state of the device is changing (was observed mostly on GUI server1). It means that the status displayed on a scene can be in general not be trusted. In many cases we observed that connecting freshly opened scenes to devices take unacceptably long times (many minutes or never).  The policy of saving projects and sub-projects leads to overwriting changes, if more people work on projects and projects stay open for longer time.  

- The fact that the PLC does actually not store values through a reboot causes many problems. It basically means that motor positions are lost (set to zero) at every reboot and can be only recovered, if homing or absolute encoders exist. Also that positions of motors cannot be reset to their theoretical values means that in practice the state before the PLC reboot cannot be recovered by resetting the values.

- The imagers, in particular the SR-imager, caused a lot of problems because they were often crashing and require advanced and complicated procedures to make them run again (e.g. setting after re-instantiating the imagers are lost and come up with wrong pre-sets). The SR-imager does auto-scaling on the intensity, also when this is disabled in the software. This make the imager only useful for qualitative studies (e.g. not for gain curve or flux measurements).

General

• Parameters of motors get lost after reboot of the PLC
• Setting trigger delays can currently only be done with special authorisation from DOOCS. 
• Suggested solution from AE: Hardwire the correct defaults in the firmware
• A better procedure is needed to communicate the parameters between hardware experts and AE
• L. Samoylova suggests that CAS person on call contacts BKR once per day to establish contact and get an overall impression
• The support of CAS during the weekend was greatly appreciated

Vacuum

• No calls to experts over the weekend
• Issue with the XGM vacuum (spurious triggers of the interlock) needs to be fixed 
• Interlock logic should be changed so that V0 always closes if one of the gate valves closes

Imagers

• SR Imager crashed frequently and was hard to bring back
• Auto-scaling of SR-imager cannot be switched off. This needs fixing with high priority

XGM

• Commissioning starts today
• See above about vacuum issues

PLC

• No calls over the weekend to the experts
• Firmware for MC2 lib (TECHNOSOFT motors) is prepared and waiting for tests

Karabo

• CAS was called a few times on the weekend
• Running more than one camera at the same time appears  to cause crashes. Possibly network bandwidth is insufficient
• A procedure can be set up to read out the motor parameters at a well defined point in time for storing them in the firmware 
• Setting trigger delays should be made to work from Karabo

Work for next days

• Karabo test with MC2 Lib in the lab. As soon as this is successful it will continue in XTD9. Will be done on the SPB attenuator as N. Gerassimova is not available for PBLM tests due to a beam time
• Fix the Photonics camera driver so that auto-scaling can be turned off

Dear all,

The OCD AE numbers for the following days will be:

PLC support: 9-8040 / 0151-68822383

DAQ Electronics (incl. timing, MicroTCA) Support:  9-6941

If there are any change in the numbers, we will let you know.

Thank you.

 Dear all,

The OCD number for tonight (15/05) for the CAS group is 98053.

thanks

 Dear All,

 Interlock in XTD10 is broken today and tunnel is open until 15:00

General:

• Unfortunately, info on access to XTD10 came too late for Vacuum group to organise the work in XTD10
• Technical issues and coordination between AE/CAS and PCT/Instruments continues to happen via SASE1 Readiness Log

Vacuum:

• Issue of spurious beam aborts due to short term vacuum spikes in XGM is solved (respective gauges taken out of interlock?)
• Change to interlock logic was requested by H. Sinn: V0 always closes if one gate valve closes) The reason is that after V0 is closed the accelerator has beam permission and can continue
• This logic will now be implemented in the next update by AE

Imagers:

• Problems with several photonics cameras for data collection exist. The data transfer is sensitive to packet losses. Hence dedicated servers need to be set up to avoid loss of data.
• ITDM has provided one additional server in balcony room. Eventually a dedicated server should be set up in the tunnel
• Maybe the way the camera sends the data could also be optimised
• Auto-scaling switch problem has not been tackled yet and is considered second priority after solving general readout problem

PLC:

• Fix for MC2 lib for TECHNOSOFT motors is ready. Tests together with N. Gerassimova can start today after motion loop update 
• SPB can perform a parallel test on the PSLITs
• Motion loop and vacuum loop updates are scheduled for 12:00. 

KARABO:

• A bug was found where device clients drop messages and hence the GUI looses the state of the device
• There is a fix that needs to be deployed.  This should happen in parallel to the cycling of the loops between 12:00 and 14:00

Next Steps:

• Cycling of vacuum and motion loop at 12:00. T. Freyermuth informs the Photon Commissioning team
• Update of Karabo betweem 12:00 and 14:00
• Tests of PBLM with updated MC2 lib today
• Tests of PSLITS with updated MC2 lib by SPB

Dear all,

The Vacuum Loop will be updated now. You will be able to connect to it again in few minutes.

Thank you,

 CAS on call service for the night 98057

Dear all,

The Motion loop will be updated tomorrow due to some delays during the day. It will include the MCP XTD2 X movement.

Thank you and sorry for the inconvenience.

During the usual recovery procedure after the planned and announced cycling of the SASE1 VAC loop yesterday, we observed a serious mistake by somebody:

The gatevalve(s) of the DP section in XTD9 were opened before the related turbo pumps were (fully) operating.
This caused a pressure increase on the UHV-side to briefly 1E-2mbar (!) and for about 30min to 1E-3mbar before this was cured.
Essentially the prevacuum side gases entered the UHV-side through the DP-turbos, that's not good at all.
Especially it would be tragic, if we had the XGM high voltages on during such a mistake (which was luckily not the case)
Below is the pressure record of the XGM UHV vacuum gauge.

This must not happen again ! Therefore please observe the following points:
1) after PLC reboots, these valves shall only be opened by the experts who know the correct sequence
2) the pressures and TP operation must be verified before opening these valves.
3) WP74 (e.g. Jan or Theo) and VAC (e.g. Raul) must be informed about a planned recycling of the VAC loop (this is an established scheme and always worked very well, also yesterday)

Explanations to graph:
13h30 the preparation of the cycling started with closing the XGM vatterfly valves by WP74
14h00 apparently the gate valves of the entire gas section closed at the PLC reboot which increases slightly the pressure at the XGM
around 14h18 the pressure shoots up although nothing changed at the XGM, and it is found that DP gatevalves are open without full operation of the associated turbos
around 14h50 (after some cross-checking and phone calls) the concerned gate valves were closed and the vatterfly valves at the XGM were opened to bring the XGM (and the entire gas section) down to about 1E-7mbar

Dear all,

The motion loop will be updated in 5 minutes. The IMGs are already in safe state, please do not operate nothing. I have already contacted the control room and Harald Sinn as representative of the commissioning team.

Thank you!

The Motion loop is again up and running.

Thank you

Dear all,

The OCD AE numbers for the following days will be:

PLC support: 9-8040

DAQ Electronics (incl. timing, MicroTCA) Support:  9-8041

If there are any change in the numbers, we will let you know.

Thank you.

Dear all,

I have already contacted Andreas Koch, and Alessandro has contacted the BKR in order to do a new update of the motion loop because of one small bug detected.

I will do the update in the next few minutes.

Thank you,

General (T. Haas)

• Due to problems with interlock cabling the approval date with the TÜV on 1 June has been cancelled
• The MB, SASE1 Instruments, AE, CAS  and PSPO met yesterday and agreed that in view of the schedule risks with technical commissioning the focus of the next weeks should be enabling both FXE and SPB/SFX to accept beam for doing radiation safety measurements no later then end of June.
• SPB/SFX were asked to identify the minimum functionality for this goal. The respective components will be marked in the component priority list

FXE (A. Galler)

• So far work has concentrated on motors
• This week PSLITs and BIU2 successfully underwent E2E tests
• JJ was back this week to fix some problems on patch panels
• Johan spectrometer is back in the hutch

SPB (B. Weinhausen)

• Work continued on the tunnel loops.
• The power supply problem with photonics cameras has been solved by AE (Additional 12V terminals installed)
• Triggers for the cameras are working
• Valve between optics hutch and tunnel vacuum was opened for the first time
• There is still the unsolved problem with the TECHNOSOFT motor.
• Work will now move to the optics hutch.

Laser (T. Jezynski)

• Alarm systems are (smoke, gas etc) are now connected
• Big problems with AC systems. The effort with FM to stabilise the humidity now results in temperature swings of up to 2 degrees. The same problem also appears in the instrument laser hutches
• AC problems lead to large instabilities of the laser
• Alignment beam is now delivered to both instrument laser hutches
• No new information was received on the topic of optical synchronisation

Detector (M. Kuster)

• Management decided that LPD will be moved to FXE in June, in weeks 23/24 to allow further testing of AGIPD in HERA-S
• Interlock test on AGIPD continue
• CAS is working on integrating the DAQ of AGIPD. When this is done an update of Karabo is needed. This will not happen this week and hence not before S. Hauf is on holidays

Electronics (P. Gessler)

• There is still some work for the tunnels:
• For integration of MCP tests were done in the lab yesterday. One of the two axes is working, the other one not yet
• Tests for PBLMs could not be done as there was no common window between on N. Gerassimova and M. Umar. Tests could be done by M. Umar if N. Gerassimova agrees. She is on holidays for two days next week
• The interface of the vacuum loop and the EPS loop to the DESY MPS requires additional work for XTD9 in the next days
• Work also continues on AGIPD for one more day
• There are some absences due to sickness and holidays that also slow down the work

Karabo (A. Silenzi/B. Brockhauser)

• There will be a swap of the GUI server today to allow debugging the issue that the device states are sometimes not properly shown. People will need to re-connect
• Tests of TECHNOSOFT motor were successful. (Does not include encoder needed for SPB motor)
• There is a fundamental problem with the Photonics camera used in the SR imager. It has a rolling shutter that does not allow to get well-timed, triggered images. The same issue might be persent for the HIREX camera
• The KARABO/DOOCS driver has been re-established. This should allow to set time delays directly from Karabo again

Discussion/Next Steps

• SPB should start with technical commissioning of OPTICS hutch
• Instruments identify components needed for beam in the hutch for radiation safety measurements
• AE finishes pending tunnel work and then focus on SPB/SFX and FXE
• Displatch meetings on Monday and Wednesday at 09:30 in XHQ/MCR (*not at DESY*)

Find attached the Instrument Component priority list discussed this morning.

Regards,

Monica

AE on call service:

    Contact person:

        Nicola Coppola

    Phone:

        9 - 5627