Contacted at home 21:30 19.05. Request for support starting SA1_XTD2_MCP/CDC/MCP3 on cppServer/sa1_utca_17.

Located project and device server. Attempted to start SA1_XTD2_MCP/CDC/MCP3. Issue was missing fastAdc plugin. Found info on sa1_utca_17 in commissioning guidelines: say server is exflutc106.

Tried to ssh in to exflutc106 unsuccessfully. Phoned back 23:49 to give status. Informed that accelerator not running due to lightning. Suggested that if I couldn't fix issue soon they should contact next OCD person from 8:00.

Tried further to fix problem. Found section in commissioning guidelines: tried ssh -Y -l xctrl.Still couldn't access exflutc106. Wrote up notes. Sent to next contact. Stopped work 1:45.

* Apparently the device installation failed with ansible on utca_17. Even worse, the requested  utca_18 is not even know, cannot be pinged and nothing is documented.

** Global problem: Karabo develop, checkout and install both return zero exit status from git.

** Device problem: The dependency was declared wrong in the fastAdc

FastAdc installed by hand, dependency built by hand.

Devices can now be instantiated. For the record, with the following problem

INFO  SA1_XTD2_MCP/ADC/MCP2  : Reset simply calls initialize() method now.
INFO  SA1_XTD2_MCP/ADC/MCP2  : Connecting to hardware driver, setting up the device's namespace...
ERROR  SA1_XTD2_MCP/ADC/MCP2  : Cannot create IibusMap object for /dev/pciedevs6 and maps in /home/utcaadm/maps since:
1. Exception =====>  {
    Exception Type....:  IO Exception
    Message...........:  Open '/dev/pciedevs6' : No such file or directory
    File..............:  /[...]/src/IibusMapImpl.cc
    Function..........:  utca::IibusMapImpl::IibusMapImpl(const string&, const string&, bool)
    Line Number.......:  32
    Timestamp.........:  2017-May-21 14:59:57.856975
}


INFO  SA1_XTD2_MCP/ADC/MCP2  : Reset simply calls initialize() method now.
INFO  SA1_XTD2_MCP/ADC/MCP2  : Connecting to hardware driver, setting up the device's namespace...
ERROR  SA1_XTD2_MCP/ADC/MCP2  : Cannot create IibusMap object for /dev/pciedevs6 and maps in /home/utcaadm/maps since:
1. Exception =====>  {
    Exception Type....:  IO Exception
    Message...........:  Open '/dev/pciedevs6' : No such file or directory
    File..............:  /[...]/src/IibusMapImpl.cc
    Function..........:  utca::IibusMapImpl::IibusMapImpl(const string&, const string&, bool)
    Line Number.......:  32
    Timestamp.........:  2017-May-21 15:56:50.638370
}
 

 For record, the last message was me (DG)

 A deeper look into the dev namespace provides only pciedev0 and pciedevs3 are available. (DG)

AE on call service:

    Contact person:

        Nicola Coppola

    Phone:

        9 - 5627

Find attached the Instrument Component priority list discussed this morning.

Regards,

Monica

General (T. Haas)

• Due to problems with interlock cabling the approval date with the TÜV on 1 June has been cancelled
• The MB, SASE1 Instruments, AE, CAS  and PSPO met yesterday and agreed that in view of the schedule risks with technical commissioning the focus of the next weeks should be enabling both FXE and SPB/SFX to accept beam for doing radiation safety measurements no later then end of June.
• SPB/SFX were asked to identify the minimum functionality for this goal. The respective components will be marked in the component priority list

FXE (A. Galler)

• So far work has concentrated on motors
• This week PSLITs and BIU2 successfully underwent E2E tests
• JJ was back this week to fix some problems on patch panels
• Johan spectrometer is back in the hutch

SPB (B. Weinhausen)

• Work continued on the tunnel loops.
• The power supply problem with photonics cameras has been solved by AE (Additional 12V terminals installed)
• Triggers for the cameras are working
• Valve between optics hutch and tunnel vacuum was opened for the first time
• There is still the unsolved problem with the TECHNOSOFT motor.
• Work will now move to the optics hutch.

Laser (T. Jezynski)

• Alarm systems are (smoke, gas etc) are now connected
• Big problems with AC systems. The effort with FM to stabilise the humidity now results in temperature swings of up to 2 degrees. The same problem also appears in the instrument laser hutches
• AC problems lead to large instabilities of the laser
• Alignment beam is now delivered to both instrument laser hutches
• No new information was received on the topic of optical synchronisation

Detector (M. Kuster)

• Management decided that LPD will be moved to FXE in June, in weeks 23/24 to allow further testing of AGIPD in HERA-S
• Interlock test on AGIPD continue
• CAS is working on integrating the DAQ of AGIPD. When this is done an update of Karabo is needed. This will not happen this week and hence not before S. Hauf is on holidays

Electronics (P. Gessler)

• There is still some work for the tunnels:
• For integration of MCP tests were done in the lab yesterday. One of the two axes is working, the other one not yet
• Tests for PBLMs could not be done as there was no common window between on N. Gerassimova and M. Umar. Tests could be done by M. Umar if N. Gerassimova agrees. She is on holidays for two days next week
• The interface of the vacuum loop and the EPS loop to the DESY MPS requires additional work for XTD9 in the next days
• Work also continues on AGIPD for one more day
• There are some absences due to sickness and holidays that also slow down the work

Karabo (A. Silenzi/B. Brockhauser)

• There will be a swap of the GUI server today to allow debugging the issue that the device states are sometimes not properly shown. People will need to re-connect
• Tests of TECHNOSOFT motor were successful. (Does not include encoder needed for SPB motor)
• There is a fundamental problem with the Photonics camera used in the SR imager. It has a rolling shutter that does not allow to get well-timed, triggered images. The same issue might be persent for the HIREX camera
• The KARABO/DOOCS driver has been re-established. This should allow to set time delays directly from Karabo again

Discussion/Next Steps

• SPB should start with technical commissioning of OPTICS hutch
• Instruments identify components needed for beam in the hutch for radiation safety measurements
• AE finishes pending tunnel work and then focus on SPB/SFX and FXE
• Displatch meetings on Monday and Wednesday at 09:30 in XHQ/MCR (*not at DESY*)

Dear all,

I have already contacted Andreas Koch, and Alessandro has contacted the BKR in order to do a new update of the motion loop because of one small bug detected.

I will do the update in the next few minutes.

Thank you,

After the update of the PLC firmware, the imagers are in following status:

SR: auto gain off seems to work, correct image, screens are out. Beamview just crashed, I asked Alessandro and Dennis to have a look.

Pop-In: Everything fine.

FEL: strange image. We had this before, no idea what the reason is. I will keep trying.

Another update of the motion loop is planned later. Contact Nerea for details.

Dear all,

The OCD AE numbers for the following days will be:

PLC support: 9-8040

DAQ Electronics (incl. timing, MicroTCA) Support:  9-8041

If there are any change in the numbers, we will let you know.

Thank you.

The Motion loop is again up and running.

Thank you

Dear all,

The motion loop will be updated in 5 minutes. The IMGs are already in safe state, please do not operate nothing. I have already contacted the control room and Harald Sinn as representative of the commissioning team.

Thank you!

During the usual recovery procedure after the planned and announced cycling of the SASE1 VAC loop yesterday, we observed a serious mistake by somebody:

The gatevalve(s) of the DP section in XTD9 were opened before the related turbo pumps were (fully) operating.
This caused a pressure increase on the UHV-side to briefly 1E-2mbar (!) and for about 30min to 1E-3mbar before this was cured.
Essentially the prevacuum side gases entered the UHV-side through the DP-turbos, that's not good at all.
Especially it would be tragic, if we had the XGM high voltages on during such a mistake (which was luckily not the case)
Below is the pressure record of the XGM UHV vacuum gauge.

This must not happen again ! Therefore please observe the following points:
1) after PLC reboots, these valves shall only be opened by the experts who know the correct sequence
2) the pressures and TP operation must be verified before opening these valves.
3) WP74 (e.g. Jan or Theo) and VAC (e.g. Raul) must be informed about a planned recycling of the VAC loop (this is an established scheme and always worked very well, also yesterday)

Explanations to graph:
13h30 the preparation of the cycling started with closing the XGM vatterfly valves by WP74
14h00 apparently the gate valves of the entire gas section closed at the PLC reboot which increases slightly the pressure at the XGM
around 14h18 the pressure shoots up although nothing changed at the XGM, and it is found that DP gatevalves are open without full operation of the associated turbos
around 14h50 (after some cross-checking and phone calls) the concerned gate valves were closed and the vatterfly valves at the XGM were opened to bring the XGM (and the entire gas section) down to about 1E-7mbar

Dear all,

The Motion loop will be updated tomorrow due to some delays during the day. It will include the MCP XTD2 X movement.

Thank you and sorry for the inconvenience.

 CAS on call service for the night 98057

Dear all,

The Vacuum Loop will be updated now. You will be able to connect to it again in few minutes.

Thank you,

 Interlock in XTD10 is broken today and tunnel is open until 15:00

General:

• Unfortunately, info on access to XTD10 came too late for Vacuum group to organise the work in XTD10
• Technical issues and coordination between AE/CAS and PCT/Instruments continues to happen via SASE1 Readiness Log

Vacuum:

• Issue of spurious beam aborts due to short term vacuum spikes in XGM is solved (respective gauges taken out of interlock?)
• Change to interlock logic was requested by H. Sinn: V0 always closes if one gate valve closes) The reason is that after V0 is closed the accelerator has beam permission and can continue
• This logic will now be implemented in the next update by AE

Imagers:

• Problems with several photonics cameras for data collection exist. The data transfer is sensitive to packet losses. Hence dedicated servers need to be set up to avoid loss of data.
• ITDM has provided one additional server in balcony room. Eventually a dedicated server should be set up in the tunnel
• Maybe the way the camera sends the data could also be optimised
• Auto-scaling switch problem has not been tackled yet and is considered second priority after solving general readout problem

PLC:

• Fix for MC2 lib for TECHNOSOFT motors is ready. Tests together with N. Gerassimova can start today after motion loop update 
• SPB can perform a parallel test on the PSLITs
• Motion loop and vacuum loop updates are scheduled for 12:00. 

KARABO:

• A bug was found where device clients drop messages and hence the GUI looses the state of the device
• There is a fix that needs to be deployed.  This should happen in parallel to the cycling of the loops between 12:00 and 14:00

Next Steps:

• Cycling of vacuum and motion loop at 12:00. T. Freyermuth informs the Photon Commissioning team
• Update of Karabo betweem 12:00 and 14:00
• Tests of PBLM with updated MC2 lib today
• Tests of PSLITS with updated MC2 lib by SPB

If the recycling of the VAC loop for XTD2+9 is required as I understand from AE,
tomorrow would be a good time because there is no beam (in SASE1 until at least 15h).

After about 10h30 would be ok for me.

@ AE : Please call me on my mobile when you want to start - I might be in XTD10.

@ CAS : please have someone ready to "reconnect karabo to Beckhoff"

Since tomorrow there is no beam I want to use the time for some checks of the imagers in XTD2. Please, let the motion loop up if possible until 10 am, otherwise let me know.

 Dear All,

Dear all,

Due to local test needed in Rack 1.18 for the Motion loop, it will be interrupted only in Rack 1.18 XTD9 from now till tests finish. It will not affect any other part of the Motion loop.

Thank you.

 Dear all,

The OCD number for tonight (15/05) for the CAS group is 98053.

thanks