Dear all,

I just checked the Karabo Scene for both XTD9 and XTD9, finding out the not-so-nice situation where all the valves and pumps were respectively closed and stopped. What really worries me is again the fact that these situations are happening without aparent reason. I´m right now trying to start the system remotely but obviously one needs again to access the tunnel for that is the only reason to actually be sure that everything is up an running normally again.

 UPDATE:

XTD2

The Upstream sector is back into operation.

The Downstream one is still out of order since the turbopumps are whether in error or in an unknown/un-instantiated status.

XTD9

Upstream sector is back to full operation

Downstream sector: only one of the turbos is running. The other one P11280T cannot be reset

Right now that is the only thing I can do.

Hi all,

in order to integrate the CRL in XTD2 I am going to update the motion loop.

I will let you know as soon as this is done.

Cheers,

    Tobias

All devices went into unknown state in the karabo XGM scenes in XTD2 and XTD9 at 19h52.

Pressures INCREASE since then (as measured/shown in DOOCS):

XTD2:  2,5E-7mbar and rising

XTD9:  7,7E-7mbar and rising

Please respond now !! We are still in but want to go home !

Wajid and Leonce from CAS have managed to get us the karabo controls back.
We have now put the XGM@XTD9 back into normal vacuum operation, and opened finally the vatterfly valves, this one is ok, and we checked also the entire XTD9 gas section to be ok.

However, in XGM@XTD2, both XGM turbo pumps are still in ERROR and we cannot get them out - it is an error on the PLC level ? It's something we have not seen before.
The vacuum is at a controlled level of 5E-7mbar thanks to interlock-closed vatterflys.

Request to AE: tomorrow morning, we must remotely get the two turbos out of error (PLC client ?), otherwise we have to request a ZZ access to the tunnel XTD2 for local intervention on the turbo pump controllers.

Request to the AE+CAS group leaders: apparently we need to organize OCD also during nights on weekdays.

Could somebody please re-establish the karabo controls of the SASE1 tunnel VAC loop (connection with Beckhoff server) ?

The karabo devices for XGM vacuum XTD2 and XTD9 are still all in UNKNOWN STATE.

We urgently would like to check the XGM vacuum status and restart the pumps before they are vented.

Thanks,

Jan

 Jan,

We are working on it. 

Since tomorrow there is no beam I want to use the time for some checks of the imagers in XTD2. Please, let the motion loop up if possible until 10 am, otherwise let me know.

If the recycling of the VAC loop for XTD2+9 is required as I understand from AE,
tomorrow would be a good time because there is no beam (in SASE1 until at least 15h).

After about 10h30 would be ok for me.

@ AE : Please call me on my mobile when you want to start - I might be in XTD10.

@ CAS : please have someone ready to "reconnect karabo to Beckhoff"

After the update of the PLC firmware, the imagers are in following status:

SR: auto gain off seems to work, correct image, screens are out. Beamview just crashed, I asked Alessandro and Dennis to have a look.

Pop-In: Everything fine.

FEL: strange image. We had this before, no idea what the reason is. I will keep trying.

Another update of the motion loop is planned later. Contact Nerea for details.

Contacted at home 21:30 19.05. Request for support starting SA1_XTD2_MCP/CDC/MCP3 on cppServer/sa1_utca_17.

Located project and device server. Attempted to start SA1_XTD2_MCP/CDC/MCP3. Issue was missing fastAdc plugin. Found info on sa1_utca_17 in commissioning guidelines: say server is exflutc106.

Tried to ssh in to exflutc106 unsuccessfully. Phoned back 23:49 to give status. Informed that accelerator not running due to lightning. Suggested that if I couldn't fix issue soon they should contact next OCD person from 8:00.

Tried further to fix problem. Found section in commissioning guidelines: tried ssh -Y -l xctrl.Still couldn't access exflutc106. Wrote up notes. Sent to next contact. Stopped work 1:45.

* Apparently the device installation failed with ansible on utca_17. Even worse, the requested  utca_18 is not even know, cannot be pinged and nothing is documented.

** Global problem: Karabo develop, checkout and install both return zero exit status from git.

** Device problem: The dependency was declared wrong in the fastAdc

FastAdc installed by hand, dependency built by hand.

Devices can now be instantiated. For the record, with the following problem

INFO  SA1_XTD2_MCP/ADC/MCP2  : Reset simply calls initialize() method now.
INFO  SA1_XTD2_MCP/ADC/MCP2  : Connecting to hardware driver, setting up the device's namespace...
ERROR  SA1_XTD2_MCP/ADC/MCP2  : Cannot create IibusMap object for /dev/pciedevs6 and maps in /home/utcaadm/maps since:
1. Exception =====>  {
    Exception Type....:  IO Exception
    Message...........:  Open '/dev/pciedevs6' : No such file or directory
    File..............:  /[...]/src/IibusMapImpl.cc
    Function..........:  utca::IibusMapImpl::IibusMapImpl(const string&, const string&, bool)
    Line Number.......:  32
    Timestamp.........:  2017-May-21 14:59:57.856975
}


INFO  SA1_XTD2_MCP/ADC/MCP2  : Reset simply calls initialize() method now.
INFO  SA1_XTD2_MCP/ADC/MCP2  : Connecting to hardware driver, setting up the device's namespace...
ERROR  SA1_XTD2_MCP/ADC/MCP2  : Cannot create IibusMap object for /dev/pciedevs6 and maps in /home/utcaadm/maps since:
1. Exception =====>  {
    Exception Type....:  IO Exception
    Message...........:  Open '/dev/pciedevs6' : No such file or directory
    File..............:  /[...]/src/IibusMapImpl.cc
    Function..........:  utca::IibusMapImpl::IibusMapImpl(const string&, const string&, bool)
    Line Number.......:  32
    Timestamp.........:  2017-May-21 15:56:50.638370
}
 

 For record, the last message was me (DG)

 A deeper look into the dev namespace provides only pciedev0 and pciedevs3 are available. (DG)

Dear all,

we are experiencing constantly crashes with the photonic science camera in the SA1 imager FR. We found now out

- If the pixel format is set to Mono16 the camera will most likely crash, go in error.

- Before instantiation the camera may show Mono16. During connection Karabo but gets a fresh update and the field in the scene switches to Mono8.

Please don't apply Mono16 again.

Best wishes, Dennis

Called at 21.18 for SASE1_XTD2_IMGSR/MOTOR/Y being in error.

Solved by pressing 'Reset' button in Karabo GUI

It was just decided to put a very low intensity beam into the SASE1 hutches starting tomorrow morning at 8 AM. Both instruments declared that they are ready to accept it. Due to the very low intensity no special measures need to be taken outside the hutches. SRP and D3 shall be in the experiment hall. Harald Sinn shall coordinate beam activities from the BKR. Please note that the TC and S1R meetings take place as announced.

The upgrade of the switches is finished to the extend that it allows Advanced Electronic group to work on KMono (Nerea is informed)

However, during the upgrade of the network one switch got stuck and this is the switch in XTD2 tunnel. The IP network in the XTD2 tunnel is not reachable at the moment (Beckhoff loops are fine) and we are going top the tunnel soon.

Dear all,

I have received a request from Konstantin Sukharnikov to break the Motion loop in XTD2 from Rack 1.5 in order to do checks in the CRLs. So you will loose communication with almost the whole Motion loop in SASE1. I will let you know when they are finished.

Thank you,

We are finished. One encoder terminal has failed and needs replacing: rack 1.5 EC14