ELOG SASE1 Status and SASE3 Readiness

Group Selection Page HELP DOC Controls Group Data Analysis Group ITDM WP76 DET Vacuum XRO EEE WP78 WP74 FXE SCS SPB MID HED SQS SXP Sample Environment Photon Commissioning Team Operation PSPO XO TS

Technical integration and coordination SASE1 Status and SASE3 Readiness

Joint Operation and Readiness Logbook, Contact: T.Haas, A. Violante, G. Wellenreuther, Page 24 of 30

Not logged in

List | Find | Login | Help

Full | Summary | Threaded | Hide attachments

597 Entries

Goto page Previous 1, 2, 3 ... 23, 24, 25 ... 28, 29, 30 Next

New Deployment of Karabo for the SASE1 control machines.

I will ramp down the system and bring it back up during the lunch break after installing the
tag 2.1.10.

To install a new version of Karabo use the script attached on your terminal

move your the karabo directory to a backup or remove it.

to backup:
$ mv ~/karabo ~/karabo.bkp

to remove:
$ rm -fr ~/karabo

then execute:
$ bash PATHTO/download.sh

Attachment 1: download.sh

#!/bin/bash

if [ -z ${outfilename} ]
then
    outfilename=.karabo.sh
fi
echo $outfilename
git clone https://xfel.redmine:K9Sss5CmWtQ_nbx3gyt2@git.xfel.eu/gitlab/Karabo/commissioning-guidelines .temp-commissioning --depth=1
cp -f .temp-commissioning/project_generation/install.sh .install.sh
rm -fr .temp-commissioning
. .install.sh
rm ${outfilename}
rm .install.sh

New Deployment of Karabo for the SASE1 control machines.

Alessandro Silenzi wrote:

I will ramp down the system and bring it back up during the lunch break after installing the
tag 2.1.10.

To install a new version of Karabo use the script attached on your terminal

move your the karabo directory to a backup or remove it.

to backup:
$ mv ~/karabo ~/karabo.bkp

to remove:
$ rm -fr ~/karabo

then execute:
$ bash PATHTO/download.sh

The system is up.

The users using virtual machines will encounter problems installing karabo 2.1.10 in the way that was described.
Further instructions will come later in the day

New Deployment of Karabo for the SASE1 control machines.

Alessandro Silenzi wrote:

I will ramp down the system and bring it back up during the lunch break after installing the
tag 2.1.10.

To install a new version of Karabo use the script attached on your terminal

move your the karabo directory to a backup or remove it.

to backup:
$ mv ~/karabo ~/karabo.bkp

to remove:
$ rm -fr ~/karabo

then execute:
$ bash PATHTO/download.sh

The system is up.

The users using virtual machines will encounter problems installing karabo 2.1.10 in the way that was described.
Further instructions will come later in the day

for Users with a VM:
- use the download script
- Remove the karabo directory:
$ rm -fr ~/karabo
- then execute:
$ bash PATHTO/download.sh

Sorry for the delayed instructions

SOLVED BROKER Problem

A pile up in the BrokerQueue was detected by CAS monitoring system and

a response action from the On-Call personnel was triggered.

Regrettably, the following device servers have been shutdown and will be re-instantiated.

- Dataloggers
- alarm server
- camera servers
- Motion Loop and Vacuum Loop beckhoff servers

Sorry for the inconvenience

Karabo for SASE1 topic restarted

Here attached the main SASE1 panel

Attachment 1: 07.png

Re: Status report after dispatch meeting 09.30

Small correction:

The coordinated motion is the KB-mirror's, not the Vacuum.

The postfix is 2.2.2.1 and includes features that were not included in 2.2.2

Karabo 2.2.2 is ready

Adriano Violante wrote:

SPB

KB chamber being pumped down
Smaract commissioning ongoing
Coordinate motion for VAC ready to be tested (probably tomorrow). It has to be agreed with Richard Bean
Clean-up slits Karabo middle layer deployed and working. - The same hase to be done for the power slits.

FXE

DAQ test ongoing - fixing of bugs
After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Still the case.

XPD

XGM in SASE3 fixed (A turbo broke - replaced)
One exit slit Imager in XTD 10 commissioned

PBLM moto problem was found to be due to a wrong Karabo-PLC linking. ==> FIXED
Soft mono commissioning started and AE support was required. DONE

ITDM

Main problems with Train ID solved. Some problems still remain for some beckhoff loops in the tunnels==> AE on that.
Some way to document the change of data format for devices has and will be found by IDTM
Data source running for FXE. Changes have to be clearly communicated.

CAS

Soft fix improving the GUI deployed. No 2.2.2 ready.

Issue with Karabo device returning the sensors data (e.g. cooling AGIPD) is being investigated. The Bug was found also in test environment but still random.

SASE3 - CY

GATT works
Inficon device to be improved (loop restart needed)
CAS will deliver a method to deal with the configuration changes asked by the experiments, redirecting the changes to the proper platform (Firmware (AE), Karabo...)

Re: Status report after dispatch meeting 13:00

Regarding CAS Support for SASE2 Lasing:

CAS is happy to support in the frame of the On Call Duty system as it has been provided in the past.

Adriano Violante wrote:

SPB

Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
Smaract motors for PAM are needed and Nerea is on it.
It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions have to be checked next week when HSinn is back.
During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it.
Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

Access of XTD1 today @14:00 to fix some EPS switches
SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

SCS exp: first loop running. 2 creates missing but loop is up.
MPS connection tunnels SASE2 done

ITDM

Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

SASE3 loops will restarted at 11:30

This announcement is to notify you that the intervention planned to configure the interlocks as read-only will be carried out at 11:30.

A following message will notify of the end of the intervention

Alessandro

Re: SASE3 loops will restarted at 11:30

Alessandro Silenzi wrote:

This announcement is to notify you that the intervention planned to configure the interlocks as read-only will be carried out at 11:30.

A following message will notify of the end of the intervention

Alessandro

Intervention completed.

SA3_SYS_EPS2MPS/DCTRL/BEAM_GATT_OK has now read-only interlock properties

Loop status schedule for KW12

Dear all,

find here the foreseen loop status for KW12. The schedule could change. All changes will be notified by entries in the logbook.

Day	Motion	Vacuum	EPS
Monday	Loop is up.	Loop up.	Loop up
Tuesday	Loop is up.	Loop down. AE updates loop (maybe short )	Loop up
Wednesday	Loop down. AE updates loop	Loop up	Loop up
Thursday	Loop up	Loop up	Loop up
Friday	Loop up	Loop up	Loop up

Status and Plans after the 9:30 meeting

XTDs Info:

Tunnel XTD2 will be accessible this Thursday, 27.04. between 9 and 12. Dosimeter and General radiation training required. Access requests should go to T.Haas.

XTD10 will be closed starting from this Thursday, 27.04. at 12:00

Provided that the authorities give the "go ahead", DESY plans to put beam in XTD2 on Thursday's evening.

Karabo:

System running but performance of the GUI still sluggish
There appears to be a resource leak. Second server will be set up tomorrow as workaround. Experts are working on it.
In parallel experts are working on a definitive solution
Deployment today on a new beckhoff device for the decodification of PLC error messages.

Vacuum:

Systematic tests of ion pumps started again

PBLM/Motion Loop:

There is a resource problem when trying to read out the cameras. This problem will be reconsidered once the general issue with the GUI server is solved.
ASilenzi will ask Natalia for acceptance test definition for PBLM
Motion loop will be down until 10:30 for update. An entry will be written when the loop is again up.

SPB Tunnel components

The Renishaw encoders can be read and positions are reliable. An error message is coming from the encoder and is probably due to a wrong voltage level delivered to the encoder. The encoders can be operated in this condition.
T. Freyermuth is in contact with the encoder company tu understand the problem and work on a fix
The same problem also affects FXE components
The cameras operated at 1Hz deliver delayed images. 1/3 Hz operation (almost live mode) is OK - probably related to GUI memory issues

FXE

Encoders of Loop1 have the same error message as SPB - the encoder positions are reliable - local components test can go on (e.g. OGT1)
Encoders of loop2 are not powered - AE to investigate

Tasks for today:

Identify a fix/workaround for the GUI server resource problem
Deployment of a beckhoff device for the decodification of the error messages.
Find a solution for the Renishaw encoder problem SPB Tunnel/ FXE Loop1
Investigation of the FXE Loop2 encoders power
Continue testing of ion pumps in XTD9
Test of gas section interlock in XTD9

Tasks for tomorrow:

Setup of the second server for karabo GUI

Status and Plans after the 9:30 meeting

XTDs Info:

Tunnel XTD2 will be accessible this Thursday (TUNNEL OPEN, NO ZZ), 27.04. between 9 and 12. Dosimeter and General radiation training required. Access requests should go to T.Haas.

XTD10 will be closed starting from this Thursday, 27.04. at 12:00

Provided that the authorities give the "go ahead", DESY plans to put beam in XTD2 on Thursday evening.

Access to XSDU2 being clarified - separate entry will be posted.

Karabo:

System running but performance of the GUI still sluggish - CAS was not able to reproduce the problem, still under investigation
Second server has been set up as workaround.
In parallel experts are working on a definitive solution
Deployment today on a new beckhoff device for the decodification of PLC and interlock error messages.

Vacuum:

Systematic tests of ion pumps ongoing. in XTD 9 two controllers/beckhoff terminal for pumps were wrongly connected (exchanged). To reconnect the controllers in the right orders the loop has to be interrupted. This will be done after the vacuum test on the XGM is done.

PBLM/Motion Loop:

There is a resource problem when trying to read out the cameras. This problem will be reconsidered once the general issue with the GUI server is solved.
ASilenzi will discuss offilen with Natalia the acceptance test definition for PBLM
List of errors and relative solution concerning ELMO absolute encoder implementation being worked out by AE/CAS

SPB Tunnel components

Error message fore the Renishaw encoders is coming from wrond specifics of the interpolator between encoder and input terminal. This is due to wrong datasheet provided by the company. This is not a problem for reading of the encoders and their reliability. Error signal is currently ingnored
The same problem also affects FXE components
The cameras operated at 1Hz deliver delayed images. 1/3 Hz operation (almost live mode) is OK - probably related to GUI memory issues

FXE

Encoders of Loop1 have the same error message as SPB - the encoder positions are reliable - local components test can go on (e.g. OGT1)
Encoders of loop2 are not powered - AE to investigate today

Tasks for today:

XGM vacuum test and solution implementation of pump restart in Karabo
GUI server resource problem ongoing
Deployment of a beckhoff device for the decodification of the error messages.
Find a solution for the Renishaw encoder problem SPB Tunnel/ FXE Loop1
Investigation of the FXE Loop2 encoders power
Continue testing of ion pumps in XTD9
Vacuum signal alarms to be checked by AE,Harlad, Janni
Implement and test the CRLs in XTD2 in the Beckhof loop for test.

Tasks for tomorrow:

Vacuum signal alarms to be checked by AE,Harlad, Janni
Implement and test the CRLs in XTD2 in the Beckhof loop for test.
Checking of the absober for first lasing (inc. filter chamber).

The XSDU2 area will be open (as XTD2) tomorrow between 9 ans 12. To register the access, please send also in this case a mail to T.Haas.

Fertigmeldung phase II cables SPB Optics Hutch D.02 and SPB laser hutch D.13

ACTEMIUM has delivered the handover document for the phase II cables installed in D,02 and D.13. The beckhoff crates can be connected to the cables in these areas of SASE1.

Access to XTDs on Thursday

This Thursday the XTDs will be accessible with ZZ procedure. Access can be requested via T. Haas.

Status after the 9.30 Meeting

Vacuum (M. Dommach):

Systematic tests of ion pumps in XTD9 almost finished - ongoing
Interface with MPS has to be tested
Problem with the fast valve in XTD2 and 9 - it is possible to close it remotely but not possible to open it again. Checking of the cabling is ongoing by Vacuum group
After restart of the vacuum loop many pumps in XTD 9 and 2 show an unknown error and some of them are not readble - CAS working on that.

PLC (P. Gessler, T. Freyermuth):

Implementation of shutters in Karabo ongoing -XS3 shutter in the VAC loop now. XTD9 to do.

Karabo (A. Silenzi):

GUI stability improved with a temporary solution. This should be enough to guarantee the proper functioning of images acquisition.
Update of devices list ongoing
Library for motors (base device), elmo and technosoft, being worked out.

Plans for today:

XGM work on gas injection
Bring back Karabo and clear XGM error codes
Perform systematic test of DPS in XTD9

Other topics:

No 09:30 meeting tomorrow - next meeting will be on Wednesday

Status after 9.30 Meeting

General

The latency observed by SPB in the GUI client has been almost completely fixed. Still some "freezing" from time to time remains.
DAQ tests in FXE with LPD worked for 12/16 of the detector. Trigger's signal stability issue being addressed by AE.
XTD 9 and 10 will remain open from 25.07. at 7:00 to 26.07. at 14:00
On Wednesday afternoon the machine will be prepare to lase with 30 bunches in order to allow the radiation measurements in SASE1 Exp. Hall on Thursday and Friday.

XTDs

Fast valve in XTD9 still has issues with noise. Issue will be addressed tomorrow with AE support.

SPB

Integration of the turbo pumps in the VAC loop ongoing.
Chiller control being included in Loop 5
Firmware for the AGIPD hv power supply was updated. The spikes during ramping are gone
SMARACT motors being implemented. (depend on Loop 13)
Work on in vacuum interlock in SPB will probably start this week.
Some problems with Karabo (While measuring profiles the level scaling does not work, Region of interest values wrong) have been reported to CAS.

Status after 9.30 Meeting

General

Today the machine is being prepared to lase with 30 bunches in order to allow the radiation measurements in SASE1 Exp. Hall on Thursday and Friday.
Today GUI will be down for a couple of minutes at 11.30 to allow modifications of the servers for bugs fixing.
Specific SASE1 areas will be off limits during the test, scheduled for SPB OPT 27.07.17 12:00-16:00 , SPB EXP 28.07.17 9:00-15:00 and FXE 03.08.17 9:00-15:00. Please address Eric Boyd's presentation for details. A mail reminder will be also sent to notify the beginning and ending of the tests.
The machine will be running with 30 bunches during the test and with 1 bunch during the night. Since no EPS is implemented, screens and sensitive components need to be taken out during the 30 bunches operation.
New version of Karabo ready to be deployed. The time slot when this will happen will be discussed on Friday,
Screens controlled by Karabo cannot be read from DOOCS. The Karabo/DOOCS interface problems will be be addressed with high priority by CAS.

XTDs

Fast valve in XTD9 still has issues with noise. Issue will be addressed tomorrow with AE support.

FXE

The detector group is updating the firmware for the LPD to fix the trigger signal problems. A test on just 1 quadrant will be done today together with ITDM
Interlock implementation in Loop 1 ongoing.

SPB

Integration of the turbo pumps in the VAC loop ongoing. Interlock possible.
Chiller control being included in Loop 5 - Crate has been modified this morning. Loop will be updated checking that this will not stop the turbo pumps.
SMARACT motors being implemented. First motor is ready to be tested today after hardware problems have been fixed.
Work on in vacuum interlock in SPB ongoing.
The ELMO controllers for the motors of the liquid jet manipulator in Loop 13 should be brought and ready by CW 32. AE working on that.

Status after 9.30 dispatch Meeting

General

SRP was not able to measure any radiation even inside D.09 and D.02. This Thursday the radiation test in D.05 (FXE) will be performed.
Tomorrow XTD9 open 7:00 ==>14:00

FXE

Gotthard detectors to be tested in Karabo

SPB

Problem with Smaract motors has been identified. Loop to be updated.
Cameras trigger for the sample environment is now working. Software issues. CAS is addressing that.
Vacuum interlock for Loop 5 (AGIPD) is being implemented. There will be a meeting soon (today?) with AE to check the interlock definitions for AGIPD and continue work on this loop.
Focus characterization with CRLs ongoing. Beam size reached: <10 um

Tunnels
Fast valve fake triggering has not been reported in the last week end.
In case of fake triggering of the fast valves, there is now a procedure to open them again remotely. Please check the related entry posted by M. Dommach.
Some problems with the cameras that stopped working have been reported by A. Koch. That could be just a matter of restarting the GUI. To be checked.

FXE Interlock definitions have been changed and this reflects in the update of loop 1 and 2. Ongoing.
Working on Alio Stages integration for Mono in the tunnel. To start this week.
SPB Loop 12 – Smaract motors and ELMO controllers implementation ongoing
SPB Loop 5 is up including the AGIPD related systems. AGIPD interlock will be added once low level tests done
SPB Loop 13 - Phase 2 cables for some modules are at different positions compared to eplans. SPB wants to leave it and rather swap module positions and hand in a change request.
Cables for interface to Person Interlock laid. Connections to the MPS system to be done this week.

DET
LPD – Time server was set up. Images collected from 1 quadrant. Firmware upgrade for the other quadrants ongoing.
AGIPD – Detector being tested in Hera south. KWrona will start the tests of DAQ in Hera today or tomorrow.

CAS
2.1.13 Karabo version update is being agreed with the interested parties. It takes approx. 1-3 hrs.

Status report after dispatch meeting 13:30

Tunnels (J. Eidam, J. Gruenert)

SASE3:
- 1st section is not recovered yet, ongoing.
- Diff. pumping XGM is being installed
- Leak tests ongoing
- MCP manipulator XTD2: company got the training - repair ongoing
- Absorber was moved ==> has to be realigned by MEA
- A. Koch is commissioning the imagers. Some bugs were found and will be solved with the new release of Karabo.
- MCP, FEL-Imager: Crates are there, AE still has to understand and define the priorities of their work
- No access to the tunnels with visitor cards! (Access to tunnels requires rebreather training, knowledge about procedures, warnings, dosimetry etc.)
- Exchange of gotthard detector in XTD 9 ongoing.
SASE1:
- XGM Inihbit system will be brought up tomorrow. VAC loop needs to be brought down . Will be notified with a message in the logbook.

Electronics (P. Gessler)

Responsibles
- SASE1: Bernard Baranasic
- SASE3: Jan Tolkiehn
- Instruments SASE1: Tobias Freyermuth
Fast valve terminals exchanged today.
Documentation by WP74 + Vacuum is not consistent ==> AE will provide an Excel-sheet to be used, which should give a consistent picture of all terminals, connections etc. Ongoing

ITDM (K. Wrona)

Both online & offline-filesystems have to be updated. Postponed to TOMORROW (16.10.): Online-FS will be down, offline-FS will be worked on continuously, should not be down, but at reduced performance, since update will be server by server.
DAQ-Upgrade: still running Karabo 2.1.12 on detectors, which requires lots of restarts. UPGRADE: TOMORROW 13==>16 to Karabo v.2.1.16

Software (H. Santos, A. SIlenzi):

Karabo 2.1.18 being tested. Test to continue for a couple of days before deployment.
S. Brockhauser will report on the status of the scan tool this Friday.

Goto page Previous 1, 2, 3 ... 23, 24, 25 ... 28, 29, 30 Next

ELOG V3.1.4-7c3fd00