Tunnels closed today and tomorrow

- AE updates Vacuum and motion loop in the morning

- V. Bondar, J. Eidam and H. Sinn test vacuum in the afternoon (emphasis on understanding interlock conditions)

- A. Koch tests XTD2 imagers in the afternoon

See also  https://in.xfel.eu/redmine/projects/sase1/agile_board

Problems solved on Friday last week:

- Arms 2 & 4 of SRA are now moving. Apparently spindles had mechanical resistance that resulted in error condition. After moving a few times, this appears to be gone

- One of the problematic ion pumps had the serial line disabled. However, other lines 

 Tunnel closed today. Might re-open around 14:00

Note that that we re-convene briefly at 17:00 today in MCR (Mock-up Control Room)

Vacuum Loop:

Status:

• Large number of ion pumps not responding properly (2 in XTD2, 63 in XTD9). There is the suspicion that the parameters on some ion pump controllers might be set incorrectly, such as baud rate or the bus address. The generated error code indicates a bad address.
• Test in the lab requires Ion pump controller and PCL. No setup currently readily available. Check in the tunnel required. Could take place between 1200 and 1300 in XTD2 with a ZZ. (Update: B. Racky thinks she will be done around 1400 and would prefer access after this time)
• Preparations needed to include XGM 

EPS Loop:

Status:

• Temperature sensors return null value. Suspected firmware bug has been fixed and EPS loop updated by TF
• EPS to MPS interface requires testing. Should be done tomorrow

Motion Loop:

Status:

• Fixes have been applied to the MC2 lib. Tested for TECHNOSOFT driver. Should now work also for PBLMs . Tests do be done starting Wednesday

Imagers

Status:

• Camera driver for SR imager not working
• Camera driver for FEL driver working for imager in XTD2 but not for XTD9

Tasks for today:

• E. Eidam, V. Bondor, , J. Tolkiehn H. Sinn go through all ion pumps in the Silenzi list
• E. Eidam, T. Freyermuth check pump controller settings in the tunnel
• A. Koch and W. Ehsan work on Photonics Camera driver for FEL imager

Tasks for tomorrow:

• M. Planas, T. Maltezopoulos, N. Jardon, J. Eidam test XGM vacuum integration in XTD2 (not XTD9)
• N. Gerasimova, N. Gerasimova and 1 firmware expert test PBLMs in the tunnel
• T. Freyermuth tests EPS -> MPS interface

Other topics

• The two signals from MCP to EPS shall be called MCP-upstream and MCP-downstream

This is the list of data coming directly from the PLC via its TCP client connected to the EPS PLC.

Format:

deviceName  temperature  status errorCode

XTD2

==================================================

SA1_XS3_SHUT/TSENS/T1 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T2 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T3 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T4 0.4000000059604645 0xa020 0x0
SA1_XTD2_ABS/TSENS/T1 0.0 0x20 0x0
SA1_XTD2_ABS/TSENS/T2 0.0 0x20 0x0
SA1_XTD2_ATT/TSENS/A1 30.739999771118164 0x20 0x0
SA1_XTD2_ATT/TSENS/A2 1372.0 0x15020 0x0
SA1_XTD2_ATT/TSENS/A3 30.380002975463867 0x20 0x0
SA1_XTD2_ATT/TSENS/A4 31.05000114440918 0x20 0x0
SA1_XTD2_ATT/TSENS/A6 31.130002975463867 0x20 0x0
SA1_XTD2_ATT/TSENS/A7 30.78000259399414 0x20 0x0
SA1_XTD2_ATT/TSENS/A8 30.98000144958496 0x20 0x0
SA1_XTD2_ATT/TSENS/A9 33.5 0x20 0x0
SA1_XTD2_COLA/TSENS/T1 0.0 0x1020 0x0
SA1_XTD2_COLA/TSENS/T2 0.0 0x1020 0x0
SA1_XTD2_COLB/TSENS/T1 0.0 0x20 0x0
SA1_XTD2_COLB/TSENS/T2 0.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_10_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_10_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_1_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_1_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_2_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_2_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_3_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_3_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_4_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_4_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_5_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_5_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_6_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_6_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_7_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_7_T2 33.5 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_8_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_8_T2 33.5 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_9_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_9_T2 33.5 0x1020 0x0
SA1_XTD2_DPS1/TSENS/FLOW 2.0300002098083496 0x20 0x0
SA1_XTD2_DPS1/TSENS/RETURN1 33.5 0x20 0x0
SA1_XTD2_DPS1/TSENS/RETURN2 2.0200002193450928 0x20 0x0
SA1_XTD2_DPS3/TSENS/FLOW 33.5 0x20 0x0
SA1_XTD2_DPS3/TSENS/RETURN1 4.160000324249268 0x20 0x0
SA1_XTD2_DPS3/TSENS/RETURN2 4.420000076293945 0x20 0x0
SA1_XTD2_PBLM/TSENS/BLADE_LEFT 32.900001525878906 0x20 0x0
SA1_XTD2_PBLM/TSENS/BLADE_RIGHT 33.5 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_W 0.0 0x20 0x0
 

XTD9

============================================================

SA1_XTD9_PBLM/TSENS/BLADE_LEFT 24.31999969482422 0x20 0x0
SA1_XTD9_PBLM/TSENS/BLADE_RIGHT 24.810001373291016 0x20 0x0
SPB_XTD9_COLB/TSENS/T1 0.0 0x20 0x0
SPB_XTD9_COLB/TSENS/T2 0.0 0x1020 0x0
SPB_XTD9_DPS1/TSENS/FLOW 33.5 0x1020 0x0
SPB_XTD9_DPS1/TSENS/RETURN1 85.0 0x1020 0x0
SPB_XTD9_DPS1/TSENS/RETURN2 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/FLOW 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/RETURN1 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/RETURN2 85.0 0x1020 0x0
SPB_XTD9_PBLM/TSENS/BLADE_BOTTOM 23.05000114440918 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_LEFT 22.860000610351562 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_RIGHT 23.05000114440918 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_TOP 23.05000114440918 0x20 0x0
SPB_XTD9_SHUT/TSENS/T1 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T2 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T3 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T4 0.0 0x1020 0x0
FXE_XTD9_COLB/TSENS/T1 0.0 0x20 0x0
FXE_XTD9_COLB/TSENS/T2 0.0 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_BOTTOM 22.780000686645508 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_LEFT 22.8700008392334 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_RIGHT 22.850000381469727 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_TOP 22.93000030517578 0x20 0x0
 

 Now the loop is accessible to Karabo as well with the project SA1_EPS

MPS informed me that they hit some problems and that they still need 30 mins to 1 h until we can go to XTD.  

Interlocks in the northern fan are broken! 

since tunnel access is now possible 

J. Eidam, V. Bondor, T. Freyermuth checked Ion pump controller settings in XTD2 and XTD9.

Many controllers were found to have incorrect settings. All pumps in XTD2 now respond and give pressure readings. There remains the problem that the PLC is looking for the wrong channel in the case where 2 pumps are connected to one controller (Is 1 & 3 and PLC looks for 1 & 2). J. Eidam now checks channel numbers on all controller. T. Freyermuth then regenerates the firmware. A. Silenzi set up pressure recording for pumps P10050I, P10060I and P10200I to run overnight. 

Here are the updated values for the temperature sensors inn the EPS loop.

Now Thermocouples work!

FXE_XTD9_COLB/TSENS/T1 23.1
FXE_XTD9_COLB/TSENS/T2 23.2
FXE_XTD9_PBLM/TSENS/BLADE_BOTTOM 22.94
FXE_XTD9_PBLM/TSENS/BLADE_LEFT 23.0
FXE_XTD9_PBLM/TSENS/BLADE_RIGHT 23.0
FXE_XTD9_PBLM/TSENS/BLADE_TOP 22.98
SA1_XS3_SHUT/TSENS/T1 4.0
SA1_XS3_SHUT/TSENS/T2 4.0
SA1_XS3_SHUT/TSENS/T3 4.0
SA1_XS3_SHUT/TSENS/T4 4.0
SA1_XTD2_ABS/TSENS/T1 19.5
SA1_XTD2_ABS/TSENS/T2 19.6
SA1_XTD2_ATT/TSENS/A1 30.7
SA1_XTD2_ATT/TSENS/A2 1372.0
SA1_XTD2_ATT/TSENS/A3 30.3
SA1_XTD2_ATT/TSENS/A4 31.02
SA1_XTD2_ATT/TSENS/A5 31.96
SA1_XTD2_ATT/TSENS/A6 31.11
SA1_XTD2_ATT/TSENS/A7 30.75
SA1_XTD2_ATT/TSENS/A8 31.0
SA1_XTD2_ATT/TSENS/A9 31.4
SA1_XTD2_COLA/TSENS/T1 1372.0
SA1_XTD2_COLA/TSENS/T2 1372.0
SA1_XTD2_COLB/TSENS/T1 27.8
SA1_XTD2_COLB/TSENS/T2 1372.0
SA1_XTD2_CRL/TSENS/LENS_10_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_10_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_1_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_1_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_2_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_2_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_3_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_3_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_4_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_4_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_5_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_5_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_6_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_6_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_7_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_7_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_8_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_8_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_9_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_9_T2 850.0
SA1_XTD2_DPS1/TSENS/FLOW 20.3
SA1_XTD2_DPS1/TSENS/RETURN1 20.3
SA1_XTD2_DPS1/TSENS/RETURN2 20.1
SA1_XTD2_DPS3/TSENS/FLOW 44.2
SA1_XTD2_DPS3/TSENS/RETURN1 41.6
SA1_XTD2_DPS3/TSENS/RETURN2 44.2
SA1_XTD2_PBLM/TSENS/BLADE_LEFT 32.8
SA1_XTD2_PBLM/TSENS/BLADE_RIGHT 33.7
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_B4C 19.7
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_W 19.4
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_B4C 18.6
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_W 18.6
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_B4C 19.3
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_W 19.2
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_B4C 19.4
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_W 19.4
SA1_XTD9_PBLM/TSENS/BLADE_LEFT 23.68
SA1_XTD9_PBLM/TSENS/BLADE_RIGHT 24.13
SPB_XTD9_COLB/TSENS/T1 22.9
SPB_XTD9_COLB/TSENS/T2 1372.0
SPB_XTD9_DPS1/TSENS/FLOW 850.0
SPB_XTD9_DPS1/TSENS/RETURN1 850.0
SPB_XTD9_DPS1/TSENS/RETURN2 850.0
SPB_XTD9_DPS2/TSENS/FLOW 850.0
SPB_XTD9_DPS2/TSENS/RETURN1 850.0
SPB_XTD9_DPS2/TSENS/RETURN2 850.0
SPB_XTD9_PBLM/TSENS/BLADE_BOTTOM 23.21
SPB_XTD9_PBLM/TSENS/BLADE_LEFT 23.01
SPB_XTD9_PBLM/TSENS/BLADE_RIGHT 23.19
SPB_XTD9_PBLM/TSENS/BLADE_TOP 23.12
SPB_XTD9_SHUT/TSENS/T1 1372.0
SPB_XTD9_SHUT/TSENS/T2 1372.0
SPB_XTD9_SHUT/TSENS/T3 1372.0
SPB_XTD9_SHUT/TSENS/T4 1372.0

 

Dear all,

Tomorrow at first hour, some hardware modules would added in Rack 1.1 and 1.9 in XTD2 in the Vacuum loop. Therefore the communication would be interrupted till the firmware is downloaded to the PLC, around 10h-10.30h should be running again to add the XGM XTD2 into the loop. Although, I will post a new entry to let you know.

Thank you,

Tunnel Open.

Vacuum Loop:

Status:

• Communication problem with Ion pump controllers is fixed after adjusting controller settings
• Loop is now down to allow inclusion of XGM

EPS Loop:

Status:

• Temperature sensors return reasonable values after EPS loop fix
• EPS to MPS interface is working but fiber to central desy interface still lacking

Motion Loop:

Status:

• Should be up for testing PBLMs and Imagers

Imagers

Status:

• Test of FEL imager ok, but hampered by occasional slow response of the system
• SR Imager needs work on the driver

Tasks for today:

• Include XGM in Vacuum loop (N. Jardon, M. Planas, T. Matsopoulos, V. Bondar)
• Test updated MC2 lib with PBLM in the tunnels (N. Gerasimova, M. Jidda)
• Work on camera driver for SR imager (A. Koch and W. Ehsan)
• Test motion system of the imagers in the tunnels (A. Koch and W. Ehsan)
• Monitor stability and responsiveness of GUI Server and Log Servers
• Set up monitoring of temperatures and pressures overnight
• Push DESY to get fiber connection to MPS (P. Gessler)

Tasks for tomorrow:

• Terminals need to be added and firmware adjustments made in motion loop (take loop down as early as possible for one hour)
• Test EPS inputs to MPS system with DESY people
• Power down crate 1.15 for grounding. Could take longer than one day

Other topics

• N. Jardon needs complete info of motor names in rack 1.18

Dear all,

The XGM XTD2 is already into the Vacuum loop and the loop is running. We would continue with the interlock tests today in the afternoon, but would not affect to other parts of the Vacuum loop.

Also was requested to change the Karaboo names of the devices in XGM XTD2, therefore a firmware updated would be necessary and would be performed today in the afternoon after VAC-MPS communication tests.

Thank you!

I expect that some of us may need the weekend as a reserve (e. g. mirror installation). So we should probably say no. Let's discuss more tomorrow at 09:30 

 Tunnel Open.

Mirrors:

Status:

• Vacuum will be closed on M1
• M2 installation tomorrow or latest on Monday

Vacuum Loop:

Status:

• Loop is up
• XGM was integrated but tests needed on the interlocks. Hence valves to DPS now in manual mode (overrides interlock)
• Interface to MPS has been successfully tested
• Need to test fast valve connection to VAC and to MPS (Status re cabling was unclear this morning)
• A few pumps still  have communication problems (to be discussed between AE and CAS)

EPS Loop:

Status:

• AE did most of the missing wiring
• Description of EPS interlock still missing. It is planned to start with just one imager in for first beam commissioning
• One fiber connection from Balcony room to DESY still to be done by DESY

Motion Loop:

Status:

• Problem on ELMO motors with linear drive persists. Needs additional testing

Imagers

Status:

• Both motors working on FEL imager
• Work should now concentrate on SR imager

Karabo:

Status:

• Problem in TCP implementation identified that is suspected to cause the long delays observed e. g. with the imagers. Now needs extensive testing
• Still no data recording achieved
• Stability still needs monitoring

Tasks for today:

• Work on GUI for the vacuum loop (J. Eidam, V. Bondar)
• Test XGM interlocks (N. Jardon, M. Planas, T. Matsopoulos)
• Continue tests with PBLM in the tunnels (M. Jidda)
• Test motion system of the imagers in the tunnels (A. Koch and W. Ehsan)
• Monitor stability and responsiveness of GUI Server and Log Servers (A. Silenzi)
• Set up monitoring of temperatures and pressures overnight (A. Silenzi)
• Push DESY to install fiber connection from Balcony rooms to MPS (P. Gessler)

Tasks for tomorrow:

• Put XGM into the vacuum interlock
• Perform careful testing of all interlock conditions for the vacuum

Other topics

• Regular SASE1 readiness meeting at 09:00 tomorrow in XHQ E1.173
• Need to perform tests with EPS interlock conditions
• Need to test fast valve connection to VAC and to MPS
• Gas system for diagnostics needs to be flushed

I have made an update for the Motion loop requested to test correctly the Attenuators. Now the loop is running again.

Thank you and sorry for inconvenience.

Dear all, I will update EPS loop in order to integrate new hardware changes. I will post a new entry when the work is done.

Thank you!

Dear all, EPS loop is again up and running.

Thank you!

I have deployed the Karabo master (the HEAD of our development) in the control network for the Vacuum loop device server.

During last weeks Karabo was not able to cope with the data coming from the PLC for this loop (our biggest to date)
and would become unresponsive after 5 to 10 minutes.

The current system is online since 3.5 hours and is still responsive.

Please note:
This is a partial release and the GUI might show unwanted behaviour.
Please leave the vacuum loop untouched during the weekend.

In attachment you can find an example of unwanted behaviour and the DPS gui

Thank you very much and have a nice weekend!

Alessandro

Dear collogues,

we have added a hotfix to the framework, to make sure, that no motor can be moved if the feedback signals are missing. This update will be done due to the mentioned issue in the readiness meeting this morning.

We will update the loops again after the weekend, with a proper solution.

The hotfix will send all devices of a loop into error if the feedback of a terminal in the loop is missing.

I will write a new entire in the Logbook, if we are done with the updates.

Cheers,

    Tobias

Dear collogues,

all loops are up and running after deploying the hotfix.

Have a nice weekend,

    Tobias