30.09. from 08:00 until 01.10. 08:00 - Andreas Koch 9-6443

01.10. from 08:00 until 02.10. 08:00 - Joakim Laksman 9-5630  

Tobias Freyermuth and me worked on the system and as of last night, we found two new issues between Karabo and the PLC interlock:
 - one is a Karabo issue from the Interlock PLC interaction, the Karabo device waits for a PLC response that does not come when the PLC is interlocked (i.e. Karabo wants to open a valve but the PLC correctly refuses and ignores the command)

 - one is a PLC issue, the interlock is not triggered in all cases after testing (e.g. a ion pump out of set point does not closes a gate valve closure as specified).

The first one was reproduced with the BeckhoffSim simulation. It can therefore be fixed by CAS without dependencies from AE (except for the next integration test).

 I reproduced the behaviour of the PLC on beckhoffSim. This reproduces the same behaviour seen in the tunnel devices (devices hang).

Implemented the fix and verified the correct behaviour with beckhoffSim.

Fixes will be verified next Monday with the tunnel system.

Have a nice weekend

Alessandro

ps: For future reference:

beckhoffSim  tags/1.1.0-1.4  (implements the recent plc behaviour (framework tag N/A Could not find tobias))

beckhoffDevices tags/2.0.1-1.4 hangs

 Karabo to PLC communication has been tested successfully in the SA1 Vacuum installation.

in attachment:

OK.png -> Initial Status, all OK

Master Component list is being updated.

https://docs.xfel.eu/share/proxy/alfresco/api/node/content/workspace/SpacesStore/1c6c10cd-d741-410d-bd05-bb00944514dc/MasterEquipmentList.xlsx?a=true

Best regards

Alessandro

 Vacuum loop:

- Interface between PLC and Karabo is not functioning.

The fault currently resides in the PLC behaviour, the self-description sent out is badly formatted (due to a buffer overrun)

The misbehaviour is confirmed, the PLC TCP Client from AE is also ailed by the same problem.

Motion Loop:

- As of Friday, the motor parameters were not completely sent to AE. In attachment the data collected by N. Jardon (AE). Input needed for PSLIT motors.

2017.02.13 19:30 Status:

- Self-Description back in shape.

- Bad SelfDescription reproduced in BeckhoffSim (Karabo implemented PLC communication) and solved in BeckhoffCom (Karabo "PLC Client").

- The build status of the Firmware is not ok. roughly 90% of the PLC soft devices are in error. It is a fault in the automatic Firmware build procedure.

Nerea and Jan will work on this until it is fixed.

The PLC is still not completely error free.

It is considerably better performing than the last days:

the current average pressure as measured by the Ion Pumps is 4.21e-09 mBar

there are Currently 96 ion pumps in the Tunnels as seen by the PLC:

3 cannot properly communicate with the PLC

SA1_XTD9_VAC/IPUMP/P10510I

SA1_XTD9_VAC/IPUMP/P11130I

SA1_XTD9_VAC/IPUMP/P12120I

3 have an ambiguous control name:

SA1_XTD2_VAC
SA1_XTD9_VAC2
SA1_XTD9_VAC0
 

And 4 have no channel assigned in the Karabo inteface (CAS to solve this)

SA1_XTD2_VAC/IPUMP/P10340I
SA1_XTD2_VAC/IPUMP/P10350I
SA1_XTD9_VAC/IPUMP/P10370I
SA1_XTD2_VAC/IPUMP/P10360I

The remaining 86 replied and:

Average pressure is  4.210e-09 mBar

Maximum pressure is  2.60-08 mBar

Minimum pressure is  8.90e-10 mBar

The motion loop is as of 18:40 online.

the current status of the devices as seen by the Karabo is shown in the attachment:

In summary 63 devices (29 Motors):

18 in error

- 14 in error 0x8004 (Terminal unavailable)

- 4 in error 0x8003 (Terminal unavailable)

This is the list of motor that are behaving according to karabo

SA1_XTD2_MIRR-2/MOTOR/HMRZ Success State.OFF
SA1_XTD2_PSLIT/MOTOR/BLADE_RIGHT Success State.OFF
SA1_XTD2_IMGFEL/MOTOR/SCREEN Success State.OFF
SA1_XTD2_FLT/MOTOR/TARGET Success State.OFF
SA1_XTD2_MIRR-1/MOTOR/HMRZ Success State.OFF
SA1_XTD2_PSLIT/MOTOR/BLADE_LEFT Success State.OFF
SA1_XTD2_MIRR-1/MOTOR/HMTX Success State.OFF
SA1_XTD2_IMGSR/MOTOR/TARGET_X Success State.OFF
SA1_XTD2_MIRR-2/MOTOR/HMRX Success State.OFF
SA1_XTD2_IMGSR/MOTOR/IRIS Success State.OFF
SA1_XTD2_MIRR-1/MOTOR/HMRY Success State.OFF
SA1_XTD2_PSLIT/MOTOR/BLADE_BOTTOM Success State.OFF
SA1_XTD2_MIRR-2/MOTOR/HMRY Success State.OFF
SA1_XTD2_IMGSR/MOTOR/FOCUS Success State.OFF
SA1_XTD2_MIRR-2/MOTOR/HMTX Success State.OFF
SA1_XTD2_MIRR-1/MOTOR/HMTY Success State.OFF
SA1_XTD9_IMGPII45/MOTOR/SCREEN Success State.OFF
SA1_XTD2_MIRR-1/MOTOR/HMRX Success State.OFF
SA1_XTD2_PSLIT/MOTOR/BLADE_TOP Success State.OFF
SA1_XTD2_IMGPII45/MOTOR/SCREEN Success State.OFF
SA1_XTD2_IMGFEL/MOTOR/FILTER Success State.OFF
SA1_XTD2_MIRR-2/MOTOR/HMTY Success State.OFF
 

Quick feedback during the meeting:

- 63 Devices

- 29 Motors

Errors only in 

- SPB_XTD9_IMGPI

- FXE_XTD9_IMGPI

Url to the readthedocs documentation:

https://goo.gl/vsz9hu

https://in.xfel.eu/redmine/projects/control-support/issues

Short Instructions:

- Add your task

- Select your work package

- Do NOT set the assignee

- Do NOT set the priority

This is the list of data coming directly from the PLC via its TCP client connected to the EPS PLC.

Format:

deviceName  temperature  status errorCode

XTD2

==================================================

SA1_XS3_SHUT/TSENS/T1 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T2 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T3 0.4000000059604645 0xa020 0x0
SA1_XS3_SHUT/TSENS/T4 0.4000000059604645 0xa020 0x0
SA1_XTD2_ABS/TSENS/T1 0.0 0x20 0x0
SA1_XTD2_ABS/TSENS/T2 0.0 0x20 0x0
SA1_XTD2_ATT/TSENS/A1 30.739999771118164 0x20 0x0
SA1_XTD2_ATT/TSENS/A2 1372.0 0x15020 0x0
SA1_XTD2_ATT/TSENS/A3 30.380002975463867 0x20 0x0
SA1_XTD2_ATT/TSENS/A4 31.05000114440918 0x20 0x0
SA1_XTD2_ATT/TSENS/A6 31.130002975463867 0x20 0x0
SA1_XTD2_ATT/TSENS/A7 30.78000259399414 0x20 0x0
SA1_XTD2_ATT/TSENS/A8 30.98000144958496 0x20 0x0
SA1_XTD2_ATT/TSENS/A9 33.5 0x20 0x0
SA1_XTD2_COLA/TSENS/T1 0.0 0x1020 0x0
SA1_XTD2_COLA/TSENS/T2 0.0 0x1020 0x0
SA1_XTD2_COLB/TSENS/T1 0.0 0x20 0x0
SA1_XTD2_COLB/TSENS/T2 0.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_10_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_10_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_1_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_1_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_2_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_2_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_3_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_3_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_4_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_4_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_5_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_5_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_6_T1 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_6_T2 85.0 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_7_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_7_T2 33.5 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_8_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_8_T2 33.5 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_9_T1 33.60000228881836 0x1020 0x0
SA1_XTD2_CRL/TSENS/LENS_9_T2 33.5 0x1020 0x0
SA1_XTD2_DPS1/TSENS/FLOW 2.0300002098083496 0x20 0x0
SA1_XTD2_DPS1/TSENS/RETURN1 33.5 0x20 0x0
SA1_XTD2_DPS1/TSENS/RETURN2 2.0200002193450928 0x20 0x0
SA1_XTD2_DPS3/TSENS/FLOW 33.5 0x20 0x0
SA1_XTD2_DPS3/TSENS/RETURN1 4.160000324249268 0x20 0x0
SA1_XTD2_DPS3/TSENS/RETURN2 4.420000076293945 0x20 0x0
SA1_XTD2_PBLM/TSENS/BLADE_LEFT 32.900001525878906 0x20 0x0
SA1_XTD2_PBLM/TSENS/BLADE_RIGHT 33.5 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_W 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_B4C 0.0 0x20 0x0
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_W 0.0 0x20 0x0
 

XTD9

============================================================

SA1_XTD9_PBLM/TSENS/BLADE_LEFT 24.31999969482422 0x20 0x0
SA1_XTD9_PBLM/TSENS/BLADE_RIGHT 24.810001373291016 0x20 0x0
SPB_XTD9_COLB/TSENS/T1 0.0 0x20 0x0
SPB_XTD9_COLB/TSENS/T2 0.0 0x1020 0x0
SPB_XTD9_DPS1/TSENS/FLOW 33.5 0x1020 0x0
SPB_XTD9_DPS1/TSENS/RETURN1 85.0 0x1020 0x0
SPB_XTD9_DPS1/TSENS/RETURN2 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/FLOW 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/RETURN1 85.0 0x1020 0x0
SPB_XTD9_DPS2/TSENS/RETURN2 85.0 0x1020 0x0
SPB_XTD9_PBLM/TSENS/BLADE_BOTTOM 23.05000114440918 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_LEFT 22.860000610351562 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_RIGHT 23.05000114440918 0x20 0x0
SPB_XTD9_PBLM/TSENS/BLADE_TOP 23.05000114440918 0x20 0x0
SPB_XTD9_SHUT/TSENS/T1 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T2 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T3 0.0 0x1020 0x0
SPB_XTD9_SHUT/TSENS/T4 0.0 0x1020 0x0
FXE_XTD9_COLB/TSENS/T1 0.0 0x20 0x0
FXE_XTD9_COLB/TSENS/T2 0.0 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_BOTTOM 22.780000686645508 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_LEFT 22.8700008392334 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_RIGHT 22.850000381469727 0x20 0x0
FXE_XTD9_PBLM/TSENS/BLADE_TOP 22.93000030517578 0x20 0x0
 

 Now the loop is accessible to Karabo as well with the project SA1_EPS

Here are the updated values for the temperature sensors inn the EPS loop.

Now Thermocouples work!

FXE_XTD9_COLB/TSENS/T1 23.1
FXE_XTD9_COLB/TSENS/T2 23.2
FXE_XTD9_PBLM/TSENS/BLADE_BOTTOM 22.94
FXE_XTD9_PBLM/TSENS/BLADE_LEFT 23.0
FXE_XTD9_PBLM/TSENS/BLADE_RIGHT 23.0
FXE_XTD9_PBLM/TSENS/BLADE_TOP 22.98
SA1_XS3_SHUT/TSENS/T1 4.0
SA1_XS3_SHUT/TSENS/T2 4.0
SA1_XS3_SHUT/TSENS/T3 4.0
SA1_XS3_SHUT/TSENS/T4 4.0
SA1_XTD2_ABS/TSENS/T1 19.5
SA1_XTD2_ABS/TSENS/T2 19.6
SA1_XTD2_ATT/TSENS/A1 30.7
SA1_XTD2_ATT/TSENS/A2 1372.0
SA1_XTD2_ATT/TSENS/A3 30.3
SA1_XTD2_ATT/TSENS/A4 31.02
SA1_XTD2_ATT/TSENS/A5 31.96
SA1_XTD2_ATT/TSENS/A6 31.11
SA1_XTD2_ATT/TSENS/A7 30.75
SA1_XTD2_ATT/TSENS/A8 31.0
SA1_XTD2_ATT/TSENS/A9 31.4
SA1_XTD2_COLA/TSENS/T1 1372.0
SA1_XTD2_COLA/TSENS/T2 1372.0
SA1_XTD2_COLB/TSENS/T1 27.8
SA1_XTD2_COLB/TSENS/T2 1372.0
SA1_XTD2_CRL/TSENS/LENS_10_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_10_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_1_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_1_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_2_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_2_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_3_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_3_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_4_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_4_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_5_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_5_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_6_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_6_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_7_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_7_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_8_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_8_T2 850.0
SA1_XTD2_CRL/TSENS/LENS_9_T1 850.0
SA1_XTD2_CRL/TSENS/LENS_9_T2 850.0
SA1_XTD2_DPS1/TSENS/FLOW 20.3
SA1_XTD2_DPS1/TSENS/RETURN1 20.3
SA1_XTD2_DPS1/TSENS/RETURN2 20.1
SA1_XTD2_DPS3/TSENS/FLOW 44.2
SA1_XTD2_DPS3/TSENS/RETURN1 41.6
SA1_XTD2_DPS3/TSENS/RETURN2 44.2
SA1_XTD2_PBLM/TSENS/BLADE_LEFT 32.8
SA1_XTD2_PBLM/TSENS/BLADE_RIGHT 33.7
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_B4C 19.7
SA1_XTD2_PSLIT/TSENS/BLADE_BOTTOM_W 19.4
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_B4C 18.6
SA1_XTD2_PSLIT/TSENS/BLADE_LEFT_W 18.6
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_B4C 19.3
SA1_XTD2_PSLIT/TSENS/BLADE_RIGHT_W 19.2
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_B4C 19.4
SA1_XTD2_PSLIT/TSENS/BLADE_TOP_W 19.4
SA1_XTD9_PBLM/TSENS/BLADE_LEFT 23.68
SA1_XTD9_PBLM/TSENS/BLADE_RIGHT 24.13
SPB_XTD9_COLB/TSENS/T1 22.9
SPB_XTD9_COLB/TSENS/T2 1372.0
SPB_XTD9_DPS1/TSENS/FLOW 850.0
SPB_XTD9_DPS1/TSENS/RETURN1 850.0
SPB_XTD9_DPS1/TSENS/RETURN2 850.0
SPB_XTD9_DPS2/TSENS/FLOW 850.0
SPB_XTD9_DPS2/TSENS/RETURN1 850.0
SPB_XTD9_DPS2/TSENS/RETURN2 850.0
SPB_XTD9_PBLM/TSENS/BLADE_BOTTOM 23.21
SPB_XTD9_PBLM/TSENS/BLADE_LEFT 23.01
SPB_XTD9_PBLM/TSENS/BLADE_RIGHT 23.19
SPB_XTD9_PBLM/TSENS/BLADE_TOP 23.12
SPB_XTD9_SHUT/TSENS/T1 1372.0
SPB_XTD9_SHUT/TSENS/T2 1372.0
SPB_XTD9_SHUT/TSENS/T3 1372.0
SPB_XTD9_SHUT/TSENS/T4 1372.0

 

I have deployed the Karabo master (the HEAD of our development) in the control network for the Vacuum loop device server.

During last weeks Karabo was not able to cope with the data coming from the PLC for this loop (our biggest to date)
and would become unresponsive after 5 to 10 minutes.

The current system is online since 3.5 hours and is still responsive.

Please note:
This is a partial release and the GUI might show unwanted behaviour.
Please leave the vacuum loop untouched during the weekend.

In attachment you can find an example of unwanted behaviour and the DPS gui

Thank you very much and have a nice weekend!

Alessandro

The karabo services will be down for an upgrade for an hour.

The access will resume at 13:30.

Please find in attachment the dowload bash script for the latest Karabo.

if your system has a "karabo" directory, this script will not overwrite it, but the update will fail.
In this case, remove or rename the karabo directory.

To use it:
1. download the script in a directory of your choice.

2. open a terminal and execute it with
   $ bash PATH_TO/download.sh

Where PATH_TO is the path where the download script was saved.

The EtherCAT loop for the vacuum PLC is interrupted at crate 1.6 (XTD2).

This triggered all Beckhoff Terminals  default behaviour from this point on (i.e. Valves Closed, Digital Outputs low, etc.).

The GUI server has been redeployed and a second one has been added.

In order to access those:

please log out of the gateway and log in again using the usual command

ssh desyusername@exflgateway -L 44444:sa1-br-sys-con-gui1:44444

or

ssh desyusername@exflgateway -L 44444:sa1-br-sys-con-gui2:44444