Group Selection Page HELP DOC Controls Group Data Analysis Group ITDM WP76 DET Vacuum XRO EEE WP78 WP74 FXE SCS SPB MID HED SQS SXP Sample Environment Photon Commissioning Team Operation PSPO XO TS
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Joint Operation and Readiness Logbook, Contact: T.Haas, A. Violante, G. Wellenreuther, Page 4 of 30  Not logged in ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
ID Date Author Location Severity Subject
  537   25 Jun 2018, 09:45 Hugo Silva SantosGeneralInformationKarabo 2.2.4 released (mainly for ITDM, AE and DET) + Deployment dates

Dear All,

I would like to inform that we released last Friday (22.06) the Karabo 2.2.4 version. We got around five days delayed to prioritize issues regarding Karabo-DAQ integration and MDL devices that were found in the last steps of the test cycle. However, fortunately, those issues are solved. 

Based on this run, the new dates that we (CAS) are proposing to start the deployment in operation will be:

  • SA3: 28.06, Thursday (CAS references: Chris, Wajid, and Valerii)
  • SA2: 02.07, Monday (CAS references: Chris, Wajid, and Valerii)
  • SA1 + SPB + FXE: 03.07, Tuesday (CAS references: Alessandro, Dennis, Gabriele, Wajid)
    • SPB deployment should not impact AGIPD firmware test week, as agreed before.
  • BKR: 28.06? (Wajid and Jan, could you please agree offline?)
  • LAS, SQS and SCS dates have to be agreed with the respective contact persons.

-
Hugo

 

  536   20 Jun 2018, 14:12 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 13:00

SASE1 Experiments

Due to the interlock PLCs reprogramming, the emergency off circuit will be triggered tomorrow and power cuts will occur. Due to that, the AGIPD in SPB is being put into a safety state. ITDM and AE are informed. FXE and Optical lasers were also informed. The order is: Experiment in a safe state==>PLCs switched off==>Servers switched off. 

SPB will put all the motors to zero position and the parameters will be saved by a script prepared by CAS (AS). 

Next week Mon-Wed the planned power cut will also occur. 

SCS

  • Control room being put into operation - New cables to arrive soon. AE addressing the so far known issues.

AE

  • SASE2 MOV2 loop is up.
  • Master timing will be down due to maintenance by DESY on Thursday, from 9:00 to 17:00

ITDM

  • GPFS ran out of space in the week end. ITDM/DET/CAS meeting tomorrow. Then the experiments will be presented a way to go.
  • On Monday we would like to replace the dns/dhcp server in the balcony room. In principle we have the secondary one working at DESY but experience shows everything gets slow and in particular Karabo may have big problems. Due to the power off in SASE1, this will be scheduled for monday afternoon.

CAS

  • Issues with DAQ observed. Conflict on data aggregators. This was found to be a typo in the DAQ code. Fixing of the DAQ required but the release of Karabo 2.2.4. can proceed.
  • Coordinated motion for SPB, SCS, SQS: Follow up tests by AE to be done this week.
  535   18 Jun 2018, 15:42 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SCS

  • Tests  continue - List of issues sent to AE. Bernard will take care of that as soon as he's back.

SQS

  • Issues with KB motors (ELMO) reported and will be addressed by AE. Also the list was sent to BB.
  • Loop 4  is not up yet. Will be probably by next week.
  • MC2 motors configuration procedure has to be clarified with AE. SCS also interested. Meeting will take place. 

AE

  • SASE2 MOV2 loop will be up on Thursday. XGM already up.
  • Master timing will be down due to maintenance by DESY on Thursday, from 9:00 to 17:00

 

ITDM

  • GPFS ran out of space in the week end. This was due to the calibrated data from the users proposal being completed in the last weeks. ITDM and DET groups will meet to discuss size and strategies to deal with calibrated data.

XRO

  • In SASE3 a slit is reported tomove left if the command "move to right" is given. CAS to look into that.

CAS

  • Karabo version 2.2.4. Test being ultimated.
  • Issues with DAQ observed. Work ongoing to fix it.
  • Coordinated motion for SPB, SCS, SQS: There was a demo (successful). RBean will send minutes around.

General

  • Next Dispatch meetings will take place on Wednesday, at 13:00.
  534   13 Jun 2018, 11:07 Hugo Silva SantosGeneralInformationRe: Status report after dispatch meeting 09.30

Hi All,

Just adding more info:

CAS

  • Karabo version 2.2.4rc. being tested with PLC 1.27. Waiting for AE support to update the test environment to 1.28 (Jan, Nerea, and Bernard are aware).
Adriano Violante wrote:

SCS

  • Tests  continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

  • Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
  • Loop 3 - Motors test continue
  • Loop 2 is up and running.
  • Loop 4  probably will run tomorrow.
  • In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

AE

  • EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

  • Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

  • Karabo version 2.2.4. being tested with PLCs.

EET

  • Firmwares fo all the SCS loops have been generated.
  • Firmwares up to Loop 5 for SQS have been generated.

 

  533   13 Jun 2018, 10:48 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SCS

  • Tests  continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

  • Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
  • Loop 3 - Motors test continue
  • Loop 2 is up and running.
  • Loop 4  probably will run tomorrow.
  • In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

AE

  • EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

  • Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

  • Karabo version 2.2.4. being tested with PLCs. 

EET

  • Firmwares fo all the SCS loops have been generated.
  • Firmwares up to Loop 5 for SQS have been generated.
  532   11 Jun 2018, 10:27 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SPB

  • Beam delivery issues.
  • After several days of usage the Karabo GUI slows down. Servers are reported to have crashed a couple of times in the last shift.
  • Same situation with data aggregators. Once per shift they have to be restarted. This in known to ITDM.
  • Interlock system seems to work fine now, but the impression is that the people in BKR cannot correctly interpret the interloch signals coming from the experiments. The issue relates to the Interlock panels to be developed in BKR (Issue list, Friday's Operation meeting)

FXE

  • Reading the logbook it seems that some problems with the optical laser's shutter forced to stop the laser operation at about 3 AM. 

SCS

  • Tests  continue

SQS

  • Loop 1 one more ion pump will be implemented today by AE
  • Loop 3 - Configuration of the rest of the  motors expected today
  • Loop 2 should be up by tomorrow according to AE
  • Loop 4 and 5 are supposed to go up in the next 3 weeks 

AE

  • EPS in SASE2 should be ready today

ITDM

  • FXE reported missing images from Gotthardt detector in the H5 files. The data does not reach the DAQ. Tests have to be made, but cannot be done while SPB has beamitime. Tests foreseen for tomorrow.
  •  

CAS

  • Some issues with scanning tool reported - working on it
  • Some issues concerning data acquisiion from imagers were reported from SPB  - Valerii on it.
  531   06 Jun 2018, 09:46 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SPB

  • Ready for the Users run. No big issues.
  • The first 2 weeks of the shutdown will be used for cleaning and rearrangement of the hutch by SPB, in preparation of the mirrors instalaltion. PLCs update and Technical interventions by TS or external companies have to be carried out in this period. 

AE

  • XTD6 VAC loop running.
  • CRL XTD1 working
  • Loop 2 SQS experiment still not running due to small issues being solved.
  • Loop2 SCS was restarted yesterday after some fix

ITDM

  • Gpfs should be fixed by now.

 

  530   04 Jun 2018, 10:46 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SCS

  • List with the problems encountered so far during the end to end test was sent to AE. Bernard will get the info and eventually inform EET if problems with cables are found.

SQS

  • Elmo motors for KB system were ocnfigured and are being moved right now. If problems arise AE will be contacted.
  • It was not possible to control the turbo pump controllers via PLC. Bernard will have a look. In general also tirbo pump cables can and will be checked with the test box.
  • Loop 2 is being prepared including the MC2 motors. Should be ready by tomorrow.
  • SQS would like to start setting up the digitizers environment in Karabo. uTCA crates can be used on the HW side. AE acan support that.

AE

  • XTD6 VAC loop is being patched. Small problems with the cabling of the CRL occured but solved. 

XRD

  • XTD6 components being cabled. 2 Imagers still need to get the EPS signals. in 1/2 weeks ready for local tests.

ITDM

  • One IBM storage server failed. Reasons not clear. No impact on the performance of the system.

CAS

  • This Week VAC loop in XTD 6 will work.
  • For the MOV 2 loop still  EPLAN exports are needed.
  • MOV1/MOV2 interfaces have to be defined.
  529   30 May 2018, 10:55 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

AE

  • CRLs in XTD1 being implemented
  • GATT and EPS loop in SASE3 being updated

SCS

  • Tests continue - No issues so far

SQS

  • ELMO controllers have to be configured for the mirror motors test. Slit motors have lower priority. Cables can be tested w/o being physically connected to the motors, but to configure the ELMO controllers the cables have to be connected to the real motors. Tomorrow AE is available to support these activities.
  • Basler cameras set up and working.

ITDM

  • SPB AGIPD test today or tomorrow

CAS

  • Hot fix for SPB released yesterday. This will address the issue of crushings when instantiating a new device. Has to be tested with SPB. 
  • XTD 6 will have no electricity for the racks today since Wille is completing the cabling. MTCA being shut down accordingly.
  528   28 May 2018, 10:28 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

General

  • For the upcoming emergency off test in SASE1 SPB and FXE should propose  a possible date in the next shutdown to be then discussed with AE.
  • The machine is down due to the failure of a cryocompressor. Beam will be probably coming back tomorrow morning. Tomorrow morning will be also confirmed whether SPB and FXE cold use the beamtime for the user run.

SPB

  • In the week end there was a failure of the PP laser shutter. It was possible to find a very temporary solution to circumvent the problem.
  • The beam has been unstable for most of the time. No drift, mostly gitter. Reasons are unclear. The instability was also measured with the XGM in XTD2. SPB is analysing the data collected to be able to quantify the instability. 
  • DAQ tests for AGIPD were planned for tomorrow but the request is to shift them to today to take advantage of the missing beam.

FXE

Some issues were reported: LPD readout of the data, Fast digitizers mainly. The details have to be clarified with the DET and ITDM groups.  

AE

SQS loop 3 is up and running. 

SCS

  • Tests continue

SQS

  • FMB Oxford will be working on the mirror tank starting form tomorrow until next week. Acces to the hutch is limited.
  • There will be a meeting with AE to discuss the set up of the motors for testing (especially for the mirrors). SCS will also participate. 

ITDM

  • SASE3 projects updated. Performance increased.
  • Possible to move the AGIPD test for SPB to today

CAS

  • Camera device had to be reinstanciated 4 times during the week end. Investigation ongoing.
  527   23 May 2018, 12:44 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

General

  • Please remember to send the list of the tasks you planned for the next shutdown to Monica Areste by the end of the week.
  • Please remember to send the list of the alignment works needed until July to Gerd Wellenreuther.

AE

SQS loop 3 is almost ready. 1 crate is currently in the AE lab but should be installed soon. The loop 3 is needed by SQS for the motor tests with FMB Oxford next week.

SQS

  • Due to a pump that stopped working (Caverion) this morning there was water on the floor of the laser hutch and part of the experiment hutch.  Some cables (not powered and packed) saw water. These are being checked. Also lasermet boxes on the wall saw water. Lasermet is checking that.
  • For the testing SQS requested the test box to AE 

SPB

  • Final DAQ test ongoing. Slow data acquisition OK. This is needed for the users program this week. Fast data acquisition (inc. AGIPD) will be tested next week.

ITDM

  • Patch released for Slow data acquisition FXE/SPB. LPD acquisition OK. AGIPD issues being investigated.

 

  526   16 May 2018, 13:52 Adriano ViolanteGeneralInformationMPS update Info from AE

The MPS comm for SASE 2 and FXE were tested and the only remaining part is to test the SPB to MPS changes. Tomorrow morning a  timeslot will be agreed between AE and SPB.

  525   16 May 2018, 11:37 Adriano ViolanteGeneralInformationSummary of the dispatch meeting 09.30

 

​​AE

  • SASE3:
    • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail ==> AE updated the loop. Nerea is making sure that the MPS signals are enabled in the BKR 
    • Stepper motor for the mirror bender was found to be miswired. Has been already fixed. Similar issue (inverse wiring) was found in SASE1. Cannot be fixed now. It will be done in the next shutdown. For the time being one has to drive the morot the opposite way.
  • SASE2:
    • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail ==> DONE. Not clear if some motor parameters for XRO devices were saved before. D. La Civita and CAS are checking that
  • SPB:
    • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people. ==> DONE. 
  • FXE:
    • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

ITDM

  • DAQ has to be upgraded to thelast verison of Karabo (just hotfix). Will be checked with Gero what is the best way to go.

 

  524   14 May 2018, 10:17 Adriano ViolanteGeneralInformationSummary of the dispatch meeting 09.30

SQS

  • Loop1 still being worked by AE. Not yet up and running. 
  • Loop 3 is needed to be up and running in 2 weeks since the mirror motors ar eincluded in this loop and they are needed for the final check with FMB.

SCS

  • Bernard has been added to the SCS operation logbook and he is getting al the updates concerning commissioning.
  • Testing of the Loop 1 motors ongoing., 2 out of 20 motors have problems. Being investigated.
  • Some connectors names are reported not to be consistent with the EPLANS. Martin Teichmann will check that with EET.

​​AE

  • SCS:
    • loop 2 running. Bernard has recommended that you start with commissioning of KBS_VDM, _HFM, _BFL, VFM motors because they are using MC2MOTOR soft device class (controller is Elmo).
    • Crate BKF00157 can be removed and refresh terminal will be added without interrupting the PLC - only equipment connected to this crate will not be available during this period of change.
  • SASE3:
    • -Request from Andreas Koch about interlocks for MCP and LED Power in SQS/SCS (this installation is unique in SASE3). For that purpose we have to update EPS, MOV2 and Master loop in SASE3. PRC will be triggered by E-Mail
  • SASE2:
    • -We want to update SASE2 in order to get beam permission interlocks properly. Therefore we need to update, EPS, Vacuum loop and Master loop. PRC will be triggered by E-Mail
  • SPB:
    • -Today we will deploy loop 12 and 13 and Master loop. In the Master loop we will add beam permission conditions, so we want to test it and therefore we will need to open all the valves in SPB as well as the last ones in the Tunnel. Has to be decided when it is possible with the SPB people.
  • FXE:
    • -All loops updated past week. Waiting for answer from FXE about when we can test the same beam permission conditions as in SPB. To be agreed with FXE.

 

DET

  • Working of the optimization of the python piece of code allowing the display of online images. Right now 30 images per second rate is possible.

ITDM

  • GPFS server fixed.

CAS

  • XTD1 CRLs will be integrated iin the control system in KW 22
  • 20.07 is the goal date for beam in MID and HED optics Hutch.  

 

  523   09 May 2018, 10:09 Adriano ViolanteGeneralInformationSummary of the dispatch meeting 09.30

SPB

  • Test on the mirrors peltier showed some issues due to the fact that the test boxes were different from the actual stuff. R.Bean will clarify it with FMB.
  • Loops 12,13 update to be done.
  • Magnetic turbos control: the serial communication has some issues. Being checked with AE and CAS

SQS

  • Loop1 cabled up. Some names haad to be corrected in the firmware. As soon as the export is delivered by EET to AE the formware can be uploaded, Probably today.
  • Upstream turbo pumps running using controlers in the rackrooms but still no PLC control.

SCS

  • Some issue with the movement of the motors found. AE is on it.
  • 1 crate does not have enough power to power up all the terminals. Will be solved today.

AE

  • The valves will be included in the interlock systems (SASE1 issue) - Loop ready to be updated - The MPS has to be disabled before updating the loop to avoid tripping of the beam.
  • FXE loops update started

 

DET

  • Hotfix for AGIPD ready to be deployed. Can start today. Details will be sorted out with SPB.

CAS

  • SASE3 Soft Mono is now technically commissioned. PBLMs next in the line.
  • XTD1 CRLs will be integrated iin the control system in KW 22
  • XTD6 priority on the MID beamline. 

General

It is not clear if and when the experiment in SASE1 could get beam during this week. Has to be checked with the PRC.

  522   07 May 2018, 10:36 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SPB

  • Loops 5,7,11, update and Karabo deployed.
  • Loops 12,13,8 update ongoing.

SQS

  • Loop1 cabled up. Today the formware will be uploaded.
  • Loop 2 will probably be ready on Wednesday

SCS

  • Out ov 50 valves tested 5 showed problems. Investigation ongoing to understand where is the problem (PLC, Cables, Valves)
  • In Loop 2 a problem with a crate containing Elmo controlers was found. AE on it. Next loop to be tested will be Loop 4.
  • 1 crate does not have enough power to power up all the terminals. CR already prepared. Will be solved soon.

AE

  • By the middle of the week the valves will be included in the interlock systems (SASE1 issue)
  • The EPS interlock definitions for the tunnels in SASE2 are being finalized by AE, XPO, XPD

ITDM

  • Failure of one IBM server for online storage. Fixing ongoing.

DET

  • Online preview and charge injection issues are being solved restarting the DAQ and re-deployng Karabo.
  521   02 May 2018, 11:17 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

SPB

  • After re-compiling the relative karabo device, the configuration procedure of the AGIPD is now smoothand reliable.
  • PLCs update ongoing. AGIPD is in a safe state to allow the PLC update.

SQS

  • Meeting today at 11:00 with AE, EET, CAS to define the next steps of the end to end tests.

AE

  • SASE2 tunnels - Only VAC conditions are transmitted from the EPS loop to the MPS. All the other conditions were supposed to be not necessary for first lasing and were not implemented. Nevertheless this was not clear for the people working in the BKR (especcially for diagnostics) during the last days. It has to be discussed with the PRC (H. Sinn this week) whether these have to be implemented on a short time. In pronciple AE can do it if requested. Otherwise the definitions will be implemented in the next shutdown.

ITDM

  • Failure of one IBM server for online storage. Thanks to the implemented redundancy this will not affect the operation. The performances have to be checked.   IBM has been contacted.
  • On monday evening ITDM OCD received a call from BKR because of some problems with the DAQ in the tunnels. It seems that this was due to the fact that the system was not properly tested by the "users" before. Seems to be solved now.
  520   30 Apr 2018, 09:51 Adriano ViolanteGeneralInformationStatus report after dispatch meeting 09.30

CAS (C. Youngman)

Following the Technical commissioning phase for the SASE2 tunnel components, the need of a technical coordination activity is strongly pointed out. This is in particular important to make sure that the proper final tests are performed by the components owner  before e.g. first lasing attempts.

 

SQS (T. Baumann)

Lopp 1 being brought up. First tests to be started soon. The CAS and AE contact persons should be in involved in the tests as early as possible.

 

 

 

 

  519   27 Apr 2018, 10:21 Adriano ViolanteGeneralInformationLasing SASE3

INFO

As reported by the PRC (J. Gruenert) during the operation meeting this morning, due to the first lasing attempt in SASE2 no lasing of SASE3 is to be expected until the middle of next week.

  518   25 Apr 2018, 18:10 Alessandro SilenziGeneralInformationRe: Status report after dispatch meeting 13:00

Regarding CAS Support for SASE2 Lasing:

CAS is happy to support in the frame of the On Call Duty system as it has been provided in the past.

Adriano Violante wrote:

SPB

  • Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
  • Smaract motors for PAM are needed and Nerea is on it.
  • It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
  • During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions  have to be checked next week when HSinn is back. 
  • During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it. 
  • Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
  • Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

  • Access  of XTD1 today @14:00 to fix some EPS switches
  • SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

AE

  • SCS exp: first loop running. 2 creates missing but loop is up.
  • MPS connection tunnels SASE2 done

ITDM

  • Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
  • MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

 

ELOG V3.1.4-7c3fd00