Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 FXE SPB HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 27 of 32  Not logged in ELOG logo
ID Date Author Location Severity Subject
  108   06 Apr 2020, 09:50 Frederik Wolff-FabrisGeneralInformationnotes Dipatch Meeting April 6th, 2020
CTRL:
- It was remotely observed that some cameras were consuming too much memory. By remotely re-starting these devices the issue was solved.  
- No calls to the OCD during the weekend.

EEE:
- No issues to report;
- Planning a meeting with Instruments/CTRL/DA to discuss long term tasks that have been working on during this reduced operation period.

VAC:
- Nothing to report; All running in good state.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM:
- Nothing to report.

XRO:
- After the recent power glitch: XRO has checked the basic parameters of special motors which presented problems in the past associated to power glitches;
- At this time, the main basic parameters remain OK (note that only few devices were checked - the ones which presented problems in the past);
- During the EuXFEL restart a deeper look on all devices has to be taken together with the instruments. 

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- Nothing to report.

HED:
- Nothing to report.

	 
  107   01 Apr 2020, 09:19 Andreas GallerGeneralInformationTechnical Integration Meeting 01.04.2020

CTRL:

  • productive meeting with HED
    • CTRL will ask the other instruments if a similar concept could be agreed upon
  • discussion with LAS this afternoon with respect to the current hardware policy

EEE:

  • HED provided tasks that EEE shall do
  • uTCA development ongoing
    • review how digitizers are being used (fastADC, SPdevices, clock & Control)

VAC:

  • power glitch

XPD:

  • nothing to report

DET:

  •  

ITDM:

  • nothing to report, no indication of power glitch

XRO:

  • no indication of power glitch

SPB:

  • nothing to report

FXE:

  • nothing to report, problem to read out pressure values
    • Bernd Baranasic: no ion pressure readings on OGT1
      • in PLC it looks ok, but karabo shows 0
    •  

MID:

  • meeting with DESY safety colleagues wrt PUMA, going to be bundled with LDM safety
  • discussing with Bruno about digitizers
  •  

HED:

  • see CTRL

SCS

  • only valves in loop1 were closed
  • every few days a pump is failing, new behaviour
    • JG: comments it is a known behaviour that one of the gauges showed an erratic behaviour (wrongly set epsilon in karabo)
    • GM: answers, what happened on Saturday was slighty different

DA:

  •  

  Draft   30 Mar 2020, 08:57 Andreas Galler   

CTRL:

  • Broker Pile Up in 4 topics over the weekend
    • LAS affected in all topics, LA2 even twice
    • SPB affected once
  • mitigation startegy worked

EEE:

  • nothing to report

VAC:

nothing to report

 

XPD

  • vac systems in the lab had been secured --> secuvac valves installed plus webcams

DET:

  • nothing to report

ITDM:

  • exflgateway hardware problem on Saturday --> fixed

XRO:

  • nothing to report

SPB:

  • rough pump turned off, possibly lack of cooling water
    • interlock tripped --> pressures raised
  • were able to set remotely

FXE:

  • nothing to report

MID:

  • everything is fine

HED:

  • VAC is fine, ion pumps are on, no roughing pumps running

DA:

  •  

 

  105   25 Mar 2020, 09:32 Andreas GallerGeneralInformationTechnical Integration Meeting 25.03.2020

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

  • instrument is fine in safe state.
  • anyone having a VAC monitoring script connected to an SMS service?
    • SH: comments that CTRL is working on that on a more general scope, but that needs a bit more time

FXE:

  • JungFrau checked and powered down.

MID:

  • everything is in a safe state.
    • Martin D. reported a VAC issue --> gonna be clarified
      • MD reports a spike in the VAC which might indicate a leak in a bellow

SCS:

  • VAC interlock closed
  • eplans for hRIXs electronics
  • MT wrote a rudimentary webservice for monitoring karabo stuff via a webbrowser --> not to be used until sanctioned by CTRL

 EEE:

  • general remark: EEE provides two key systems: utca, and PLCs
    • currently only PLCs for critical systems are covered by OCDs
    • however all other PLCs are kept running, but not necessarily fixed on-site
    • utca not critical --> thus no OCD for that

CTRL:

  • all systems running
    • limitations on karabo in the control network as stated in MB message --> finding out what that means for work on CTRL topics

ITDM:

  • everything working, dropbox for urgently needed IT equipment available

HED:

  • working on scenes / projects, recombining and cleaning up.

DA:

  • not present

XRO:

  • tunnel components secured, something on CRLs to be clarified with VAC

XPD:

  • everything is fine

VAC:

  • VAC in tunnel is stable
  • new software for vacuum pump carts developed and being rolled out

  104   23 Mar 2020, 08:57 Andreas GallerGeneralInformationnotes Dispatch meeting Mar 23 2020

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config Files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

  • reached safe state
  • rougning pumps still runnning and valves open
  • PLCs need to run for monitoring

FXE:

  • n

MID:

  • absolutely safe
  • cooling water connection
  • PLCs switched off

SCS:

  • S

 EEE:

  • MID switched off completely, all other PLCs are running
  • MID utca also shut down
  • Jörn, plenty of time for additional planning

CTRL:

  • broker logs ok from last weekend
  • ICI contacts are in discussion instruments

ITDM:

  • while instrument schutdown, network and servers are still running

HED:

  • safe state, all motors switched off ( parameters saved) all valves close, ion pumps are stil running.

DA:

  • work to do

XRO:

  • M1s parked on Friday, tunnel seems ok, lenses seem to be fine
  • small problems in the GUI,

XPD:

  • x-ray diagnostics had been secured last week, i.e. power ramped down. XGM vaccum needed.

FXE:

  • Jungfrau chiller switched on, AMPHOS shutter chiller
  • OGT3 no pump cart, ion pumps still running, gate valves closed

VAC:

  • V0 closed, OCD looking at vacuum twice a day, PLCs needed and running
  103   18 Mar 2020, 09:26 Andreas GallerGeneralInformationTechnical Integration Meeting 18.03.2020

SPB:

  • interument parked
  • power cut --> someon needs to come in
  • NJ: what'S that status of the VAC
  • KC & RB: safe state, roughing system still running, in the event of a power cut this needs to be looked at

FXE:

  • nothing to report
  • NJ: VAC in safe state? --> MK yes everything is switched off

MID:

  • instrument parked, VAC is running but pumps will be stopped today and partially vented
    • gate valves are closed hopefully the VAC stays

SCS:

  • SCS in safe state, all pumps running
  • valves around KB are permenantly closed
  • in the event of a power cut all other gate valves will close
  • loop update done yesterday

 EEE:

  • preparation for safe state ongoing
    • discussed

CTRL:

  • is master timing going away or not?
  • if master timing goes down, some devices
  • ICI contacts identify what can be remotely done and what not
  • u

ITDM:

  • nothing special

HED:

  • HED is going to sleep within the next two days

DA:

  • focus on long standing tasks

XRO:

  • beam transport is going to be parked today
  102   16 Mar 2020, 09:36 Frederik Wolff-FabrisGeneralInformationNotes Dipatch Meeting March 16th, 2020
FXE:
- Beamline is in safe state;
- Minor Jungfrau calibration issues over last week. In touch with DET and EEE;

SPB:
- Good exercise: experiment run with only 2 users. Successful beamtime with data taken: to be analysed by users;

MID:
- Successful beamtime with good beam conditions and data has been taken;
- Beamline is in safe state and most of staff is in telecommuting;

EEE:
- Start collecting a list of broad activities to take place in case of a long shutdown;
- No major issues to be fixed;

CTRL:
- Few cameras and detectors hiccups – e.g. DAQ re-start bring system back;
- Most of people will run from home office / telecommuting;
- Tests together with EEE/FXE still to be performed this week - to be scheduled;

DET:
- DET is placing a plan to warmed up detectors and be ready for shutdown;
- AGIPD, DSSC are already in safe mode;
  101   10 Mar 2020, 12:44 Frederik Wolff-FabrisGeneralInformationnotes Dipatch Meeting March 9th, 2020
* Users are scheduled to be here this week (4 beamlines in user runs). SPB, HED, MID report several users have cancelled the travel. Beamlines are making mitigation plans 
to run experiments.

FXE:
- Setting up and working with detectors; Working with 4-bounds MONO; Von-Amos detector working;
- Jumps of beam position have been observed and have to be evaluated;
- Jungfrau detector crashed and DET was contacted;

SPB:
- Smooth run after new tuned up on Tuesday;
- Jump on beam position comes from mirrors;
- It was observed a change of number of bunches on Tuesday night: from 2 to 30 for example; XO comment: This time should be not considered as regular beam delivery and 
only parasitic use. Nevertheless, better information on what kind of changes the machine may perform during the shift can help for the future use of parasitic beam.
- AGIPB was successful; also preparation for users; Mitigated troubles with missing user;

SCS:
- Not present

SQS:
- Not present

HED:
- Solving issues with motors together with EEE last week;
- Hexapod vacuum feedthrough: missing flange has to be borrowed from another group  - still to be found;

MID:
- The Pump-probe laser test last week was very successful;
- Re-start of some devices was needed;
- For this week: reduction of user group from 11 to only 3 visitors. Beamtime is a continuation from last year's interrupted time due to the power outage. Local staff is 
mitigating missing users;

EEE:
- Nothing to report

CTRL:
- Nothing to report

ITDM: 
- Not present

VAC:
- Gas attenuator in SQS: not above 10.2 mbar - in investigation. A stress test has to be made and shall be agreed with BKR;

DET:
- Not present

LAS:
- Not present

XPD:
- Not present
  100   04 Mar 2020, 08:49 Andreas GallerGeneralInformationnotes Dispatch meeting Mar 2 2020

SPB:

  • AGIPD improvement campaign great success
  • EEE --> 2nd C&C worked just fine

FXE:

  • BTM tests successful
  • Johann MDL motors should be in closed loop
  • JF CTRL device crashed

MID:

  • AGIPD Flat Fields done
  • Data Correction pipeline was down --> 1 day lost
  • PPLaser comm this week
  • extrafoam issues

SCS:

  • nothing to report

 EEE:

  • updates for SDL (MID)
  • fastADC issue --> don't take more than 40k samples; bugfix in progress

CTRL:

  • 8 OCD events
    • mostly deployment isues
    • FXE DA3 overload
    • old version of scantool @ SCS

 

 XPD:

  • unusual behaviour of beam pos measured with XGM
    • cross check done yesterday

ITDM:

  • DA crashes --> being investigated

HED:

  • tecnosoft terminal issue
    • motor ran into Ty limit (CCW) and got stuck due to tecnosoft logic

 

  99   26 Feb 2020, 09:06 Andreas GallerGeneralInformationnotes Dispatch meeting Feb 24 2020

SPB:

  • nothing to report

FXE:

  • not present due to group retreat

MID:

  • Mono done, no severe issues from commissioning weekend

SCS:

  • not present

 EEE:

  • SQS pnCCD cables have been installed, waiting for crate to connect

CTRL:

  • two minor OCD call and one regarding AGIPD
    • SPB not aware that no support was scheduled for AGIPD

 

 XRO:

  • piezo karabo device and firmware in HED mono need to be checked for compatibility

ITDM:

  • 2 calls
    • derver down and restarted
    • data aggregator down and restarted
  • new RunConfig delpoyed at FXE

DA:

  • jupyterhub @SCS showing performance issues --> investigated by DA

 

  98   17 Feb 2020, 12:02 Frederik Wolff-FabrisGeneralInformationnotes Dipatch Meeting Feb. 17th, 2020
FXE:
Nothing to report: no issues;

MID:
Delay line pumping down; Connecting all devices in preparation to be ready for Friday;

EEE:
Finishing SQS interlock scenes and SPB loops;
ELMO parameters lost themselves (in connection to XRO/Maurizio request). Parameters to be checked this week and next shutdown;
XPD ticket to be checked;
Cabling shall be started for SQS/SCS: waiting for open hutch;

CTRL:
Ticket from XPD (shared EEE/CTRL): dry run did not work at BKR: missing pieces. 
Please instruments check the systems during the day prior Friday when the Test Run starts;
Request from CTRL/Chris: Define guidelines for the users/support on the ticketing and e-log (who has responsibilities to check devices?).

ITDM: 
Ticked from XPD/Liuba: it is a ticket with link (?) to E-log.


* Mc2 motors: XRO needs a list of all motors in order to check configurations. Example: SA3 MONO pre-mirrors with torque set to zero; SA3 exit 
slits with wrong settings.

* CTRL reminds the instruments to check devices during working hours prior Friday when the Test Run starts with beam delivery.
  97   10 Feb 2020, 10:12 Frederik Wolff-FabrisGeneralInformationNotes Dispatch meeting Feb 10th, 2020
SPB:	Nothing to report;

FXE: 	Everything ongoing; nothing special to be reported;

MID:    Finalize motors of split and delay line; MID has to test interlock for vacuum loop update.

HED:	Commissioning of laser heat table. It may need assistance from EEE and CTRL during the week.

SCS:	Nothing to report - Last week's mounted setup will be disassembled in order to mount the standard setup for users;

EEE: 	OCD was called from XRO during beamline alignment of SA2: 1 BMPI motor had a problem and is under investigation - EEE reminds on the new homing position for 
negative direction motion.

CTRL: 	Following reset of train-ID and timing system for today.

XPD: 	Shifted Beam Pointing Studies from A. Koch to Tuesday. 

DET: 	Nothing to report

*machine is currently not running due to filter clean in the cold compressor. Shall be re-started this afternoon and by tomorrow must be lasing.
*Master time reset and train-ID reset this morning. Then EEE (check systems), followed by CTRL (karabo servers re-start) and ITDM (check DAQ operation). LAS and DET are 
informed.
*Wednesday: Network stress test with Cameras by CTRL on Wednesday 12:00-14:30.
  96   22 Jan 2020, 09:16 Andreas GallerGeneralInformationnotes Dispatch meeting Jan 20 2020

SPB:

  • nothing to report

FXE:

  • MC2 library motor commissioning on-going

MID:

  • SDL motor parameters defined, however one crate is blocked by Wille and 6 motor connectors (Harting material) are still missing
    once fixed, motor paramaters can be uploaded
  • WCB (wall connection boxes) in otics hutch are partially finished

HED:

  • Wille is working on some crates and the respective Beckhoff loops are down for the moment
  • loop updates from Tuesday on

 EEE:

  • additional limit switched for XTD6 CRLs (MID) are being installed
  • HED is asking for limit switches on their XTD6 CRLs as well --> EEE and HED will investigate when and how
  • cable change in FXE starts this week with exchaging wrongly poled SHV cables

CTRL:

  • nothing to report

VAC:

  • nothing to report

XRO:

  • nothing to report

  95   15 Jan 2020, 09:09 Andreas GallerGeneralInformationnotes Dispatch meeting Jan 13 2020

SPB:

  • Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
  • optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished

FXE:

  • XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE

MID:

  • SDL should enter the hutch by the end of January

SCS:

  • pulse (train) picker to be installed in the tunnel by the end of this week

 EEE:

  • FXE had been migrated to MC2 lib
    • motors need to be recalibrated
  • XTD6 motion part is to be migrated to MC2 lib
  • SQS gets interlock updates

CTRL:

  • busy with tunnel, additional support for XPD (imagers motion components)

  94   08 Jan 2020, 12:36 Adriano ViolanteGeneralInformationReport of the Technical integration meeting

SPB:

  • Coordinatd motion for MC2 motors is going to be rolled out by EEE soon.

FXE:

  • No issues to report.

MID:

  • SDL tests ongoing. 
  • All SDL cables have been installed and tested apart from 6 cables (still waiting for pins to be delivered). 
  • Various loops update and hexapod integration ongoing.
  • CRLs - EPS loop will be update probably this week. (involves position sensors in the y direction for CRLs in SASE2 and 1. To be done in the framework of the full beam operation project)

 EEE:

  • SA1 VAC loop update. Issues with the eplan export. Discussion ongoing with VAC and XPD.
  • SA2 MOV loop update for imagers MC2 motors. This week a meeting with all the stakeholder will be held (MID, HED, EEE, Controls) and a suitable date (during next week) for the update will be found. PMO will ask MDommach to provide some feedback on the Tunnel closing schedule. As soon as the feedback is available, it will be distrubuted through the readiness mailing list.
  93   18 Dec 2019, 10:06 Adriano ViolanteGeneralInformationReport after tech. Int. meeting 09.30

MID:

  • SDL tests ongoing. Important motors have been tested. Alignment of the mechanics with the alignment laser ongoing. Will be pumped over Christmas.

 EEE:

  • uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
  • SASE2 EPS loop will be updated probably today.
  • SPB/SFX is getting a new CPU.
  • Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

ITDM

  • Meeting this week ITDM/EEE/TS to discuss how to mitigate issues caused by accidents (e.g. cooling water interruption caused by power cut). 

Controls:

  • CY sent a mail several weeks ago to XO and the DH. The topic is emergency procedures during Christmas. No reply. CY announces that in the period 23.12-07.01 the Controls OCDs would not be available.

AoB

  • No dispatch meeting on 06.01. Next meeting will be the Technical Integration one on 08.01.
  92   16 Dec 2019, 09:28 Adriano ViolanteGeneralInformationReport after dispatch meeting 09.00

FXE:

  • uTCA update has been completed by EEE on Friday evening. Information will be passed to controls for the finalization.

SCS

  • The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP.
  • MT will talk to VAC to clarify how to go on with the train picker integration in the tunnel.

SQS

  • uTCA has been updated.
  • A channel in a digitizer is bisbehaving. The issue is being fixed.
  • Steinmeyer stages will be commissioned today with MS.

MID:

  • SDL tests ongoing
  • Hexapod controllers will be moved to the rackroom. Cables have to be pulled and only after the position of the controllers can be chosen.
  • Loop12 will be updated today by EEE

 EEE:

  • uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
  • SASE1 and 3 EPS loop will be updated probably today.
  • PLC in SPB with issues is being worked on.
  • Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.

 XPD:

  • Some issue with the font size in the new Karabo GUI. Controls will have a look.

 Controls:

  • Karabo updates ongoing. SASE3 today until 13:00. SA1, 2 on Wed.
  • MC2 motors updates has been started also in FXE. 
  • SASE3: meeting with Lasermet for the integration of interlock signals in Karabo. Controls has handed over the scenes to the instruments that will optimize them.
  • firmware of the power supply for the AGIPD has to be updated urgently (by Wed.). CY sent a mail to AM. If no reply the power supplies will be switched off for the update.
  91   11 Dec 2019, 09:54 Adriano ViolanteGeneralInformationReport after Tech. Int. meeting 09.30

FXE:

  • New optical table will be installed today. Access to the hutch could be limited.
  • The new Airbox installed by D3 is colliding with the sliding door. It might be problematic to bring the new laser table in. SRP informed. It is being followed up. 
  • uTCA update will be done on Thursday.

SCS

  • The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP. It is not so clear who are the main players so far.

MID:

  • SDL will be completed at the beginning of next week (inc. cables measurement).
  • SDL tests ongoing
  • Hexapod controllers will be moved to the rackroom. KS is waiting for the controller allocation in the racks in relation to the cables length from MFobian. 

 EEE:

  • uTCA puppet migration: in MID it has been done yesterday. Today SQS and SCS will undergo the update. Tomorrow HED and FXE. Next week tunnels.
  • 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. ITDM has to come back to BF to define a date.
  • Update of loops in the tunnels are scheduled for next week
  90   09 Dec 2019, 09:22 Adriano ViolanteGeneralInformationReport after dispatch meeting 09.00

FXE:

  • No major issues.
  • Karabo will be updated to the version 2.7.0. on Friday.
  • uTCA update will be done on Thursday.

SCS:

  • it was forgotten to involve EEE in the PLC update of the train picker in the tunnel. Now the issue is solved.

MID:

  • SDL cabling ongoing.
  • SDL preliminary tests ongoing
  • Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 

 HED:

  • IC1: cheking of the encoder is going on with the help of EEE (Hazem Ali)
  • New interlock definitions for Loop 6 are giong to be delivered with some delay. NJ is informed.
  • Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian. 
  • XPD installed the diamond screens in the pop in monitors in the tunnel.

EEE:

  • uTCA puippet migration  will start today in the optical laser systems. Next weeks it is planned to do it in the tunnel (XPD is informed). Info mail will follow.
  • 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. Info mail will follow.
  • SCS: new set of cables (microscope and others) will be ready for installation tomorrow.

Controls:

  • 13.12. Karabo 2.7.0 deployment will begin.
  • Normal motors will be upgraded to MC2. First tests are foreseen in MID tunnels. XPD and XRO are affected. XPD has been informed in this dispatch meeting. KS will inform MV (XRO). MS (EEE).
  89   04 Dec 2019, 14:18 Adriano ViolanteGeneralInformationSummary of the technical integration meeting

FXE

  • No major issues - PLCs update has been started by EEE (BB)

SCS

  • Hrixs and related components will be delivered on 13 and 14.01.2020. SCS is coordinating the works with TS in order to crane the chambers into the hall (collision with the scaffolding required for HED dipole laser)
  • Puls picker update in XTD10. Requires update of loops. EEE is not informed. JTD follows the issue up.

EEE

  • General work is ongoing.
  • Next week the EPS loops will be updated in the framework of the full beam operation project.
  • Issue with a PLC in SPB crashing after update to Win10. Beckhoff is working to a solution. If this does no work the PLC will be replaced.

Controls

  • A mail will be sent out early next week to announce th deployment schedule of the new Karabo release.
ELOG V3.1.4-7c3fd00