Group Selection Page HELP DOC Controls Group Data Analysis Group ITDM WP76 DET Vacuum XRO EEE WP78 WP74 FXE SCS SPB MID HED SQS SXP Sample Environment Photon Commissioning Team Operation PSPO XO TS
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 24 of 29  Not logged in ELOG logo
Entry   Notes XO Integration meeting 27.04.2020, posted by Frederik Wolff-Fabris on 28 Apr 2020, 16:19 
General Announcement:
- The CTRL group has received 3 Control Network tasks that can be carried out this week;
- Requests from the groups to Tunnel activities from May 4th are being treated and an answer will come in the next couple of days. 
- With respect to Re-start activities XO/XML has 17 group submissions. We thank the groups to submit them. We have not spotted conflicts so far.
- In case of need to contact staff, please use the working mobiles, emergency phone list or group's mailing lists to address any issue.

CTRL:
- First training on the notification service with 5-6 people. Next training is scheduled for today.

DA:
- All good, doing progress without conflicts.

EEE:
- Nothing special; Group is working on the submission of tasks, to prioritize and make schedule for next week. 
- The group is requested to inform XO in case you have re-start tasks needed to be performed in tunnel.
- An electrical safety person will be provided on site by EEE to serve the instruments from next week. EEE is clarifying and shall respect the safety rules. 

VAC:
- Nothing special. Assembling for the tasks that can be done in tunnel. Also CTRL is involved. Progressing normally.

XPD:
- We have submitted the tasks and looking forward to coordinate with VAC, XRO, EEE to plan the tunnels tasks. 

DET:
- Same as XPD.

ITDM: 
- Nothing to report.

XRO:
- We sent re-start task list. The main problem is to evaluate what work can be done due to safety regulations in the case of complex installations.

LAS:
- We are preparing to switch SA1 on beginning of next week and other lasers will follow during the next week (PP and lab lasers). 
- SA2 may delay due to the needed presence of external company.

SPB:
- Nothing special to report; The group believes it has no activities in the tunnel;

FXE:
- Similar when compared to SPB;

MID:
- Preparing the start-up;

HED:
- Preparing for the start-up.
- HED will need people to switch on electric components on the racks.

SCS:
- Nothing special to report.

SQS:
- Nothing special to report. We are ramping up the vacuum system.
Entry   Notes XO Integration Meeting 24.04.2020, posted by Frederik Wolff-Fabris on 24 Apr 2020, 13:48 
Relaxation control network: 
- CTRL and XO remind the groups to submit the planned activities on e.g. scenes, projects, until Fridays. 
- For next week case: please limit any hardware change. Reason is the current limitation on the physical presence of staff. 
Groups will support only emergency and priority tasks. From May 4th the situation will be different. 
- However, we do not want to stop any progress on activities. This also includes work on Karabo that affects critical work like VAC in tunnels.

Tunnel access for the coming weeks:
- A scheduling plan will be made due to the restrictions of allowed people working on tunnel at the same time. 
Plan will be made in order to respect safety rules and minimize number of people working in tunnels. 
- Tunnel access will be a first agenda point from next week.

General:
- For next meetings, we will continue with the small report from the support and instrument groups.
- Tasks to happen in the hutches are decided by the groups. XO/PMO plan tasks in tunnels.
- We encourage groups to start as earlier as May 4th the tasks which were originally planned to be conducted during the summer shutdown. 
If it needs tunnel access, then one must respect the coming schedule.
- XO is looking together with Desy on all tunnel activities. A schedule is being prepared and will be communicated to the groups as soon as possible. 
- In case of delay of activities due to e.g. excess of people in tunnel, prioritization will be made together with XO and Desy.
Entry   Notes XO Integration Meeting 22.04.2020, posted by Frederik Wolff-Fabris on 23 Apr 2020, 10:04 
General Announcement:
- An e-mail was sent yesterday about the relaxation of restrictions on the control network.
- From now on, the XO integration meeting series will happen on Monday and Wednesday at 09:30 and Friday at 11:30.
- These meetings serve two purposes: 1.exchange of information for the re-start phase, and 2. Controls issues. 
- This procedure is set to help the re-start of activities coming from the super-restricted operation mode.
- Groups are asked to send by every Friday to controls-integration@xfel.eu a summary of activities to be carried out during the following week regarding the Control 
Network. 
This helps CTRL and XO to prepare and coordinate the full scope of activities while having an overview on actions that require large scale resources.
- For example, groups and staff who move motors have to be cautious, once reduced number of people will have access to debug devices in case they get stuck.
- This process could be reviewed in the course of next weeks. 

CTRL:
- Training for the Notification service: half of the groups have delegate people to attend it. Other groups please do so at the distributed Doodle link.

DA:
- Request that groups should inform DATA if people on site are required from next week.

EEE:
- Nothing major to report on micro-TCA and technical side.
- Cablings are ready to be installed when allowed to.

VAC:
- Preparing for the start-up.

XPD:
- Preparing for start-up.

DET:
- Same as VAC and XPD.

ITDM: 
- Preparing for start-up.

XRO:
- Nothing to report.

LAS:
- Work together with Desy last week.
- It is planned to test the laser pattern synchronization on next week if possible. 
An e-mail has to be send to controls-integration@xfel.eu with this request prior Friday 11:30AM XO Integration meeting.

SPB:
- Nothing to report.

FXE:
- Not present.

MID:
- Planning the re-start steps for May 4th.
- MID reminds that at least one person from the Safety groups shall attend this series of meetings.

HED:
- Not present.

SCS:
- Nothing to report.

SQS:
- Not present.
Entry   Notes Dispatch Meeting 20.04.2020, posted by Frederik Wolff-Fabris on 20 Apr 2020, 10:59 
General:
- Desy informed a power glitch occurred on Thursday 16th at 04:55AM. TS has not seen any traces of it in the EuXFEL internal GLT.
- From next week the Dispatch meeting will start at 09:30AM.
- An additional Friday's Integration Meeting will happen at 11:30AM. Proper information and invitation will be send soon.

CTRL:
- The notification service for the reduced operation mode is ready for start-up and an e-mail will be send around with instructions. 
- Groups/staff will be requested to subscribe to the lists.
- Training will be conducted on next Wednesday at 15:00 and/or next week. One member of each group is required to attend it.

DA:
- Nothing to report. Working busily from home.

EEE:
- No issues to report

VAC:
- Two pumps located in a gas attenuator were OFF due to the power glitch of last Thursday 16/04 at 04:55AM.

XPD
- No issues related to the power glitch were observed.
- XPD has received devices and they were successfully stored in vacuum / clean rooms. Another devices shall arrive this week.

DET:
- Nothing to report.

ITDM:
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Not present.

SPB:
- Nothing to report.

FXE:
- Not present.

MID:
- Everything is fine.

HED:
- All is fine, all is running.

SCS:
- Not present.

SQS:
- Not present.
Entry   Technical Integration Meeting 15.04.2020, posted by Andreas Galler on 15 Apr 2020, 09:03 

CTRL:

  • notificagion service device going to be tested this week
    • ITDM: SMS part is operational but needs to be implemented

EEE:

  • shutdown task list to be submitted today

VAC:

  • no one present

XPD:

  • shutdown task list submitted

DET:

  • nothing to report

ITDM:

  • nothing to report
  •  

XRO:

  • nothin to report

SPB:

  • VAC i

LAS

  • w

FXE:

  • shutdown list

MID:

  • shutdown list
  • Jörg and Jörn (EEE) align for shutdown task list

HED:

  • aligning with Jörn (EEE) on shutdown taks

SCS

  • working on shutdown list

DA:

Entry   Technical Integration Meeting 08.04.2020, posted by Andreas Galler on 08 Apr 2020, 08:45 

CTRL:

  • no bigger issues
  • announcement: inital simulation environment ready which can simulate BECKHOFF devices
    • shall be opened to all groups for karabo development

EEE:

  • continue work
    • shutdown task deadline may be too early
  • if people have CRs CRDs, cable issues please submit

VAC:

  • water pumps

XPD:

  • nothing to report

DET:

  •  

ITDM:

  •  

XRO:

  • nothing

SPB:

  • nothing to report

LAS

  • waiting for karabo policy

FXE:

  • nothing to report

MID:

  • nothing to report
  • two weeks deadline extensions like EEE, DET

HED:

  • nothing to report

SCS

  • shutdown list week extension

DA:

Entry   notes Dipatch Meeting April 6th, 2020, posted by Frederik Wolff-Fabris on 06 Apr 2020, 09:50 
CTRL:
- It was remotely observed that some cameras were consuming too much memory. By remotely re-starting these devices the issue was solved.  
- No calls to the OCD during the weekend.

EEE:
- No issues to report;
- Planning a meeting with Instruments/CTRL/DA to discuss long term tasks that have been working on during this reduced operation period.

VAC:
- Nothing to report; All running in good state.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM:
- Nothing to report.

XRO:
- After the recent power glitch: XRO has checked the basic parameters of special motors which presented problems in the past associated to power glitches;
- At this time, the main basic parameters remain OK (note that only few devices were checked - the ones which presented problems in the past);
- During the EuXFEL restart a deeper look on all devices has to be taken together with the instruments. 

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- Nothing to report.

HED:
- Nothing to report.

	 
Entry   Technical Integration Meeting 01.04.2020, posted by Andreas Galler on 01 Apr 2020, 09:19 

CTRL:

  • productive meeting with HED
    • CTRL will ask the other instruments if a similar concept could be agreed upon
  • discussion with LAS this afternoon with respect to the current hardware policy

EEE:

  • HED provided tasks that EEE shall do
  • uTCA development ongoing
    • review how digitizers are being used (fastADC, SPdevices, clock & Control)

VAC:

  • power glitch

XPD:

  • nothing to report

DET:

  •  

ITDM:

  • nothing to report, no indication of power glitch

XRO:

  • no indication of power glitch

SPB:

  • nothing to report

FXE:

  • nothing to report, problem to read out pressure values
    • Bernd Baranasic: no ion pressure readings on OGT1
      • in PLC it looks ok, but karabo shows 0
    •  

MID:

  • meeting with DESY safety colleagues wrt PUMA, going to be bundled with LDM safety
  • discussing with Bruno about digitizers
  •  

HED:

  • see CTRL

SCS

  • only valves in loop1 were closed
  • every few days a pump is failing, new behaviour
    • JG: comments it is a known behaviour that one of the gauges showed an erratic behaviour (wrongly set epsilon in karabo)
    • GM: answers, what happened on Saturday was slighty different

DA:

  •  

Entry   , posted by Andreas Galler on 30 Mar 2020, 08:57 

CTRL:

  • Broker Pile Up in 4 topics over the weekend
    • LAS affected in all topics, LA2 even twice
    • SPB affected once
  • mitigation startegy worked

EEE:

  • nothing to report

VAC:

nothing to report

 

XPD

  • vac systems in the lab had been secured --> secuvac valves installed plus webcams

DET:

  • nothing to report

ITDM:

  • exflgateway hardware problem on Saturday --> fixed

XRO:

  • nothing to report

SPB:

  • rough pump turned off, possibly lack of cooling water
    • interlock tripped --> pressures raised
  • were able to set remotely

FXE:

  • nothing to report

MID:

  • everything is fine

HED:

  • VAC is fine, ion pumps are on, no roughing pumps running

DA:

  •  

 

Entry   Technical Integration Meeting 25.03.2020, posted by Andreas Galler on 25 Mar 2020, 09:32 

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

  • instrument is fine in safe state.
  • anyone having a VAC monitoring script connected to an SMS service?
    • SH: comments that CTRL is working on that on a more general scope, but that needs a bit more time

FXE:

  • JungFrau checked and powered down.

MID:

  • everything is in a safe state.
    • Martin D. reported a VAC issue --> gonna be clarified
      • MD reports a spike in the VAC which might indicate a leak in a bellow

SCS:

  • VAC interlock closed
  • eplans for hRIXs electronics
  • MT wrote a rudimentary webservice for monitoring karabo stuff via a webbrowser --> not to be used until sanctioned by CTRL

 EEE:

  • general remark: EEE provides two key systems: utca, and PLCs
    • currently only PLCs for critical systems are covered by OCDs
    • however all other PLCs are kept running, but not necessarily fixed on-site
    • utca not critical --> thus no OCD for that

CTRL:

  • all systems running
    • limitations on karabo in the control network as stated in MB message --> finding out what that means for work on CTRL topics

ITDM:

  • everything working, dropbox for urgently needed IT equipment available

HED:

  • working on scenes / projects, recombining and cleaning up.

DA:

  • not present

XRO:

  • tunnel components secured, something on CRLs to be clarified with VAC

XPD:

  • everything is fine

VAC:

  • VAC in tunnel is stable
  • new software for vacuum pump carts developed and being rolled out

Entry   notes Dispatch meeting Mar 23 2020, posted by Andreas Galler on 23 Mar 2020, 08:57 

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config Files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

  • reached safe state
  • rougning pumps still runnning and valves open
  • PLCs need to run for monitoring

FXE:

  • n

MID:

  • absolutely safe
  • cooling water connection
  • PLCs switched off

SCS:

  • S

 EEE:

  • MID switched off completely, all other PLCs are running
  • MID utca also shut down
  • Jörn, plenty of time for additional planning

CTRL:

  • broker logs ok from last weekend
  • ICI contacts are in discussion instruments

ITDM:

  • while instrument schutdown, network and servers are still running

HED:

  • safe state, all motors switched off ( parameters saved) all valves close, ion pumps are stil running.

DA:

  • work to do

XRO:

  • M1s parked on Friday, tunnel seems ok, lenses seem to be fine
  • small problems in the GUI,

XPD:

  • x-ray diagnostics had been secured last week, i.e. power ramped down. XGM vaccum needed.

FXE:

  • Jungfrau chiller switched on, AMPHOS shutter chiller
  • OGT3 no pump cart, ion pumps still running, gate valves closed

VAC:

  • V0 closed, OCD looking at vacuum twice a day, PLCs needed and running
Entry   Technical Integration Meeting 18.03.2020, posted by Andreas Galler on 18 Mar 2020, 09:26 

SPB:

  • interument parked
  • power cut --> someon needs to come in
  • NJ: what'S that status of the VAC
  • KC & RB: safe state, roughing system still running, in the event of a power cut this needs to be looked at

FXE:

  • nothing to report
  • NJ: VAC in safe state? --> MK yes everything is switched off

MID:

  • instrument parked, VAC is running but pumps will be stopped today and partially vented
    • gate valves are closed hopefully the VAC stays

SCS:

  • SCS in safe state, all pumps running
  • valves around KB are permenantly closed
  • in the event of a power cut all other gate valves will close
  • loop update done yesterday

 EEE:

  • preparation for safe state ongoing
    • discussed

CTRL:

  • is master timing going away or not?
  • if master timing goes down, some devices
  • ICI contacts identify what can be remotely done and what not
  • u

ITDM:

  • nothing special

HED:

  • HED is going to sleep within the next two days

DA:

  • focus on long standing tasks

XRO:

  • beam transport is going to be parked today
Entry   Notes Dipatch Meeting March 16th, 2020 , posted by Frederik Wolff-Fabris on 16 Mar 2020, 09:36 
FXE:
- Beamline is in safe state;
- Minor Jungfrau calibration issues over last week. In touch with DET and EEE;

SPB:
- Good exercise: experiment run with only 2 users. Successful beamtime with data taken: to be analysed by users;

MID:
- Successful beamtime with good beam conditions and data has been taken;
- Beamline is in safe state and most of staff is in telecommuting;

EEE:
- Start collecting a list of broad activities to take place in case of a long shutdown;
- No major issues to be fixed;

CTRL:
- Few cameras and detectors hiccups – e.g. DAQ re-start bring system back;
- Most of people will run from home office / telecommuting;
- Tests together with EEE/FXE still to be performed this week - to be scheduled;

DET:
- DET is placing a plan to warmed up detectors and be ready for shutdown;
- AGIPD, DSSC are already in safe mode;
Entry   notes Dipatch Meeting March 9th, 2020, posted by Frederik Wolff-Fabris on 10 Mar 2020, 12:44 
* Users are scheduled to be here this week (4 beamlines in user runs). SPB, HED, MID report several users have cancelled the travel. Beamlines are making mitigation plans 
to run experiments.

FXE:
- Setting up and working with detectors; Working with 4-bounds MONO; Von-Amos detector working;
- Jumps of beam position have been observed and have to be evaluated;
- Jungfrau detector crashed and DET was contacted;

SPB:
- Smooth run after new tuned up on Tuesday;
- Jump on beam position comes from mirrors;
- It was observed a change of number of bunches on Tuesday night: from 2 to 30 for example; XO comment: This time should be not considered as regular beam delivery and 
only parasitic use. Nevertheless, better information on what kind of changes the machine may perform during the shift can help for the future use of parasitic beam.
- AGIPB was successful; also preparation for users; Mitigated troubles with missing user;

SCS:
- Not present

SQS:
- Not present

HED:
- Solving issues with motors together with EEE last week;
- Hexapod vacuum feedthrough: missing flange has to be borrowed from another group  - still to be found;

MID:
- The Pump-probe laser test last week was very successful;
- Re-start of some devices was needed;
- For this week: reduction of user group from 11 to only 3 visitors. Beamtime is a continuation from last year's interrupted time due to the power outage. Local staff is 
mitigating missing users;

EEE:
- Nothing to report

CTRL:
- Nothing to report

ITDM: 
- Not present

VAC:
- Gas attenuator in SQS: not above 10.2 mbar - in investigation. A stress test has to be made and shall be agreed with BKR;

DET:
- Not present

LAS:
- Not present

XPD:
- Not present
Entry   notes Dispatch meeting Mar 2 2020, posted by Andreas Galler on 04 Mar 2020, 08:49 

SPB:

  • AGIPD improvement campaign great success
  • EEE --> 2nd C&C worked just fine

FXE:

  • BTM tests successful
  • Johann MDL motors should be in closed loop
  • JF CTRL device crashed

MID:

  • AGIPD Flat Fields done
  • Data Correction pipeline was down --> 1 day lost
  • PPLaser comm this week
  • extrafoam issues

SCS:

  • nothing to report

 EEE:

  • updates for SDL (MID)
  • fastADC issue --> don't take more than 40k samples; bugfix in progress

CTRL:

  • 8 OCD events
    • mostly deployment isues
    • FXE DA3 overload
    • old version of scantool @ SCS

 

 XPD:

  • unusual behaviour of beam pos measured with XGM
    • cross check done yesterday

ITDM:

  • DA crashes --> being investigated

HED:

  • tecnosoft terminal issue
    • motor ran into Ty limit (CCW) and got stuck due to tecnosoft logic

 

Entry   notes Dispatch meeting Feb 24 2020, posted by Andreas Galler on 26 Feb 2020, 09:06 

SPB:

  • nothing to report

FXE:

  • not present due to group retreat

MID:

  • Mono done, no severe issues from commissioning weekend

SCS:

  • not present

 EEE:

  • SQS pnCCD cables have been installed, waiting for crate to connect

CTRL:

  • two minor OCD call and one regarding AGIPD
    • SPB not aware that no support was scheduled for AGIPD

 

 XRO:

  • piezo karabo device and firmware in HED mono need to be checked for compatibility

ITDM:

  • 2 calls
    • derver down and restarted
    • data aggregator down and restarted
  • new RunConfig delpoyed at FXE

DA:

  • jupyterhub @SCS showing performance issues --> investigated by DA

 

Entry   notes Dipatch Meeting Feb. 17th, 2020, posted by Frederik Wolff-Fabris on 17 Feb 2020, 12:02 
FXE:
Nothing to report: no issues;

MID:
Delay line pumping down; Connecting all devices in preparation to be ready for Friday;

EEE:
Finishing SQS interlock scenes and SPB loops;
ELMO parameters lost themselves (in connection to XRO/Maurizio request). Parameters to be checked this week and next shutdown;
XPD ticket to be checked;
Cabling shall be started for SQS/SCS: waiting for open hutch;

CTRL:
Ticket from XPD (shared EEE/CTRL): dry run did not work at BKR: missing pieces. 
Please instruments check the systems during the day prior Friday when the Test Run starts;
Request from CTRL/Chris: Define guidelines for the users/support on the ticketing and e-log (who has responsibilities to check devices?).

ITDM: 
Ticked from XPD/Liuba: it is a ticket with link (?) to E-log.


* Mc2 motors: XRO needs a list of all motors in order to check configurations. Example: SA3 MONO pre-mirrors with torque set to zero; SA3 exit 
slits with wrong settings.

* CTRL reminds the instruments to check devices during working hours prior Friday when the Test Run starts with beam delivery.
Entry   Notes Dispatch meeting Feb 10th, 2020, posted by Frederik Wolff-Fabris on 10 Feb 2020, 10:12 
SPB:	Nothing to report;

FXE: 	Everything ongoing; nothing special to be reported;

MID:    Finalize motors of split and delay line; MID has to test interlock for vacuum loop update.

HED:	Commissioning of laser heat table. It may need assistance from EEE and CTRL during the week.

SCS:	Nothing to report - Last week's mounted setup will be disassembled in order to mount the standard setup for users;

EEE: 	OCD was called from XRO during beamline alignment of SA2: 1 BMPI motor had a problem and is under investigation - EEE reminds on the new homing position for 
negative direction motion.

CTRL: 	Following reset of train-ID and timing system for today.

XPD: 	Shifted Beam Pointing Studies from A. Koch to Tuesday. 

DET: 	Nothing to report

*machine is currently not running due to filter clean in the cold compressor. Shall be re-started this afternoon and by tomorrow must be lasing.
*Master time reset and train-ID reset this morning. Then EEE (check systems), followed by CTRL (karabo servers re-start) and ITDM (check DAQ operation). LAS and DET are 
informed.
*Wednesday: Network stress test with Cameras by CTRL on Wednesday 12:00-14:30.
Entry   notes Dispatch meeting Jan 20 2020, posted by Andreas Galler on 22 Jan 2020, 09:16 

SPB:

  • nothing to report

FXE:

  • MC2 library motor commissioning on-going

MID:

  • SDL motor parameters defined, however one crate is blocked by Wille and 6 motor connectors (Harting material) are still missing
    once fixed, motor paramaters can be uploaded
  • WCB (wall connection boxes) in otics hutch are partially finished

HED:

  • Wille is working on some crates and the respective Beckhoff loops are down for the moment
  • loop updates from Tuesday on

 EEE:

  • additional limit switched for XTD6 CRLs (MID) are being installed
  • HED is asking for limit switches on their XTD6 CRLs as well --> EEE and HED will investigate when and how
  • cable change in FXE starts this week with exchaging wrongly poled SHV cables

CTRL:

  • nothing to report

VAC:

  • nothing to report

XRO:

  • nothing to report

Entry   notes Dispatch meeting Jan 13 2020, posted by Andreas Galler on 15 Jan 2020, 09:09 

SPB:

  • Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
  • optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished

FXE:

  • XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE

MID:

  • SDL should enter the hutch by the end of January

SCS:

  • pulse (train) picker to be installed in the tunnel by the end of this week

 EEE:

  • FXE had been migrated to MC2 lib
    • motors need to be recalibrated
  • XTD6 motion part is to be migrated to MC2 lib
  • SQS gets interlock updates

CTRL:

  • busy with tunnel, additional support for XPD (imagers motion components)

ELOG V3.1.4-7c3fd00