General Announcement:

- nothing.

CTRL:

- LAS reported the pulse statistics did not work properly: re-started systems due to lost connections.

- OCD call related to HED AGIPD test system: fix ongoing.

- Update on train-ID slow data on XGM: CTRL is in touch with Desy and Theo. Question to SPB and XPD: how much effort is needed for that CTRL and ITDM to place the missing information in the run. Involved parts will discuss an agreement by the end of the week. Steffen H. will prepare a summary to inform XPD.

DA:

- no important news.

EEE:

- no important news.

VAC:

- nothing to report.

XPD:

- nothing to report.

DET:

- nothing special.

ITDM:

- all fine.

XRO:

- Not present.

LAS:

Going fine: Laser pattern issue from yesterday was solved by CTRL.

UNSYS:

- Problem on Monday with quad mover cell18/SA1 took 6 hours on ZZ. It is now fixed.

- A problem with undulator segment on cell36/SA1 needs diagnostic and eventual ZZ access. RCs are contacted.

SPB:

- The samples to be used next week are challenging to prepare. 

FXE:

- Used beam yesterday. No problems to report.

MID:

- all fine.

HED:

- nothing to report.

SCS:

- nothing to report.

SQS:

- all good.

General Announcement:

- User Operations Meeting takes place on next Friday 09:00.

CTRL:

- CTRL would like to excuse itself to people receiving an automatic SMS tonight for notification. This was a test message and now it’s turned OFF (Good side: test for lasers went well).

- ADQ digitizers: two fixes are driven and based for SQS commissioning. CTRL encourage instruments to get in touch with contacts for making the update for the last version.

DA:

- Follow up from Monday / calibration update (on behalf of cal team): it worked as planned. AGIPD seems to run faster now.

- For SQS Remi detectors/digitizers can also be used.

EEE:

- nothing to report

VAC:

- nothing to report

XPD:

- are preparing for FD studies next week

DET:

- Follow-up on LPD online calibration pipeline pointed last Monday’s meeting: work is on progress and is expected to be ready for next week.

- DAQ processing for AGIPD: a minor bug in validation is reported and under investigation. But it is currently working.

- Issue reported for SPB for data aggregators: follow up with CTRL.

ITDM:

- nothing to report

XRO:

- nothing to report

LAS:

- nothing to report

UNSYS:

- nothing to report

SPB:

- not present

FXE:

- Thank to DET and DA for pipeline calibration. Initially it looks promising.

- Last week: digitizer went to wrong configuration. To be fixed with new updated.

- FXE will use the beam during the nights. D. Khakhulin will inform PRC and XO via e-mail.

MID:

- all fine

HED:

- not present

SCS:

- nothing to report

SQS:

- nothing to add

General Announcement:

- Interlock test next 2 weeks starting on monday - from next wednesday the photon tunnels will be closed; Tests are the XHEXP1 take place on 30.06, 01.07 and 02.07 with the last date as reserve; An email is going to be send by FWF to the xo-info@xfel.eu list.

CTRL:

- List of items to be sent to readiness: 1. problems with lima based cameras (Lima library does not work – needs to be updated: instruments contacts shall get in touch); 2. In SA1 there is legacy Undulator Karabo scenes – vector properties have to be adapted; 3. Counter Clocking motor devices upgraded next week.

- A. Koch asks if SA1 was working this morning. Steffen H. says there were updates but must be running now.

DA:

- Next wednesday will be the Offline calibration update deployment

EEE:

- Shutdown activities going well; "Wille" is short in people but scheduling adjustments with FXE and MID is organized.

- If there are short circuits or mal functions in the Beckhoff crates and current is not high enough to trigger the fuse it may be a fire risk. Solution is developed without impact in firmware, just a replacement of fuses. EEE starts today at SPB the changes; Based on this experience a plan will be made for the other systems and EEE will get in touch with groups.

VAC:

- try to get the works finished in the tunnels

- waiting for the Beckhoff crate works

XPD:

- Most of the tunnels works have been done; others are ongoing; No surprises are expected.

- MKK logbook from Desy reporting about electrical issue. XO will check. (XO update at 13:30: automatic mailing system is being commissioned and under testig; no power failure happened at the time).

DET:

- Work with DSSC in SQS going as planned.

ITDM:

- Work currently being done at FXE.

- Reminder of Friday afternoon gateway update.

XRO:

- Not present.

LAS:

- Nothing special to report.

UNSYS:

- Greasing the linear stages for undulators but not in SA2 (missing grease).

- SA1/SA2 after middle plane alignment UNSYS redefined the minimum gap and will be communicated through XO.

- Cell#36/SA2 there is a cable disconnected that need to be fixed.

- SA1/SA2 working with PS tables for extending operating tables for higher photon energies.

SPB:

- Work with EEE changing all the crates (Beckhoff)

FXE:

- Nothing to add

MID:

- Several ongoing infrastructure works are going fine.

- Worried with Beckhoff fuses (reported two years ago by Konstantin). It will be good to hear experience with SPB.

HED:

- Nothing to add

SCS:

- Nothing to add

SQS:

- Nothing to add

General Announcement:

- Power glitch 3 times at night, what brought down the beam for 1,5 hour each time. Please report to XO if you find devices affected.

CTRL:

- Datalogger update to take place today between 16:00-17:00. Please inform CTRL in case of problems with this schedule.

- Aravis camera and bunch pattern decoder to be updated at around 14:30. Please inform CTRL in case of problems with this schedule.

DA:

- OCD side: no new problems.

- Follow up last Wednesday: better is to use the new Karabo with the fix and not to downgrade the Karabo version.

EEE:

- Ticket from SCS was solved.

VAC:

- Some trouble with the pumps in the gas attenuator SA3. It needs to be fixed today in the tunnel with a ZZ (30 mins).

ME.

- Not present.

XPD:

- Are investigating, if something is affected by the power glitch. So far nothing was found.

DET:

- Nothing to report.

ITDM:

- Two issues to follow with MID: 1. Bassler camera seems to be the data wrongly recorded at DQA. Tests need to be done; it’s a camera used for feedback. Ulrike B. will talk offline with K. Wrona.

- Second issue: on Friday there was one ADC from tunnels to be used from MID was requested a bit late. Data comes from the uTCA from Desy network. If this device is going to be used in full, it has to be set inside Desy network.

XRO:

- Not present.

LAS:

- Two incidents: the power glitch at 19:23 yesterday took Amphos in SA1 down. It is back and operational; In SA3 LAS is further investigating the Amphos.

UNSYS:

- No problem with the power glitch.

- The temperature in SA3 Undulators is stable.

SPB:

- SPB imager at XTD9: is not working and has to be changed the outcome needs to be fixed of XPD. Naresh K. will contact SPB.

FXE:

- Not present.

MID:

- All good at MID – not heard on affected systems due to the power glitch.

- Friday afternoon confusion with the fix of Extra Foam bridge. Probably DA is aware.

HED:

- Vacuum system ok after power glitches.

- Nothing to report.

SCS:

- Not present.

SQS:

- Not affected by the power glitch.

General Announcement:

- No general announcement.

CTRL:

- Problem with cameras / DAQ: fixed yesterday and tested at SA2/MID.

- Issues with SA3 vacuum loop: work ongoing and further investigations are needed.

DA:

- Nothing to report.

EEE:

- Cables available for HED to be installed next week – date to be discussed offline.

- Fast electronics side: working on FXE and HED issues;

- Nerea J. reports on the VAC/SA3 issues: instability with devices going into an unknown state. Alessandro S. works at the night shift in the DOC and will try to fix a permanent fix during the night.

VAC:

- SA3 issues are affecting also the gas attenuator (not only general SA3 vacuum loop but also GATT is affected, which has its own loop). There are no physical events - seems to be electronic. Stability issues to connect devices; Seems to be more related to infrastructure (PLCs). Further investigations are needed.

ME:

- Not present.

XPD:

- SPB imager: change setting of motors (it shall work now) and next week when A. Koch is back further investigations will be done.

DET:

- During the night after 02:00AM: SPB AGIPD K-bridge went OFF. Under investigation. Detector recovered this morning.

ITDM:

- Nothing to report.

XRO:

- Not present.

LAS:

- Nothing special to report.

UNSYS:

- Nothing to report.

SPB:

- Issues with wings of Agipd, causing also interlock trigger. Under investigation.

- 11PM last night: laser interlock triggered. It happened also in the past and SPB will get in touch with Safety today.

- XTD9 IMG: currently working but a permanent fix is welcome in the future.

FXE:

- Not present.

MID:

- All fine.

HED:

- Major issues with Agipd detector yesterday (thanks to DOC for help): 1. Could not kill the service and CTRL investigated; 2. Clock and control did not work (it has worked at CFEL last week). The issue was solved by changing the clock controller back to old configuration (EEE helped). Bruno F. comments that the clock control had to be downgraded in this detector (it worked in other detectors, only this one did not work). Further investigations will follow.

SCS:

- Change of speed from the monochromator was not possible. EEE and CTRL are working on it.

- Next week will run with a special train picker / bunch sequence pattern as the one used last week.

SQS:

- Vacuum problem as mentioned.

- Commissioning DSSC for future beamtimes.

FXE: no issues

SPB: no issues

MID: Split & Delay line arriving on Friday, MID aims for a quick integration into instrument / beamline
         EEE informed to switch Hexapod motors from loop3 to loop 7

HED: changing loop1 (JJ X-ray motors) from open to closed loop operation

SQS: KB works, see sometimes spontaneous switching of end switch logics

EEE: MID, SPB, VAC loop updates running this week;
         no triggers for experiments available until end of power off work
         Jörn: cabling for MID is in full swing, for future installations first check phase 1 cabling, for non blocking
                                   HED: cabling
                                   SCS company Witte 30 ccables
                                   FXE: two CRDs not approved yet, no cabling during this shutdown

Controls:
 

SPB:

• Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
• optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished

FXE:

• XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE

MID:

• SDL should enter the hutch by the end of January

SCS:

• pulse (train) picker to be installed in the tunnel by the end of this week

 EEE:

• FXE had been migrated to MC2 lib
  • motors need to be recalibrated
• XTD6 motion part is to be migrated to MC2 lib
• SQS gets interlock updates

CTRL:

• busy with tunnel, additional support for XPD (imagers motion components)

SPB:

• nothing to report

FXE:

• MC2 library motor commissioning on-going

MID:

• SDL motor parameters defined, however one crate is blocked by Wille and 6 motor connectors (Harting material) are still missing
  once fixed, motor paramaters can be uploaded
• WCB (wall connection boxes) in otics hutch are partially finished

HED:

• Wille is working on some crates and the respective Beckhoff loops are down for the moment
• loop updates from Tuesday on

 EEE:

• additional limit switched for XTD6 CRLs (MID) are being installed
• HED is asking for limit switches on their XTD6 CRLs as well --> EEE and HED will investigate when and how
• cable change in FXE starts this week with exchaging wrongly poled SHV cables

CTRL:

• nothing to report

VAC:

• nothing to report

XRO:

• nothing to report

SPB:

• nothing to report

FXE:

• not present due to group retreat

MID:

• Mono done, no severe issues from commissioning weekend

SCS:

• not present

 EEE:

• SQS pnCCD cables have been installed, waiting for crate to connect

CTRL:

• two minor OCD call and one regarding AGIPD
  • SPB not aware that no support was scheduled for AGIPD

 XRO:

• piezo karabo device and firmware in HED mono need to be checked for compatibility

ITDM:

• 2 calls
  • derver down and restarted
  • data aggregator down and restarted
• new RunConfig delpoyed at FXE

DA:

• jupyterhub @SCS showing performance issues --> investigated by DA

SPB:

• AGIPD improvement campaign great success
• EEE --> 2nd C&C worked just fine

FXE:

• BTM tests successful
• Johann MDL motors should be in closed loop
• JF CTRL device crashed

MID:

• AGIPD Flat Fields done
• Data Correction pipeline was down --> 1 day lost
• PPLaser comm this week
• extrafoam issues

SCS:

• nothing to report

 EEE:

• updates for SDL (MID)
• fastADC issue --> don't take more than 40k samples; bugfix in progress

CTRL:

• 8 OCD events
  • mostly deployment isues
  • FXE DA3 overload
  • old version of scantool @ SCS

 XPD:

• unusual behaviour of beam pos measured with XGM
  • cross check done yesterday

ITDM:

• DA crashes --> being investigated

HED:

• tecnosoft terminal issue
  • motor ran into Ty limit (CCW) and got stuck due to tecnosoft logic

SPB:

• interument parked
• power cut --> someon needs to come in
• NJ: what'S that status of the VAC
• KC & RB: safe state, roughing system still running, in the event of a power cut this needs to be looked at

FXE:

• nothing to report
• NJ: VAC in safe state? --> MK yes everything is switched off

MID:

• instrument parked, VAC is running but pumps will be stopped today and partially vented
  • gate valves are closed hopefully the VAC stays

SCS:

• SCS in safe state, all pumps running
• valves around KB are permenantly closed
• in the event of a power cut all other gate valves will close
• loop update done yesterday

 EEE:

• preparation for safe state ongoing
  • discussed

CTRL:

• is master timing going away or not?
• if master timing goes down, some devices
• ICI contacts identify what can be remotely done and what not

• u

ITDM:

• nothing special

HED:

• HED is going to sleep within the next two days

DA:

• focus on long standing tasks

XRO:

• beam transport is going to be parked today

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config Files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

• reached safe state
• rougning pumps still runnning and valves open
• PLCs need to run for monitoring

FXE:

• n

MID:

• absolutely safe
• cooling water connection
• PLCs switched off

SCS:

• S

 EEE:

• MID switched off completely, all other PLCs are running
• MID utca also shut down
• Jörn, plenty of time for additional planning

CTRL:

• broker logs ok from last weekend
• ICI contacts are in discussion instruments

ITDM:

• while instrument schutdown, network and servers are still running

HED:

• safe state, all motors switched off ( parameters saved) all valves close, ion pumps are stil running.

DA:

• work to do

XRO:

• M1s parked on Friday, tunnel seems ok, lenses seem to be fine
• small problems in the GUI,

XPD:

• x-ray diagnostics had been secured last week, i.e. power ramped down. XGM vaccum needed.

FXE:

• Jungfrau chiller switched on, AMPHOS shutter chiller
• OGT3 no pump cart, ion pumps still running, gate valves closed

VAC:

• V0 closed, OCD looking at vacuum twice a day, PLCs needed and running

Currently EuXFEL enforces a hand-off policy for instrument/tunnel hardware. That includes remote control of motors, pumps, etc.. as well as changing config files. If and when this policy is lifted will be communicated during Dispatch and Technical Integration Meetings. Please also don't hesitate and contact XO in case of any questions.

SPB:

• instrument is fine in safe state.
• anyone having a VAC monitoring script connected to an SMS service?
  • SH: comments that CTRL is working on that on a more general scope, but that needs a bit more time

FXE:

• JungFrau checked and powered down.

MID:

• everything is in a safe state.
  • Martin D. reported a VAC issue --> gonna be clarified
    • MD reports a spike in the VAC which might indicate a leak in a bellow

SCS:

• VAC interlock closed
• eplans for hRIXs electronics
• MT wrote a rudimentary webservice for monitoring karabo stuff via a webbrowser --> not to be used until sanctioned by CTRL

 EEE:

• general remark: EEE provides two key systems: utca, and PLCs
  • currently only PLCs for critical systems are covered by OCDs
  • however all other PLCs are kept running, but not necessarily fixed on-site
  • utca not critical --> thus no OCD for that

CTRL:

• all systems running
  • limitations on karabo in the control network as stated in MB message --> finding out what that means for work on CTRL topics

ITDM:

• everything working, dropbox for urgently needed IT equipment available

HED:

• working on scenes / projects, recombining and cleaning up.

DA:

• not present

XRO:

• tunnel components secured, something on CRLs to be clarified with VAC

XPD:

• everything is fine

VAC:

• VAC in tunnel is stable
• new software for vacuum pump carts developed and being rolled out

CTRL:

• Broker Pile Up in 4 topics over the weekend
  • LAS affected in all topics, LA2 even twice
  • SPB affected once
• mitigation startegy worked

EEE:

• nothing to report

VAC:

nothing to report

XPD

• vac systems in the lab had been secured --> secuvac valves installed plus webcams

DET:

• nothing to report

ITDM:

• exflgateway hardware problem on Saturday --> fixed

XRO:

• nothing to report

SPB:

• rough pump turned off, possibly lack of cooling water
  • interlock tripped --> pressures raised
• were able to set remotely

FXE:

• nothing to report

MID:

• everything is fine

HED:

• VAC is fine, ion pumps are on, no roughing pumps running

DA:

CTRL:

• productive meeting with HED
  • CTRL will ask the other instruments if a similar concept could be agreed upon
• discussion with LAS this afternoon with respect to the current hardware policy

EEE:

• HED provided tasks that EEE shall do
• uTCA development ongoing
  • review how digitizers are being used (fastADC, SPdevices, clock & Control)

VAC:

• power glitch

XPD:

• nothing to report

DET:

ITDM:

• nothing to report, no indication of power glitch

XRO:

• no indication of power glitch

SPB:

• nothing to report

FXE:

• nothing to report, problem to read out pressure values
  • Bernd Baranasic: no ion pressure readings on OGT1
    • in PLC it looks ok, but karabo shows 0
  •  

MID:

• meeting with DESY safety colleagues wrt PUMA, going to be bundled with LDM safety
• discussing with Bruno about digitizers
•  

HED:

• see CTRL

SCS

• only valves in loop1 were closed
• every few days a pump is failing, new behaviour
  • JG: comments it is a known behaviour that one of the gauges showed an erratic behaviour (wrongly set epsilon in karabo)
  • GM: answers, what happened on Saturday was slighty different

DA:

CTRL:

• no bigger issues
• announcement: inital simulation environment ready which can simulate BECKHOFF devices
  • shall be opened to all groups for karabo development

EEE:

• continue work
  • shutdown task deadline may be too early
• if people have CRs CRDs, cable issues please submit

VAC:

• water pumps

XPD:

• nothing to report

DET:

ITDM:

XRO:

• nothing

SPB:

• nothing to report

LAS

• waiting for karabo policy

FXE:

• nothing to report

MID:

• nothing to report
• two weeks deadline extensions like EEE, DET

HED:

• nothing to report

SCS

• shutdown list week extension

DA:

CTRL:

• notificagion service device going to be tested this week
  • ITDM: SMS part is operational but needs to be implemented

EEE:

• shutdown task list to be submitted today

VAC:

• no one present

XPD:

• shutdown task list submitted

DET:

• nothing to report

ITDM:

• nothing to report
•  

XRO:

• nothin to report

SPB:

• VAC i

LAS

• w

FXE:

• shutdown list

MID:

• shutdown list
• Jörg and Jörn (EEE) align for shutdown task list

HED:

• aligning with Jörn (EEE) on shutdown taks

SCS

• working on shutdown list

DA:

General Announcement:

• Default beam positions are being implemented for each SASE at the FEL imager to reduce re-alignment offers for the instrumetns after shutdowns and facility development weeks

CTRL:

• minor OCD calls
  • 6541, 6542
• few MC2 motors not working -> target velocity was set to 0
• in SA1 the bunch pattern changes @10 Hz --> data logging system (with timers) gets overloaded ...
  • if this was to become a normal mode of operation, the system would need to be adpated --> as for now bunch pattterns are not logged anymore
• remote user access to karabo
  • soo to be tested with SQS
• inlfuxDB
  • FXE 95% migrated, probably 100% at the end of the day
  • LAS SA1 100% migrated
  • MID, SCS shall reply when migration is possible (causes a short down time of datalogging)
  • SPB, HED shall reply s.a.

DA:

• nothing to report

EEE:

• issue with encoders in SPB --> some errors had been disabled --> checks for cable lengths (OCD call)
• SQS interloop comunication working? EEE, yes, need to investigate
• N. Jardon reminds about the default parameter procedure
• pre shutter in SA3 works commence when Zunaira, Martin and Maurizio

VAC:

• nothing to report

XPD:

• intervention Monday in SA3, PES is operational

DET:

• no issues

ITDM: 

• last week new DAQ deplyoed in HED, FXE and SPB and tunnels, the three other instruments had been delayed
  • MID today, SQS yesterday
• scratch space in gpfs --> running out of space --> data needs to be removed
  • procedure not yet agreed upon
  • all involved parties should identify data that can be removed from scratch
  • 100 TB are left
• internal proposals for commissioning time needed
  • in the future please prepare those in advance

XRO:

• during the weekend, XRO was recommissioning of the beam transport
  • some qctivities were completed on Tuesday, M3 repositioned from damaged spot
  • recommissioning is completed --> however small problems are to be expected over the frst few days, please contact XRO

LAS:

• no issues

UNSYS: 

• cell 33 in SA2 communication crippled, ZZ planned for next monday

SPB:

• reasonably succesful first night with beam, instrument aligned with alignment laser
• tunnel EPS system --> the FXE_XTD9_popin blocked SPB to 1 pulse
  • MPS signals from SPB might be masked by the machine -> needs to be checked
• 1 OCD call to EEE
• dMigration to influxDB can be done

FXE:

• re align beam to FXE pop in
• mono re alignment
• issue with FastValva --> naming convention on the PLC level had been changed, now only capital letters are allowed --> Bernard is looking into this
• tunnel project using step function with motors throws an error on the slot
  • CTRL is workig on a fix already, it happens for very small steps
  • J. Nardon: was also reported by SQS
• all TView devices had to be restarted as they were out of sync

MID:

• alignment went smooth, until diagnostic endstation
  • data loggers are fine today to migrate to influxDB
    • Steffen aims at afternoon
• pre shutter
  • for CRL2 not possible to laign as the pre shutter always moved in
    • XRO looks into it

HED:

• beamline aligned
  • beam profile excellent in fornt of hutch
    • spatial jitter present, and horizontal movement
  • mirror feedback macro not working
• XGM investigation together twith Theo
• IPM didodes are not read out --> comment by Peter, check the timers
•  

SCS:

• no issues

SQS:

• beam from around noon, beamline aligned, set points on imagers need to be re found

General Announcement:

• XO discusses the need for sharing information.

CTRL:

• support calls
  • small movements cause errors on MC2 motors, fis is going to be tested today and it will be communicated when ti is going to beployed
• swithcing to influxDB only logging, if not heard otherwise then this should happen this afternoon (Monday)
  • SCS, SPB, HED, MID
• switching to influxDB in SA1, SA2 and SA3 to follow as well
• notification service updated such, that only more serious messafe are being send out

DA:

• no issues

EEE:

• cable installations in this week at HED, should start from tomorrow, 1.5 days (Jörn)
• few OCD calls,
  • soft mono in SASE3, motor not working
  • mirror
  • FXE motor runs into emergency stop --> fixed in the latest PC framework, but FXE is running an earlier version
  •  

VAC:

• no issues

XPD:

• i

DET:

• Thursday, OCD was not scheduled, despite some calls

ITDM: 

• bug for big detectors in DAQ had been addressed and should be deployed soon
  • this bug seems to also affect some slow detectors
    • only happens when there is no slow data added to the aggregator
• data reduction effort
  • one outcome: record bunch pattern in every run
    • instruments are asked to include the datasource XXX_timing
    • SH comments: in the future the bunch pattern information will be made available in the pipeline

XRO:

• some recommissioning last week
• sometimes mirrors are not moving

LAS:

• no issues

UNSYS: 

• c

SPB:

• good run for the first wek
• trouble with detector, not being able to record darks -> fixed next day
  • issue with a device that checks correct datasources
• FXE imager EPS interlock issue

FXE:

• FXE imager EPS interlock issue

MID:

• successful week
• a couple of OCD calls
  • CTRL, EEE, DET, everything solved
• AGIPD trainID offset by 1 wrt to the XGM
• karabo update --> changes are incompatible with existing scenes
  • e.g. number of digits
  • SH comments that it would be appreciated if those issue were reported to CTRL

HED:

• no issues
• MC2 motor update, is that going to be done today?
  • SH: yes and if possible for HED, then 11:30 - 13:30 would be the preferred time window.

SCS:

• exit slits did not move --> EEE OCD could help
  • SCS requests tthat components are tested before tunnels are closed
• Navitar didn't work
  • focussing lens lost position
• when component software is updated/upgraded, SCS would liek to see a test
• NG comments: motor parameters are sometimes wrong
• If things need to be modified, test beforehand

SQS:

• no issues