Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 FXE SPB HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 25 of 32  Not logged in ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
ID Date Author Location Severity Subject
  148   03 Aug 2020, 09:23 Andreas GallerGeneralInformationNotes XO Integration Meeting 03.08.2020

General Announcement:

  • .

CTRL:

  • few issues on SA3 vacuum server, over the weekend.
    • twice some devices were not running,
    • when restarting Beckhoff loops from the GUI please remember that it takes time until they are back
  • device servers with - in their name. restarting with the service manager cannot cope with this --> fix is underway
  • scheduling:
    • Beckhoff: small movement error fix is underway and it is suggested to update karabo Beckhoff servers later this week.
    • influxDB data logging
      • current version does not allow easy correlation data from two devices in Grafan (web front-end) an update to the next higher version would cure that
        • suggest to update Wednesday morning early
  •  

DA:

  • no issues

EEE:

  • utca OCD calls
    • nothing serious
  • PLC no OCD calls
    • some updates are planned for FXE, MID SCS on Beckhoff

VAC:

  • no issues

XPD:

  • no issues

DET:

  • comm AGIPD detectors
    • detectors are stable, but issues with online processing

ITDM:

  • new version of DAQ had been tested
    • should be deployed this week
  • 1 OCD call for recording data with ADQdigitizer
    • requires coord between CTRL and ITDM and EEE
    • this operation mode had been tested
  • 1 OCD call JUNGFRAU SPB in burst mode --> did not work
    • SH: was the receiver crashing?
    • KW: not clear, nothing was working in that config
    • SH: the problem had been identified at FXE --> fix should be deployed at SPB as well

XRO:

  • update:
    • follwoing up the issue with the attenuator damage in SAx
  •  

LAS:

  • all three laser are running
    • some problems with shutters
      • SQS: shutter in ILH, cooling water refilled --> but interlocks could not be set
        • XO to coordinate discussion with Pouneh

UNSYS: 

  • no issues

SPB:

  • karabo sessions lost connections to GUI server on exflcon026
  • JUNGFRAU receiver crashing
  • DAQ took long time to change state
    • even switching to ignore after run take a long time
    • this relates to JUNGFRAU
  • GOTTHARD specref device crasing multiple times
    • restarting had to be done multiple times
  • optics interlock hutch was observed to be broken on Thursady night

FXE:

  • digitizers at some point stopped
    • connection to boards and triggers
    • BF: this has to be investigated!
  • beam stability
    • mirror cooling
  • EXAFS within 700 eV had been measured
  • PP Laser was very stable

MID:

  • VAC incident in CRL2 --> which closed V0 --> VAC solved the issues
  • due to high pulse energies small damage to attenuator and instrument component had been
  • DAQ update scheduling
  •  

HED:

  • beam is jittering (when collimated 100 um are observed)
  • CRL scene optics hutch is mixed up for one or two motors --> Thomas Preston investigate

SCS:

  • no issues

SQS:

  • interlock / shutter issue
  147   29 Jul 2020, 08:59 Andreas GallerGeneralInformationNotes XO Integration Meeting 29.07.2020

General Announcement:

  • .

CTRL:

  • INfluxDB migration is running smoothly, tomorrow the last topics will be switched to influx only
  • some Basler / Andor cameras not working at 10 Hz is related to the data network --> depends

DA:

  • n

EEE:

  • Company Wille is on-site for hardware installations
    • hRIXS,
    • HED cable installations
    •  

VAC:

  • no issues

XPD:

  • i

DET:

  • commissioning shifts with AGIPD
  • pn CCD at SQS also on commissioning

ITDM: 

  • issue from Monday wrt crippled DAQ when having no sow data in a datasource
    • the fix reported on Monday only works partially --> earliest estimate for fixing is now next week
  •  

XRO:

  • s

LAS:

  • n

UNSYS: 

  • no issues

SPB:

  • OCD for Basler cameras
  • OCD for MC2 stages (problem with encoders, EEE is informed)

FXE:

  • Major issue with the machine wrt to big brother
  • JUngFRAU seems to be more affected by the DAQ glitch
  • GOTTHARD detector should be re-integrated into the DAQ
    • K. Wrona clarifies that only a reconfig of datasources is needed

MID:

  • beam looks unstable
    • horizontal jitter is higher than usual

HED:

  • no issues

SCS:

  • no issues
    • Daniele: is the soft-mono solved?
    • TM comments: the pre mirror is in place piezos are working

SQS:

  • no issues
  • pnCCD commissioning
  146   27 Jul 2020, 09:05 Andreas GallerGeneralInformationNotes XO Integration Meeting 27.07.2020

General Announcement:

  • XO discusses the need for sharing information.

CTRL:

  • support calls
    • small movements cause errors on MC2 motors, fis is going to be tested today and it will be communicated when ti is going to beployed
  • swithcing to influxDB only logging, if not heard otherwise then this should happen this afternoon (Monday)
    • SCS, SPB, HED, MID
  • switching to influxDB in SA1, SA2 and SA3 to follow as well
  • notification service updated such, that only more serious messafe are being send out

 

DA:

  • no issues

EEE:

  • cable installations in this week at HED, should start from tomorrow, 1.5 days (Jörn)
  • few OCD calls,
    • soft mono in SASE3, motor not working
    • mirror
    • FXE motor runs into emergency stop --> fixed in the latest PC framework, but FXE is running an earlier version
    •  

VAC:

  • no issues

XPD:

  • i

DET:

  • Thursday, OCD was not scheduled, despite some calls

ITDM: 

  • bug for big detectors in DAQ had been addressed and should be deployed soon
    • this bug seems to also affect some slow detectors
      • only happens when there is no slow data added to the aggregator
  • data reduction effort
    • one outcome: record bunch pattern in every run
      • instruments are asked to include the datasource XXX_timing
      • SH comments: in the future the bunch pattern information will be made available in the pipeline

XRO:

  • some recommissioning last week
  • sometimes mirrors are not moving

LAS:

  • no issues

UNSYS: 

  • c

SPB:

  • good run for the first wek
  • trouble with detector, not being able to record darks -> fixed next day
    • issue with a device that checks correct datasources
  • FXE imager EPS interlock issue

FXE:

  • FXE imager EPS interlock issue

MID:

  • successful week
  • a couple of OCD calls
    • CTRL, EEE, DET, everything solved
  • AGIPD trainID offset by 1 wrt to the XGM
  • karabo update --> changes are incompatible with existing scenes
    • e.g. number of digits
    • SH comments that it would be appreciated if those issue were reported to CTRL

HED:

  • no issues
  • MC2 motor update, is that going to be done today?
    • SH: yes and if possible for HED, then 11:30 - 13:30 would be the preferred time window.

SCS:

  • exit slits did not move --> EEE OCD could help
    • SCS requests tthat components are tested before tunnels are closed
  • Navitar didn't work
    • focussing lens lost position
  • when component software is updated/upgraded, SCS would liek to see a test
  • NG comments: motor parameters are sometimes wrong
  • If things need to be modified, test beforehand

SQS:

  • no issues

  145   22 Jul 2020, 09:22 Andreas GallerGeneralInformationXO integration meeting 22.07.2020

General Announcement:

  • Default beam positions are being implemented for each SASE at the FEL imager to reduce re-alignment offers for the instrumetns after shutdowns and facility development weeks

CTRL:

  • minor OCD calls
    • 6541, 6542
  • few MC2 motors not working -> target velocity was set to 0
  • in SA1 the bunch pattern changes @10 Hz --> data logging system (with timers) gets overloaded ...
    • if this was to become a normal mode of operation, the system would need to be adpated --> as for now bunch pattterns are not logged anymore
  • remote user access to karabo
    • soo to be tested with SQS
  • inlfuxDB
    • FXE 95% migrated, probably 100% at the end of the day
    • LAS SA1 100% migrated
    • MID, SCS shall reply when migration is possible (causes a short down time of datalogging)
    • SPB, HED shall reply s.a.

 

DA:

  • nothing to report

EEE:

  • issue with encoders in SPB --> some errors had been disabled --> checks for cable lengths (OCD call)
  • SQS interloop comunication working? EEE, yes, need to investigate
  • N. Jardon reminds about the default parameter procedure
  • pre shutter in SA3 works commence when Zunaira, Martin and Maurizio

VAC:

  • nothing to report

XPD:

  • intervention Monday in SA3, PES is operational

DET:

  • no issues

ITDM: 

  • last week new DAQ deplyoed in HED, FXE and SPB and tunnels, the three other instruments had been delayed
    • MID today, SQS yesterday
  • scratch space in gpfs --> running out of space --> data needs to be removed
    • procedure not yet agreed upon
    • all involved parties should identify data that can be removed from scratch
    • 100 TB are left
  • internal proposals for commissioning time needed
    • in the future please prepare those in advance

XRO:

  • during the weekend, XRO was recommissioning of the beam transport
    • some qctivities were completed on Tuesday, M3 repositioned from damaged spot
    • recommissioning is completed --> however small problems are to be expected over the frst few days, please contact XRO

LAS:

  • no issues

UNSYS: 

  • cell 33 in SA2 communication crippled, ZZ planned for next monday

SPB:

  • reasonably succesful first night with beam, instrument aligned with alignment laser
  • tunnel EPS system --> the FXE_XTD9_popin blocked SPB to 1 pulse
    • MPS signals from SPB might be masked by the machine -> needs to be checked
  • 1 OCD call to EEE
  • dMigration to influxDB can be done

FXE:

  • re align beam to FXE pop in
  • mono re alignment
  • issue with FastValva --> naming convention on the PLC level had been changed, now only capital letters are allowed --> Bernard is looking into this
  • tunnel project using step function with motors throws an error on the slot
    • CTRL is workig on a fix already, it happens for very small steps
    • J. Nardon: was also reported by SQS
  • all TView devices had to be restarted as they were out of sync

MID:

  • alignment went smooth, until diagnostic endstation
    • data loggers are fine today to migrate to influxDB
      • Steffen aims at afternoon
  • pre shutter
    • for CRL2 not possible to laign as the pre shutter always moved in
      • XRO looks into it

HED:

  • beamline aligned
    • beam profile excellent in fornt of hutch
      • spatial jitter present, and horizontal movement
    • mirror feedback macro not working
  • XGM investigation together twith Theo
  • IPM didodes are not read out --> comment by Peter, check the timers
  •  

SCS:

  • no issues

SQS:

  • beam from around noon, beamline aligned, set points on imagers need to be re found

 

 

  144   20 Jul 2020, 14:18 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 20.07.2020
General Announcement:
- Regarding Operation, VAC and PRC had a workaround to restore the North branch beam permission on Sunday. 
- XTD9 Turnstiles were not working this morning for ZZ access planned by FXE. 
During the meeting we heard the information that Michael Prollius / SRP has fixed it by rebooting the computer. 

CTRL:
- Since Wednesday CTRL had one OCD call regarding the SA3/GATT gauges and controller configuration.
- SQS server - issues with interactions. Work ongoing.
- INFLUX roll out: test on CAS monitor topic is successful. For the double loggers, there will be an impact of 5-10 minutes down time when login. 
This will happen on FXE (2x10min without data): it is OK for FXE and for LAS today. IF that is successful, CTRL will contact other instruments.

DA:
- Nothing to report

EEE:
- It has been working on the follow-up of HED topics.
- EEE OCD had a call on the SA3/GATT gauge controller.

VAC:
- Severe trouble with SA3 GATT: damage on the gauge controllers was a result of series of problems.
- Bearing of a downstream booster pump make vibration and developed two leaks; then pressure went bad; 
vacuum gauges showed wrong numbers and resulting in electronic damage in the controller cards.
- The booster pump was exchanged on Friday; on Sunday a leak test found a broken vacuum welding and part was removed. 
Currently ZM3 at Desy is fixing it. Expected to be ready for tunnel installation by 10AM.
- Sunday from 13:00: the GATT was in UHV mode to allow beam permission and lasing in SA1 and SA3 up to Monday 6AM.
- Currently XTD10 is open for GATT repair and PES exchange of turbo pump.
- CTRL mailing service worked quite well: an alarm e-mail of GATT fault was sent every 12 hours.

XPD:
- SA3/XGM valve was also OFF on Friday. Martin Dommach will contact Theo while in tunnel.

DET:
- Business as usual.

ITDM: 
- Nothing to report.

XRO:
- Not present due to the night shift for beamline transport alignment activities.
- F. Wolff-Fabris informs the pre-shutters are commissioned by XRO during Sunday's shift.

LAS:
- Nothing to report

UNSYS: 
- Nothing to report

SPB:
- Work is ongoing.

FXE:
- Preparing for beam operation: no big problems.
- Thursday we reported issues with motors not moving. EEE helped and it was found the current of the motors were not correctly migrated. 
Now motors are working. Currents were set too low - this no risk of damage.

MID:
- Preparing for beam.

HED:
- Preparing and checking equipment.

SCS:
- Preparing for beam tomorrow.
- FFT end station problems reported last week are solved.

SQS:
- Waiting for the beam.
  143   17 Jul 2020, 10:19 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 15.07.2020
General Announcement:
- SASE3 tuning will start this afternoon and possibly will last until Thursday afternoon. SCS and SQS will share the closures of their hutches.

CTRL:
- Swap server of one of the two Karabo GUI servers for SA1 (mail to readiness): suggestion is to do Friday 9-10AM. Please let CTRL know in case you have any issue.
- Status on INFLUX roll-out: testing to take place at the end of this week. A coordination with SA1 has to take place due to the running macros. 

DA:
- Not present.

EEE:
- EEE is working in FXE and HED with the open issues.
- EEE gives an update of the discussed points on Monday:
   Communication: From now on EEE will include the contact persons, as well as to the instrument mailing list.
   SA1 motors in OFF state: after a PLC update the safest configuration for re-start is bringing them back in OFF state. EEE will continue with this procedure. 
If instruments want differently, please make a list of devices that has to be changed.
   HED issue: EEE is talking with Konstantin and found it was just a misconfiguration.
   HED pump controller: Konstantin is already working together with VAC;
   XPD turbo pump in SA3: no relation with PLC update. Yesterday's ZZ showed that the pump has to be replaced.
- Nerea and Patrick will provide a write up to ensure the problems are followed up and solved.

VAC:
- ZZ yesterday to install the Neon gas minican to allow the monochromator calibration in SCS/cw31.
- Exchange of PES turbo pump will take one day and break the interlock - North brunch will see no electrons and photons. 
A date has to be found in agreement with involved groups and facility's operation.

XPD:
- Turbo pump in PES: pump itself is the problem. Vacuum beamline is safe but PES does not operate. We need the PES for some user experiments still in 2020. 
- Operation of beam in XTD6 was not possible with multi-bunch due to devices in the beam. We do not know if in reality they were in beam or not. 
Karen Appel confirms the devices were out of beam. This has to be followed up.
- This weekend, together with XRO, XPD will do the further recommissioning of beamlines and diagnostics. 

DET:
- Vacuum issue reported on last Monday was confirmed to be related to the gauge.
- Started with DAQ tests with instruments. All ongoing.

ITDM: 
- Not present.

XRO:
- Not present.

LAS:
- SA2 laser maintenance taking place on Friday.

UNSYS: 
- Nothing to report.

SPB:
- Preparations are going smoothly. All good in the SPB's plan.

FXE:
- DAQ test with LPD deployment - all work fines but for one module need to be fixed in the detector. It is already under care;
- EEE/Bernard is finishing the loops.
- Monochromator commissioning next days (we need a ZZ to align the camera).

MID:
- It is scheduled MID DAQ test for Monday next week.

HED:
- Ulf will investigate with Konstantin on the EEE issues. In XTD6 there were two motors with wrong parameters configurations. HED will communicate directly with Nerea.

SCS:
- EEE is needed to solve issue with the FFT end station.

SQS:
- Not present.
  142   14 Jul 2020, 13:45 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 13.07.2020
General Announcement:
- XO will reset the calendar invites for the XO Integration Meetings of Mondays and Wednesdays as well as for the User&Operation Meetings on Fridays. Usual times will be 
kept.
- Machine is on startup and the RF window will be 400us instead of 600us in the first 2-3 weeks. Instruments can expect about 160 bunches in this time.
- Instruments must take tunnel components out of beam path, especially in case they are connected to the EPS which could potentially block beam operation.
- XPD and HED report on turbo pumps that failed last week. One has to investigate of it is linked to the Karabo re-start due to the DESY-NOC activities last Tuesday. 
FXE also reports motors in tunnel components were OFF.

CTRL:
- Further investigations are ongoing with Basler cameras that did not start after Karabo reboot.

DA:
- Not present.

EEE:
- Waits for feedback on the FXE XGM commissioning.

VAC:
- Nothing to report.

XPD:
- SASE XGMs are started and being operated. 
- Lasing in SASE2 was confirmed in the FEL-IMG; SA1 was lasing @2mJ last night.
- Re-alignment of SASE3 was not yet confirmed - no beam visible on the XGM or FEL IMG, only spontaneous radiation. 
- XPD did a successful DAQ test. 
- ZZ in XTD10 will be required to investigate failed PES pump. This will take place on Tuesday.

DET:
- Poor vacuum SPB/AGIPB last Friday: under investigation (may be a 40 gauge) and will be followed up next days. 

ITDM: 
- New DAQ version was released to fix detectors and slow data acquisition: tested in SPB and FXE (Agipd and LPD). 
- This week ITDM makes upgrades in all instruments but MID takes place next week; SCS still needs to agree on the date for this week.

XRO:
- Plan for the week is the recommissioning of the beam transport. XRO has shift plans for the weekend.

LAS:
- All lasers are running.

UNSYS: 
- Nothing to report.

SPB:
- Vacuum issue with the 40 gauge under investigation by DET.
- No big problems for the start of next week.

FXE:
- DAQ test taking place today.
- XGM is going with the plan but for a cable that may need to be exchanged.
- Upgrade of Beckhoff system: motors were not moving. There is a lack of communication between the Beckhoff upgrade, control support and instrument scientists.

MID:
- Not present.

HED:
- HED had to check and/or entry the motor parameters after the reboot of the control systems that place motors in OFF state. 
CTRL informs the information must be stored in the PLCs / Karabo. It is solved but it demanded a lot of work during last week.

SCS:
- Not present.

SQS:
- Problems occurring with turbo pumps. There is a particular controller in one rack that had to be replaced 7 times in 2 years. 
EEE and SQS will investigate and do inspections on this rack. 
  141   10 Jul 2020, 13:50 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 08.07.2020
General Announcement:
- Machine start-up continues to be ahead of schedule. Beam transmission to T4D and T5D has been done during the night (with open undulators). 
- An Invitation for the User&Operation Meeting on next Friday 09:00 was sent to the user&opmeeting mailing list. If you did not receive it, please contact XO.
- XO Integration Meeting takes place only on Mondays and Wednesdays at 09:30AM. 

CTRL:
- The synchronized shutdown, f/w update, configuration and restart of both CTRL network routers by DESY-NOC was performed yesterday. 
This included the Karabo shutdown and restart in tunnels, instruments, etc.
- Problems linked to the re-start: in LAS a miss-behaving ADC and a faulty host cpu issue; in CTRL some obsolete deployment information. 
- This Router update has to be schedule during the shutdown to avoid above mentioned issues. 
- CTRL observed two Basler cameras which did not reestablish a network connection after the router updates: Please try power cycling the camera to restore system.

DA:
- No complications due to routers update.

EEE:
- Cabling installation in SCS is in progress. 
- Technical staff will commission the FXE's two new brakes for XGM.

VAC:
- Not present.

XPD:
- Re-starting X-ray photon diagnostics devices this week.
- XGMs are ready for recommissioning.

DET:
- Nothing to report.

ITDM: 
- Maintenance of ELOG server and update of application has been done last night. System has restarted.

XRO:
- Not present.

LAS:
- All LASERS are back today.
- Due to the router update, SA1 laser was the most affected due to loose of thermalization and was OUT of operation a full day. SA2 was OUT only for the afternoon.

UNSYS:
- Nothing to report.

SPB:
- Getting ready for beam: few DAQ issues still open but being worked out.

FXE:
- All motors in SA1 were OFF due to Karabo re-start linked to router update.

MID:
- Update of Linux computers: ITDM will communicate with MID for the re-installation. 

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  140   07 Jul 2020, 13:23 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 06.07.2020
General Announcement:
- Since the end of the shutdown last Friday, the current Program is "Access/Tuning/Setup";
- DESY/MXL has started ramping up the Gun, powering and testing the Magnets and phasing the RF stations. 
- Currently it has electron beam transmission up to the TLD and is ahead of the planned schedule for these days.

CTRL:
- Data logger issue: Friday CTRL has rolled back to the old data login system. Meeting today at 11:00 with involved groups to define next steps. 
Information will follow via readiness mailing list. 

DA:
- No issues to report.

EEE:
- Issue with HED PLCs: a fuse has been tripped and it has been replaced. A CPU was broken and is believed to have caused the fuse tripping. 
This is the first time this happens in the facility.
- Cabling activities in SCS is scheduled to take place this week.

VAC:
- Nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- ITDM has made a request to DESY-IT for a routers restart. This will cause the network to be interrupted and restarted. Also Karabo will restart. 
A meeting today will define the proper time for this action, currently schedule for tomorrow morning. Information will be send to the readiness mailing list.

XRO:
- Not present.

LAS:
- All lasers are up and running again. 

UNSYS: 
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- Few problems with motors were already solved last Friday by EEE.

HED:
- All going fine.

SCS:
- Not present.

SQS:
- Nothing to report.
  139   03 Jul 2020, 14:21 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 03.07.2020
General Announcement:
- The Interlock tests are finished in all tunnels and XHEXP1 areas, following information from D3 and SRP.
- From next week we will continue with XO Integration Meeting only on Mondays and Wednesdays at 09:00. The Fridays' XO Int. Meetings will be terminated.
- Next Friday 10.07 the User and Operation Meeting will re-start at 09:00 in the virtual format. Invitation and corresponding Zoom link will be send in a separate e-mail.

CTRL:
- Preparation for operations has started. Three issues have been addressed: 
1. XPD issues are currently been worked and will extend over next week; 
2. XFEL gateway troubles seems linked to a router configuration; 
3. Migration of old data logger: problem with MID LA2 and SA3 (running wrong migration). 
- Recovering complex configurations: meeting 12:00 with ITDM - please wait until Monday for update.

DA:
- Not present.

EEE:
- Follow-up of the shutdown activities is ongoing; Modification of PLCs was done. 

VAC:
- Vacuum at beamlines are running; Shutters are ready.
- VAC plans to open XTD6 again from next Monday up to Friday, allowing HED access and work.
- Please inform VAC if XTD09 is needed to be accessed. Currently a ZZ is foreseen for this tunnel (EEE request).
- XTD10 can be accessed by ZZ during next week.

XPD:
- Small bugs on the DOOCS side of XGM were observed: bridges were checked and looked fine.
- XPD is planning the re-start of devices and support for machine tuning in coming weeks.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- SA2 laser: Train-ID showed issues (Update software failed).
- SA1 laser: went down due a crate failure and is warming.
- SA3 laser is fine.

UNSYS: 
- Nothing to report.

SPB:
- AGIPB update needs to be fixed. 

FXE:
- LPD new DAQ could not save data.
- Calibration pipeline for LPD is OK.
- Calibration pipeline for JungFrau does not work - also CTRL fixed the crash with DAQ (how the image gets the train ID's). The crash is not observed anymore.

MID:
- Discovered problems with motors in Laser room: EEE is informed and investigates.

HED:
- Will need access to XTD6 for final checks from Monday. Please inform VAC when access is finished.

SCS:
- Not present.

SQS:
- Not present.
  138   01 Jul 2020, 13:53 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 01.07.2020
General Announcement:
- Interlock tests are starting in SASE3 instruments right now. Zunaira, Eric and Wolfgang Clement are coordinating the TUV tests in the XHEXP1.
- MXL announced a draft for the machine start-up schedule which is published at the BKR e-log. The link to the schedule is: 
https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/doc/Schedule&pos=2020-06-29T15:40:57

CTRL:
- Group has a plan to switch the data loggers today; MID was already done in the past; Migration of old data was already done in advance. 
- There was a report of issues with the GUI server. Currently it is under investigation. A workaround is to disconnect and reconnect again or move to a second GUI. 
More information will follow.

DA:
- DA has two major plans for the upcoming days/weeks: 
1. Implement auto triggering in pipelines for the instruments (used for data analysis); 2. Stablish a plan to have spread knowledge on development tasks.

EEE:
- Installation of a SCS component is arranged for July 8th. 
- Cabling in HED is scheduled to be performed in cw31; EEE will contact FXE to fix a cabling installation date.
- More than 50% of Beckhoff modules are changed.
- Working on PLCs business as usual.

VAC:
- All fine: nothing to report.

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- DESY/NOC needs to perform software update of the control network routers for all SASEs. 
This will take place next Tuesday, the 7th of July, between 6AM and 8AM in the morning. An e-mail will be sent to the Readiness list.

XRO:
- Not present.

LAS:
- All lasers are running.

UNSYS:
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Nothing to add.

MID:
- All running fine.

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  137   30 Jun 2020, 08:50 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 29.06.2020
General Announcement:
- The Photon Run Coordinator will re-start next Friday July 3rd. Please also feel free to contact him/her (9-4800) for daily coordination and communication with the 
machine/BKR. 
- Interlock / TUV tests are ongoing in tunnels. Next Wednesday it starts in the experimental Hall / SASE instruments.

CTRL:
- CTRL has a plan to switch the Data Loggers on Wednesday. This will cause a downtime on this service for about 5 minutes. A follow up will be announced afterwards.

DA:
- Not present.

EEE:
- EEE has made an agreement with HED on the cabling issues. 

VAC:
- Tunnels are closed: maybe next week after Interlock tests the XTD6 and XTD9 tunnels will be open for a couple of days.
- If access in tunnels are needed (not only XTD6 and XTD9), please make a request by filling the table at the Desy/BKR elogbook 
(https://ttfinfo.desy.de/XFELelog/index.jsp), located under "Access Requests".

XPD:
- Nothing to report.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- All lasers are fine.

UNSYS: 
- Nothing to report.

SPB:
- It has tested the new Karabo for the Jungfrau detector and result is OK.
- This week SPB plans to test cameras.

FXE:
- Nothing to add.

MID:
- All fine.

HED:
- All fine.

SCS:
- Not present.

SQS:
- Nothing to report.
  136   25 Jun 2020, 10:27 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 24.06.2020
General Announcement:

- Exceptionally next Friday the XO Intergration Meeting will not take place. The meeting will resume next Monday. 
Please send us an e-mail to xo@xfel.eu in case of any issue.

- Details on the Interlock Tests: 
1. D3/SRP has the responsibility to carry on the tests; 
2. TS and XO (communication) are supporting groups; 
3. There is no rehearsal date; 
4. Tests on the Instruments/SASE wll take place on July 1st, 2nd and 3rd (reserve date); 
5. Access to the instruments and optics hutches is forbidden; 
6. Access to the laser rooms are not restricted, apart from the HED laser which needs clarification; 
7. Emergency off will be disabled on July 1st at 08:00AM. As soon as the emergency offs are tested, they can be enabled again; 
8. The participating EuXFEL Radiation Protection Officer can inform the instruments as soon as tests are ready with the respective SASEs for resuming access and work.

- Maurizio V. and Nerea J. mention that an EPS check must be scheduled prior beam delivery. XO will check with Zunaira/SRP on its status.
- Suren K. points the PKG in SA3 undulators is still switched off. TS shall switch it on prior tunnel closure.
- Suren K. asks on tunnel access for next weekend. XO states tunnels will be put in Interlock tomorrow (June 25th) and shall remain interlocked until the next shutdown.
Access to XTD6, XTD9 and (maybe) XTD10 can be done for few days after the interlock tests. Access to all other tunnels must be via ZZ. 
Tunnels will not be open for the weekend. (after the meeting DESY/MXL confirms this procedure).  

CTRL:
- Nothing to report.

DA:
- Working on small requirements with the instruments.

EEE:
- SA1 VAC signal's issue from last Monday: it has been fixed. There was problems related to filter in the train-ID communication: the addresses were not properly set. 
- Cabling is ongoing on SCS; HED cabling waits for time/schedule for installation (to be discussed between EEE and HED); 
Cabling works ongoing with DESY-MPS which connect MID to PPS.

VAC:
- Closing the activities - SA1 VAC signals are OK.

XPD:
- Nothing to report.

DET:
- XO asks for an status on the MID/AGIPD train-ID issue from last Monday. S. Aplin and K. Wrona say it has to be further investigated as is not clear where the problem 
lays. 
The DAQ gets data from AGIPB, DOOCS, Karabo, labelled with train-ID and the labeling have to be checked for consistence. This is not a new issue and has to be addressed 
in the frame of data reduction. 
- A clear independent signal has to be also checked. P. Gessler says it is needed to get various patterns for a complete understanding. A clear strategy to develop and 
address has to be made. 
- S. Aplin will overlook the efforts among different groups.

ITDM: 
- DAQ installations: this morning a fix for big detectors is done; after leaving DAQ for longtime there are some issues that will be fixed next week. 
- Majority of acceptance tests are done - slow data is fixed.

XRO:
- Everything is done. The last check on tunnels was performed. 

LAS:
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- Nothing to report.

HED:
- Finished with the tunnel components check.

SCS:
- Nothing to report.

SQS:
- Nothing o report.
  135   23 Jun 2020, 14:12 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 22.06.2020
General Announcement:
- The User & Operation Meeting that took place at beginning of this year on Fridays will re-start on July 10th. 
Proper announcement and Zoom link will be sent. In this context, the XO Integration Meeting will continue only on Mondays and Wednesdays. 

CTRL:
- Some of XFEL Elogs were not working during the weekend. ITDM is taking a look into the problem.
- SA1 Vacuum devices show some errors (not running). It seems the problems comes from the PLC and is under investigation. 
The controllers in the tunnel were OK and pumps were placed in safe state. 
This has affected all SA1 loop, at XTD2 and XTD9. Also the devices were not properly communicating.

DA:
- No issues to report.

EEE:
- Issues with the SA1 VAC loop.
- Updates of timing systems affecting LAS were successfully done on Friday. EEE encourages people to report in case of problems. 

VAC:
- Reporting on the SA1 VAC loop issue.

XPD:
- Last Karabo update done on Thursday.

DET:
- No issues to report.
- MID observe a train mismatch between AGIPB and data sources such as SA2 XGM. The AGIPD has a shift of 1 train-ID. 
This issue involves several groups and coordination and solution still miss.
- LPD/FXE also takes the train info from clock control but shows no mismatch.

ITDM: 
- Investigating the Elog access issue.

XRO:
- Not present.

LAS:
- Maintenance in SA3 and SA2 are done. All lasers are running.

UNSYS: 
- Nothing to report.

SPB:
- Nothing to report.

FXE:
- Nothing to report - checking the vacuum.

MID:
- Report on the AGIPD train-ID.

HED:
- Working with big lasers.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  134   19 Jun 2020, 14:28 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 19.06.2020
General Announcement:
- Next Monday it is planned the NOT-AUS experiment in the frame of Interlock tests and the emergency off will be disabled. 
Only groups with EQP persons can continue working in the hutches. An e-mail was sent by TS.
- There will be test on smoke extraction in Tunnels on next Monday and Tuesday. Information was sent to Readiness mailing list.
- An e-mail to Readiness list was sent this morning regarding TUV/Interlock tests in SASE areas on July 1-3: 
No access or work on the experimental and optics hutches is allowed in these days. More specific details will follow in the coming week. 
If you have questions, please contact XO.

CTRL:
- Finished 2.9 deployment - control rooms also have the updates;
- CTRL is working on the XGM together with Theo (XGM does not update some properties). This came after an update done at the Desy side.

DA:
- Nothing to report.

EEE:
- Shutdown activities are ongoing. 
- Migration to mc2motors was ongoing. EEE encourage groups to check motors after start running again.
- For next week: cabling work in SCS and PLC modifications with the help of two external colleagues.
- Timing system update: SA1 and SA3 timing system laser and instruments are done; EEE will do the SA2 this afternoon; Still to be arranged a date for the balcony rooms.
If problems occur, please contact Bruno Fernandez.

VAC:
- Finished all shutdown activities. 
- The recovery of HED pulse picker branch needs help of HED staff. HED points Konstantin is around today.

XPD:
- Finished all XPD tunnel works.
- Testing ongoing: SA2 motors after mc2 migration.
- Checking the trigger on uTCA crates after upgrade.
- DAQ test was successful with ITDM.

DET:
- New DAQ system has some issues with AGIPD in SPB (did not store data) - to be followed next week.
- Deployment of new calibrations is ongoing.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- Warming up SA3 laser.
- SA1: LAS maintenance is not finished. So we ramp up later. Bruno reminds SA1 must be back and switched on again.

UNSYS:
- Enclosure of panels in SASE3 has been installed and all tunnel activities are done.

SPB:
- Nothing to report.

FXE:
- Works on Detectors are ongoing.

MID:
- Nothing special.

HED:
- Activity with lasers alignment.

SCS:
- Not present.

SQS:
- Nothing to report. Working in different things/parts.
- SQS has planned activities for July 1st-3rd. It suggests that more specific communication on TUV tests and access restrictions must be done differently
than the e-mail distributed on June 8th and today. XO will look for improvements.
  133   18 Jun 2020, 13:40 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 17.06.2020
General Announcement:
- Today and tomorrow it takes place the tunnel ventilation maintenance between 6:30 and 8:00AM;
- An e-mail was sent to readiness list with tunnel access restrictions for next Friday morning at the Osdorf Born site in and XTD4 work between 10-11AM.
- Today there the maintenance of the SA3 hutch crane is scheduled.

CTRL:
- CTRL will follow up the uTCA updates.
- Mc2 migration in SA2 is almost done - only XPD needs to test some componenents.
- CTRL is working in some layout problems in scenes of the latest GUI update (new Karabo). Groups are encouraged to write tickets to CTRL if they require changes. 

DA:
- Nothing to report.

EEE:
- Shutdown activities are ongoing with, e.g, VAC loop updates and PLC updates;
- The firmware updates affecting the PP-laser timing will take place on next Friday (19.06) in all SASEs. 
- EEE is following the delay on the MID cabling related to personal interlock. 

VAC:
- Shutdown activities in XTD6 and XTD10 are ongoing. 

XPD:
- Currently finishing own summer shutdown tasks in tunnels.
- First DAQ test together with ITDM: had to wait the timing server re-start. To be continued today also after full mc2 migration.

DET:
- Calibrations of detectors at instruments are ongoing.

ITDM: 
- Nothing to report.

XRO:
- Not present.

LAS:
- So far all in good state.

UNSYS:
- Nothing to report.

SPB:
- DAQ update was not an entire success - but it has been taken care. DAQ is needed for the calibration test of detectors.

FXE:
- Nothing to report.

MID:
- Nothing to report.

HED:
- Nothing special.

SCS:
- Nothing to report.

SQS:
- PC updates are ongoing.
- SQS raises the question on hutch access during the disabled emergency off planned for next Monday. It is possible that SQS EQP will be in sick leave on that day. 
XO will contact Torsten Schoen to get more details on the time schedule and possible access for that day.
  132   16 Jun 2020, 13:32 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 15.06.2020
General Announcement:
- There will be additional ventilation maintenance on all tunnels, scheduled for next WED/17th and THU/18th between 06:30AM and 08:00AM. 
We request staff to do not enter in tunnels during this period.
- TS has sent information on the hutch crane maintenance starting today up to Wednesday 16th. Please make sure one instrument colleague is present.
- XTD4 will be closed on next FRI/19th as part of Interlock Tests (§88 - TUV Prufung).

CTRL:
- Few minor things were solved on Friday.
- Notification service worked as expected in SPB; VAC also got notification.
- CTRL will follow the uTCA updated after EEE is done today.
- CTRL is working on the PMO projects.

DA:
- Nothing to report.

EEE:
- Activities running well.
- Today afternoon migration plan for tunnel crates and tomorrow SA2.

VAC:
- Nothing to report.

XPD:
- Nothing to report. 
- XPD will check the uTCA crates after its update/restart.

DET:
- Data has been taken at SCS; also performing online tests.

ITDM: 
- Nothing major - Continuing updates with SQS, MID and HED instruments.

XRO:
- Nothing to report.

LAS:
- Scheduled SA2 maintenance on Tuesday. EEE firmware update also takes place. 
- Timing related updates on SA1 and SA3 are under discussion between the groups.

UNSYS:
- Everything is OK - All work in tunnels are done.

SPB:
- Nothing to report.

FXE:
- Nothing to report.

MID:
- Nothing to report.

HED:
- Water leak from last week: TS has contacted the external company and a faulty air humidifier was the cause. It will be changed/fixed.

SCS:
- Not present.

SQS:
- Nothing to report.
  131   12 Jun 2020, 13:48 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 12.06.2020
General Announcement:
- There will be tunnel ventilation additional maintenance on two mornings next week from 6:30 to 8:00AM. An e-mail to Readiness list will be send when dates are known.
- There was again a situation that the interlock door between XTD4 and XTD10 was blocked open. SRP reminds all that please do not leave this door opened. 
Further investigation with Desy side will be done if this was a mistake done by an external company. 
- EuP training: Tobias Haas has extended trainings until end of August. There will be refreshers and first time trainings starting on week 26 as in-person EuP training.

CTRL:
- CTRL is done with 2.9.0 deployment - now DAQ and calibrations checks are ongoing.
- No major issues during deployment: still few devices need to be updated again; CTRL is working on it.

DA:
- DA has released new version of extract data. Documentation will be send around and people are invited to give feedback.

EEE:
- Activities are running fine.
- PLCs side: this week there is migration for mc2 in SA2 - EEE ask people to crosscheck systems.
- Two cables in MID will be finished late next week (regarding personal interlock).
- EEE is conducting migration microTCA systems/crates: SA1 and balcony rooms.
- Timing for the Lasers updates has to be arranged with LAS. CTRL and EEE recommend making it during the shutdown. It needs green flag from LAS and Instruments. 
- Upgrade of firmware of timing system: this will not be done to avoid stop timing system for lasers. Tunnels are done.
- Puppetization work is ongoing. SA2 and SA1 tunnel remains to be completed at beginning of next week. 

VAC:
- No issues to report.

XPD:
- Follow up of SA2 mc2 migration: we recheck motors in tunnel; update scenes with CTRL.
- XPD is almost finished with large interventions.

DET:
- Test of Agipd with new Karabo is successful. The DSSC is also working.

ITDM: 
- DAQ updates: to be started today or next week.
- Hardware updates in SASEs that ITDM will be performing.

XRO:
- Nothing to report: XRO is finished with big installations. Only small tasks to be done next week.

LAS:
- The broker update caused the timing in SA3 to be lost. It took several hours to bring up the system. 
- Updates in SA2 (affecting timing) will be performed next Tuesday.
- SA1 and SA3 have to be scheduled and it depends also on instrument's plans. 

SPB:
- It has planned tests on the calibration of pipeline / DAQs for next week.

FXE:
- It has upgraded to new Karabo and some controls will be updated today.

MID:
- Work ongoing as planned.
- CTRL test the influx data logger today and Monday and this causes loosing of information for few minutes. 

HED:
- Water incident in SA2: Water spills caused a flooded experimental hutch. TS have repaired and will check with the external company if systems have been exchanged. 
Such events must be avoided in future. 
- In many hutches there are water sensors in the floor which are not connected: TS and SRP shall take a look.
- Summer shutdown: there was scheduled task (presumably from Desy groups) on June 8-10th that nobody showed up. By the other side, suddenly someone was working inside the 
laser room. 
Ulf would like to see Desy people working with better planning and coordination. As positive, EEE tasks on microTCA are an example of good practice in communication and 
coordination. 

SCS:
- Not present.

SQS:
- No issue to report.
  130   10 Jun 2020, 11:12 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 10.06.2020
General Announcement:
- Maintenance in the Tunnel Air Conditioning is completed. No more work restrictions are in place besides the regular COVID-19 rules. 

CTRL:
- Deployment of Karabo 2.9.0 started yesterday accordingly to the information sent to readiness mailing list. So far the work goes with the plans.
- CTRL mentions Broker servers need update and will bring down Karabo again. Possible intervention date (early morning) will be discussed with LAS and Instruments.
A warning e-mail will be send 1-2 days in advance.

DA:
- Related to new Karabo: in case the upgrade is not covered, please contact DA.

EEE:
- Migration of mc2 done in SA2. 
- Preparing call for tender of HED cabling; other instruments follow up next.
- There will be maintenance in SA1 and SA3 balcony rooms. A discussion with LAS is in place (risk of possible shutdown can be accepted).

VAC:
- Not present.

XPD:
- Nothing special to report - resuming tunnel activities after AC tunnel maintenance.

DET:
- Nothing to report.

ITDM: 
- The updates on the online cluster side are almost finished. Only a check has to be done.
- Access to online cluster is labelled.
- DAQ has to be re-started.
- Updates are planned for tomorrow: reboot exflgateway at 10:00AM - an e-mail will be send to readiness mailing list.

XRO:
- Not present.

LAS:
- SA1 and SA3 lasers are in operation.

SPB:
- SPB asks for sending warning notice a day or two prior the Karabo restart intervention.

FXE:
- Nothing to add.

MID:
- MID will inform ITDM today on the dates for the updating the linux PCs in the hutches.

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- SQS will also inform ITDM on possible dates for updating PCS.
- Some macros are not running (they were not able to start).
  129   09 Jun 2020, 15:47 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 08.06.2020
General Announcement:
- SA2 lasing: up to 2mJ@9keV was achieved with strong linear and quadratic tapering with 250pC bunch charge. 
Lasing with 500pC achieved 0.45mJ@9keV and with 100pC achieved 0.75mJ@9keV.
- From today we start the summer shutdown phase.
- Please note the last minute information on air conditioning maintenance in tunnels between Tuesday 15:00 up to Wednesday 08:00. 
We ask groups to shift planned works to outside this time slot. In case you need to be in tunnel, it is mandatory to use FFP2 masks.

CTRL:
- Karabo 2.9.0 deployment will start tomorrow and follow over the week. Typical activity time is from 8AM-16:00 each day.

DA:
- No problems. 

EEE:
- EEE plans for tomorrow the mc2 migration.

VAC:
- HED mono has adjusted gap between crystals. All other tunnel activities ongoing.

XPD:
- Major task for the coming week is related to the PES maintenances. All other tasks are also organized.

DET:
- Will carry tests after the Karabo installation.

ITDM: 
- Group will be busy with updates to be done this week.

XRO:
- Completed big shutdown activities: M3 cooling mirror - leak tested OK and pumping during the weekend.

LAS:
- All lasers are in operation.

SPB:
- Progressing with regular planned tasks.

FXE:
- Nothing to report.

MID:
- Nothing to add.

HED:
- Working with EEE activities.
- HED requests suitable dates for testing the THz laser. This date has to avoid infra-structure, interlock and other works. 

SCS:
- Nothing to report.

SQS:
- Nothing to report.
ELOG V3.1.4-7c3fd00