Notes XO Integration Meeting 23.06.2021, posted by Frederik Wolff-Fabris on 23 Jun 2021, 10:41
|
General Announcement:
- In case groups had issues with power glitch on last Sunday, please let XO know for that this will be discussed with Desy.
- Interlock tests are happening and will reach the hutches on next week. In the past there is little to no implications to the instruments.
During the tests it shall be transparent but no access shall happen during the tests time interval. As soon as XO receives a detailed plan from SRP/D3 this will be
communicated to the groups.
CTRL:
- Influx logging issue has been solved by changing settings together with ITDM.
- Announced updates for the PLC (ELMO motors / settings of rotary encoders): to be deployed tomorrow early in the morning (PLC continues to run);
For the tunnel topics CTRL proposes to make tomorrow at 8AM and instruments shall coordinate with contacts in case there is restrictions for this time.
- Found small issues with a fuse when connecting to the middle layer of Beckhoff devices. It has not been a problem.
- Natalia G. informed a motor was not working and EEE could solve it early enough. Checking systems in advance is a good practice.
- CC updates to be done tomorrow and rolled to the back properties.
- Read the Docs legacy mode will be switched off on 5th July.
DA:
- Reminder that today the offline detector calibration is not available.
EEE:
- Beckhoff terminals / Fuse issue work is ongoing. EEE waits for contact feedback from other instruments.
- SA3 balcony room and PP-lasers uTCA updates ongoing.
VAC:
- Not present.
XPD:
- One XTD2 door was defective and Jan G. reported on Monday evening to TS which informed Desy colleagues. Ralf E. informed this morning that the door was fixed.
Issue was a disc of the magnetic contact which had fallen and therefore door was not working properly.
- Motor issue in XTD2 (stuck in inserted position): technosoft control motor for MCP detector. EEE and XPD worked on it and is mostly solved.
One needs a feedback from CTRL to find why this particular motor looks different from similar devices.
- Tunnels works are done.
DET:
- DSSC improvements taken place and is progressing.
ITDM:
- Nothing to report.
XRO:
- Not present.
LAS:
- Proceeding with SA1 AMPHOS maintenance and expecting to be done by today.
UNSYS:
- Nothing to report.
SPB:
- Doing well with fuses changes. It tries to finish all of them in the next few days.
- Couple of issues with uTCA updates but Bruno F. is helping.
FXE:
- Nothing to report.
MID:
- Contacted Joern R. / EEE to give feedback on cabling.
HED:
- Nothing to report.
SCS:
- All fine at SCS.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 21.06.2021, posted by Frederik Wolff-Fabris on 21 Jun 2021, 10:56
|
General Announcement:
- Interlock tests starting this week. An e-mail was sent to xo-info@xfel.eu and if clarifications are needed please get in touch with XO.
- Zunaira A. asks if XO knows if emergency OFF will be tested and power would be bridged.
XO has no info on specific tests for emergency buttons and understands power will be bridged as it was done in the past. XO will confirm with Wolfgang C. / D3.
- From Wednesday of this week tunnels will be closed. If a task is pending, please contact XO.
- Jan G. asks if uTCA maintenance in tunnels are finished. Bruno F. says that tunnels are completed.
- Sunday morning 7:30AM there was a warning on the power stations but no systems faced power outage as much as it was reported. Please check your devices and systems for
potential power failures yesterday.
CTRL:
- Control monitors update happening this week.
- Today CTRL needs exclusive SA3 access to motion loops for middle layer devices updates.
- Cameras with Clone work in the Lima version: intervention for these cameras between today and tomorrow.
- Influx logging system: CTRL asks XPD that there are messages being received frequently from the PES and needs to switch it OFF for a while.
CTRL will contact Joakim Laksman if tests are being made otherwise it is not a problem to stop PES for minutes.
DA:
- Nothing to report.
EEE:
- uTCA updates to continue on Wednesday.
- SPB crates modifications ongoing and discussion with other instruments is taking place.
- Missing feedback from MID and HED for cable installation.
VAC:
- Finishing tunnels tasks.
- Question to EEE on when the issue with SA1 vacuum loop could be addressed. Bernard B. says he will check the issue mentioned Friday afternoon by today.
XPD:
- SA1 alignment of Shimadzu camera ongoing; Other works from A. Koch are finished.
- Jan G. says that tasks are finished apart of a MCP motor in SA2 (ticket has being issued and Bernard B. will take care of it after this meeting). Issue is that motor
does not move in reality.
DET:
- Nothing to report.
ITDM:
- Reminder was sent on creating new proposals for the next half-year. Related in a change in how to clone the commissioning proposals. Please give a feedback to ITDM
today.
XRO:
- Working in tunnels.
LAS:
- SA1 AMPHOS activity is ongoing and takes longer than expected. To be finished tomorrow or Wednesday. It has to be discussed with XO as other activities are planned
which affect SA1 PP-laser.
- On Friday SA3 there was cooling water maintenance that switched off PP-laser. To be addressed by XO.
UNSYS:
- Nothing to report.
- Tunnels can be accessed until Tuesday 21.06. Zunaira A. adds that the Interlock tests started this week and go on during next week.
SPB:
- Supporting EEE with changing the crates. Good progress has been done.
- Problems with uTCA crates after the upgrades (fast-ADC can't be reset; previous settings are missed after the upgrade).
FXE:
- Nothing to report.
MID:
- All fine at MID.
- Contact to EEE will be made on the cabling feedback.
HED:
- All good at HED.
SCS:
- Nothing to report.
SQS:
- Last Friday tried to run digitizer again after upgrade: minor issue and if not solved today SQS will get in touch with EEE/CTRL. |
Notes XO Integration Meeting 16.06.2021, posted by Arvid Gebhardt on 16 Jun 2021, 08:37
|
General Announcement:
- Interlock test next 2 weeks starting on monday - from next wednesday the photon tunnels will be closed; Tests are the XHEXP1 take place on 30.06, 01.07 and 02.07 with the last date as reserve; An email is going to be send by FWF to the xo-info@xfel.eu list.
CTRL:
- List of items to be sent to readiness: 1. problems with lima based cameras (Lima library does not work – needs to be updated: instruments contacts shall get in touch); 2. In SA1 there is legacy Undulator Karabo scenes – vector properties have to be adapted; 3. Counter Clocking motor devices upgraded next week.
- A. Koch asks if SA1 was working this morning. Steffen H. says there were updates but must be running now.
DA:
- Next wednesday will be the Offline calibration update deployment
EEE:
- Shutdown activities going well; "Wille" is short in people but scheduling adjustments with FXE and MID is organized.
- If there are short circuits or mal functions in the Beckhoff crates and current is not high enough to trigger the fuse it may be a fire risk. Solution is developed without impact in firmware, just a replacement of fuses. EEE starts today at SPB the changes; Based on this experience a plan will be made for the other systems and EEE will get in touch with groups.
VAC:
- try to get the works finished in the tunnels
- waiting for the Beckhoff crate works
XPD:
- Most of the tunnels works have been done; others are ongoing; No surprises are expected.
- MKK logbook from Desy reporting about electrical issue. XO will check. (XO update at 13:30: automatic mailing system is being commissioned and under testig; no power failure happened at the time).
DET:
- Work with DSSC in SQS going as planned.
ITDM:
- Work currently being done at FXE.
- Reminder of Friday afternoon gateway update.
XRO:
- Not present.
LAS:
- Nothing special to report.
UNSYS:
- Greasing the linear stages for undulators but not in SA2 (missing grease).
- SA1/SA2 after middle plane alignment UNSYS redefined the minimum gap and will be communicated through XO.
- Cell#36/SA2 there is a cable disconnected that need to be fixed.
- SA1/SA2 working with PS tables for extending operating tables for higher photon energies.
SPB:
- Work with EEE changing all the crates (Beckhoff)
FXE:
- Nothing to add
MID:
- Several ongoing infrastructure works are going fine.
- Worried with Beckhoff fuses (reported two years ago by Konstantin). It will be good to hear experience with SPB.
HED:
- Nothing to add
SCS:
- Nothing to add
SQS:
- Nothing to add |
Notes XO Integration Meeting 14.06.2021, posted by Frederik Wolff-Fabris on 14 Jun 2021, 10:15
|
General Announcement:
-
CTRL:
- Announcement: tomorrow starts Karabo deployment on FXE in SA1 as well as ITDM; Wednesday the MC2 upgrade for the rest of facility (already done in SA2) will take place.
- Device clone fix to be deployed today (e-mail sent to readiness).
- Two versions of ReadtheDocs exit (Old one will be OFF on the July 5th). If someone has an old project at the old version of ReadTheDocs please change to the new
version.
DA:
- Nothing to report.
EEE:
- Reminder that uTCA systems are under maintenance this week (interruptions planned in balcony rooms).
- EEE received report last Monday from SPB with issues on motor cables from Beckhoff controllers. Electrical current could go very high but not enough to trigger the
fuses and consequently overheats and has a fire risk.
EEE is looking implementations to take place this week in crates. Ordered parts shall arrive beginning of this week.
It is a large intervention to introduce 1-2 fuses in the modules/terminals of all crates in tunnels and experimental Hall.
- Steve A. adds that the changes are piece cases, each crate will be touched; if 50% of the crates are changed during this maintenance, one reduces 50% the chances for
fire.
P. Gessler comments that one has to limit the current of the 24V terminals, by introduction the additional fuse. All crates will be touched but it is a linear change.
VAC:
- Investigating the situation with motor terminals. A. Galler says tunnels will start being closed next Monday and will find the exact date for XTD6 and XTD9 closure.
- Shutdown work ongoing but no issues to report.
XPD:
- A. Koch, N. Kujala and W. Freund say there is nothing to report.
DET:
- Nothing to report.
ITDM:
- Reminder on the e-mail of the gateway maintenance proposed to take place on Friday. In case of issues please contact ITDM.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
- Pouneh S. asks when AMPHOS engineer will come this week. Man Jiang (LAS) says that work starts today from 10AM with the amplifier in SA1.
Man J. will inform back Pouneh on a detailed schedule for this week.
UNSYS:
- After a delay in Afterburner project activities with MEA group, the infrastructure problems at the beginning of last week were solved by TS.
The work is currently ongoing for the afterburner project and mechanical elements have been installed.
- Today there will be adjustment of the base plates pedestals; from tomorrow work with concrete filling takes place. Planned tasks for the afterburners will be completed.
- Cell18/SA1 problem: it was not needed to replace the quad mover.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- All fine at HED.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 09.06.2021, posted by Frederik Wolff-Fabris on 09 Jun 2021, 10:14
|
General Announcement:
- Zunaira A. says that on HED on Saturday there was the 2 hour break on the search mal-function. It was solved by manual restart done by Eric B. (origin was the DACHS
issue) and we do not have an OCD for this matter - basically depends on the goodwill of people.
Jan G. says that there is no OCD for what Eric B. and Dennis K. did. It would be nice to find a solution for such situations - if this happens during the weekend, this
may have the end of an experiment.
Zunaira A. asked Eric B. to file a non-conformity report to be shared with XO.
- There is no UOM meeting next Friday.
CTRL:
- Karabo updates on most of topics are performed yesterday. Few issues in SA3 and HED are under work today. Few open points:
1. new GUI features: a configuration may be an issue but an e-mail is sent;
2. EuXFEL gateway hosts to be updated today after lunch with interruptions on running scenes. GUI 2.11 will be installed.
DA:
- Karabo installation on online calibration will be done under arrangements with instruments, mostly likely this week.
EEE:
- Monday point concerning cooling water effects: uTCA will not be affected (no interruptions for e.g. PP-Lasers).
- Next week schedule for uTCA maintenance: it does not affect LAS and balcony rooms. E-mails will be sent when maintenance is done and systems are available.
- SPB issue with PLC wiring on fuses - EEE is analyzing for proposing modifications.
VAC:
- Working on tunnels.
XPD:
- All XPD devices are in safe state.
- Beryllium covers in Hirex are placed.
DET:
- Nothing to report.
ITDM:
- ITDM finished maintenance of online resources (clusters/storages/migrations). Most of the DAQ is back in operation.
- In the course of June ITDM close the commissioning like proposals. All PIs/contacts of instruments will receive an e-mail with details for procedures to be done.
- Plans for the usual XFEL gateway update: tricky to schedule. A proposed date is by the end of the next week, on Friday 18.06 later afternoon.
Janusz S. and/or XO will send an e-mail to readiness for spotting eventual issues with this date.
XRO:
- Maintenance work is going well accordingly to the schedule.
- Karabo updates: Maurizio V. understands the gateway update just makes Karabo not usable and asks if it changes anything else. His concerns are with having time for
check.
Janusz S. says the operating system reboot only and Karabo installations are done before. Steffen H. says that today between 11:12:00 there is an update on the gatway
already.
No effects are expected for devices. New servers are to be installed and commissioned. Karabo updates are being done already and there is time for checks.
LAS:
- Nothing to report.
UNSYS:
- Monday's problem with MEA and Apple-X pre-installation works: It was solved and works continue from today. It is still planned to finish until the end of the shutdown.
- For Quad in SA1/Cell18 the problem was a loosen cable and simple to recover.
SPB:
- On the damaged 20um Si attenuator: it locates at SPB branch.
- Romain L. points to a communication issue with the DAQ maintenance done yesterday: Janusz S. says that this was submitted to the maintenance and happens every
maintenance. XO will address the case.
FXE:
- Work with crate in the tunnel: cooling water for the Mono stopped. Nerea J. says that there is works by the VAC on this crate. Cooling water will be restored after
finished work.
MID:
- Nothing to report.
HED:
- Issues with HED Karabo update are being taking care (issues with fuses).
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 07.06.2021, posted by Frederik Wolff-Fabris on 07 Jun 2021, 10:29
|
General Announcement:
- Accelerator has been shutdown this morning and the Summer/21 maintenance period has started.
- A. Galler asks the instruments to check the beamlines are in safe state for that tuning and setup starts after the maintenance without potentially damaging equipment.
CTRL:
- Reminder that tomorrow CTRL starts the deployment of larger Karabo topics (details with contact persons).
DA:
- Nothing to report from OCD service.
- Detector calibration: Philipp S. reminds that the pipelines/services will be temporarily removed for updates but it will be brought back.
EEE:
- Nothing apart of maintenance tasks.
VAC:
- Nothing to report.
XPD:
- Jan G. reports as PRC many technical issues happened during the weekend:
1. Problem with DACHS system (hutch search at HED was not able to be performed). This was a communication problem and DACHS server had to be rebooted There is no OCD over
weekend).
2. EPD went down due to network failure (down for all tunnels and was fixed on Sunday). Martin Dommach confirms that the entry at TXD9 was possible this morning.
- XPD reports on a comment from SPB on June 1st says a damage of 25um Si was found. Jan G. asks more details which device it is.
M. Sikorski will check that. Naresh K. says it shall be in the SPB branch.
- XGMs are powered down for the maintenance.
DET:
- DRCs/DOC is OFF during the maintenance period. If support is requested, please contact support via ticketing system.
ITDM:
- For the shutdown period, the emergency IT phone is available only for critical service (old number 98070).
- Storage systems will be serviced and will not be in operation all the time for the next weeks.
XRO:
- Not present.
LAS:
- SA2 and SA3 PP-lasers will be running next weeks for instruments.
- AMPHOS work taking place next week: needs cooling water and timing system until June 22nd. For short interruptions of 20 minutes the doors will be opened.
Patrick G. says that local timing is a good solution. Tomasz J. says they have equipment to provide for 1 PP-Laser.
UNSYS:
- Quad mover issue in cell18/SA1 will be looked on today or tomorrow.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
- A. Galler asks if cross talk was observed during the weekend. Jan G. says he talked to SQS this morning and they made checks and on Saturday they did not find any shift
or cross-talking.
MID:
- All fine at MID.
- Accesses in the optics hutches may be restricted. Also the experimental hutch has restricted access. If access is needed, please contact MID staff beforehand.
HED:
- DACHS system: one has to investigate further in the cache if this will happen again.
Jan G. says that there is an entry by D. Kruse and the hutch search is another issue. Maybe the cache works only for one and not both issues.
- For the ADC T. Preston will investigate why the bunch pattern was not read correctly when changing the settings.
SCS:
- Nothing to report.
SQS:
- Not present. |
Notes XO Integration Meeting 02.06.2021, posted by Frederik Wolff-Fabris on 02 Jun 2021, 10:24
|
General Announcement:
- XO reminds staff to check the safety trainings and the DACHS access rights/permissions for the tunnels when planning tunnel accesses during the maintenance period.
- A. Galler asks all instruments who work with X-ray beam that in case it is found any damaged component, please take a snapshot and post in the operation e-log.
This allows to get known prior/during the maintenance period.
- Last UOM prior maintenance period takes place next Friday.
CTRL:
- For the maintenance period: CTRL will send around the Karabo schedule for updates such as MC2 motors. CTRL will get also in touch with affected people.
- Macro update announcement on Monday will not take place anymore.
DA:
- Nothing to report.
EEE:
- EEE is gathering and scheduling all shutdown tasks and will start next week with tunnel works.
- SASE1 VAC, EPS and MOV, and SASE2 MOV2 loops will be updated next week.
- EEE will also assist, if needed, on the testing of the Mc2 interface migration that Steffen H. / CTRL has informed.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- This is the last week of DOC operation. The DOC will re-start on cw28.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Man Jiang reports all 3 PP-Lasers are running.
- SCS can't open the shutter (interlock issue) but this is not related to LAS.
UNSYS:
- Nothing to report.
SPB:
- DOC has contacted SPB saying Maxwell calibrations failed. This will be checked.
FXE:
- Nothing to add.
MID:
- All fine at MID.
HED:
- Issue with camera at 10:30PM. It Needed to be power cycled. Steffen H. says that the camera worked this morning. Maybe case is already solved. One will keep an eye if
it happens again.
SCS:
- Issue with Interlock mentioned by LAS. Pouneh S. is working on it.
SQS:
- User operation this week including optical laser.
- Thomas B. says SQS struggles with temporal overlap but spatial overlap is fine. They have first to find clear signal to investigate more on the temporal overlap. |
Notes XO Integration Meeting 31.05.2021, posted by Frederik Wolff-Fabris on 31 May 2021, 10:18
|
General Announcement:
- Next Friday is the last User & Operation meeting for this run. We will have presentations of 3 user groups.
- Today 31.05 is a normal delivery beam day.
CTRL:
- Information for maintenance period: CTRL will update macros for compatibility at SPB and SCS instruments. CTRL will contact the instruments for scheduling.
- Undulator energies were not corrected updated in Karabo. Re-starting the servers may help. A mismatch of values with Doocs were found.
DA:
- Nothing to report - quiet OCD week from last Wednesday.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Couple of connection issues due to power glitch this morning at 4:56AM. All affected systems were recovered.
- The DOC is not operational today. Tomorrow at 7AM the DOC is back to operation. Calls goes today directly to DRC.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- PP-Laser in SA1 is down due to power glitch. Cooling water (8 or 11 C) for AMPHOS was affected and caused this downtime. Water has come back few minutes ago.
- SA2 and SA3 lasers were not affected. There are different pumps for each SASEs. Lorenz K. is investigating (e-mails from GLT and no warning from MKK).
- SA1 PP-laser will be stable by the afternoon as requires time for stabilization.
UNSYS:
- Nothing to report.
SPB:
- Laser interlock on Wednesday to Thursday night was triggered: panic button switch was falsely indicated that was pressed. It seems to be a failure in the system.
Desy / BKR reset it and operation could be restored.
Zunaira A. says that if panic switch is pressed, then the beam shutter will not be able to be moved (SPB will not be able to receive beam).
There is no relation with the power glitch.
FXE:
- Nothing to report.
MID:
- Looks fine at MID. Joerg H. asks for an e-mail informing on situations like the power glitch for that groups be informed and check if devices are affected.
A. Galler says currently the reason for the power glitch is under investigation and will be communicated.
HED:
- This week will be a busy week due to energy changes already at 7:00AM.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 26.05.2021, posted by Frederik Wolff-Fabris on 26 May 2021, 10:23
|
General Announcement:
- XO reminds the procedure for ZZ access is organized through the PRC.
If a ZZ is needed, please get in touch with the PRC for advice and fill it up the Redmine access request (link located at the TTfinfo e-log).
The PRC will organize and communicate the access with machine RCs, specially in cases of urgent matters.
It is not beneficial to do not inform the PRC as components in tunnels have to be checked and/or grounded prior the ZZ access takes place.
CTRL:
- Announcement for the summer maintenance: when updating to Karabo 2.11 some instrument macros have to be changed. CTRL will care that and will get in touch with
instruments to properly organize it.
DA:
- Couple of E-log entries referring to DA but nothing major to report.
EEE:
- Reminder to FXE, MID, SPB, SCS: please check processes for eligible documentation.
Bruno F. And Joern R. inform that in Alfresco there are various folders with pending documentations (automatic e-mail was sent to referents). Please contact Jorn
Reifschlager in case of doubts.
- PES system in SA3: digitizers were stopping after acquiring data for a random amount of time. This will be addressed during the maintenance period, as the fix needs
several hours of ZZ access.
VAC:
- Not present.
XPD:
- Nothing to report.
DET:
- Couple of issues with FXE Jungfrau detector this morning: solved or under fix.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Everything is fine with Lasers.
UNSYS:
- All good with Undulators.
SPB:
- Progressing well with beam activities during the nights.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Nothing to report. HED does not receive beam this week.
SCS:
- Nothing to report.
SQS:
- Nothing to report. PES digitizer issue affects SQS but one can continue with it until the maintenance period. |
Notes XO Integration Meeting 19.05.2021, posted by Frederik Wolff-Fabris on 19 May 2021, 10:13
|
General Announcement:
- A. Galler says the Interlock Tests schedule is placed at the EUXFEL Infrastructure Calendar.
Concerning the TUV tests at Schenefeld side, this happens between June 30th and July 2nd (the 2nd is a reserve date). When a detailed schedule is made, this will be
communicated.
Tomasz J. asks on the effects for the PP-lasers when, e.g. triggering NOT-AUS. XO will enquire with Zunaira A.
CTRL:
- The ADQ digitizer fix for SQS is working well. If other instruments would like it deployed, please contact CTRL.
- Bassler camers upgrade worked for SPB. One can also contact CTRL for the deployment.
- ITDM will need to exchange memory module in SA2 - Karabo will go down. This has to be scheduled between tomorrow and Friday. It will be coordinated by the PRC and CTRL.
DA:
- Nothing to report.
EEE:
- Yesterday incident in SQS: faulty PLC CPU triggered the OFF for 5 CPUs. Four of them have recovered and the fifth one will be replaced or fixed today.
VAC:
- Not present.
XPD:
- FD studies are ongoing this week. Nothing else to report.
DET:
- Nothing to report.
ITDM:
- Today the update on the Maxwell cluster. Action take place between 10:00 and 12:00 and running jobs are not affected. Submission of new jobs will not be possible during
this time.
- Issues with Network yesterday: Janusz S. says they do not have yet the final e-mail from Desy-IT.
XRO:
- A. Galler says that on Monday the beam power reached 32 W at 2125 bunches. Mirrors were working properly during the whole test period.
LAS:
- All fine with PP-lasers.
- Tomasz J. mentions LAS will ask CTRL for update on the functionality of sync system properties for the PP-laser. In case instruments have also interest on it please
contact LAS.
UNSYS:
- Issue with cell#18/SA1 quad mover: it was agreed to run in this condition until the shutdown.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- All fine at HED.
SCS:
- Nothing to report.
SQS:
- PLC issue yesterday afternoon: vacuum was lost from ^-11 to ^-6. This was a near-miss and almost made SQS to cancel next experiment. It seems to be under fix now. |
Notes XO Integration Meeting 17.05.2021, posted by Frederik Wolff-Fabris on 17 May 2021, 10:25
|
General Announcement:
- Adrian Mancuso tells on the workaround for the crosstalk between SA1/SA3 documented by Harald Sinn. The idea was to switch off the feedback prior the bunch changes.
Harald S. says that it is not sure if this has worked. Adrian M. says that fir the future this has to be addressed as both SA1 and SA3 would like to change bunch number.
CTRL:
- Follow up from the DOC last week: some cameras in HED started in unexpected states. This has been understood and when the update is ready the deployment has to be
coordinated with HED.
DA:
- From DOC OCDs side only minor issues, that were solved by restarting devices.
EEE:
- Due to the special FD week starting today, there is no DOC this week.
The DRCs (Patrick G. until Thursday and Monica T. from Thursday) will receive the calls: for normal issues, please send a ticket; for urgent cases call the DRCs.
One can also contact directly the individual groups.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Incident at SPB with AGIPD detector: damage to several pixels. The cause for it is not yet known.
The pixel modules have the replaced and the cause relates to intense radiation. It's important to follow up and understand how it happens. SPB will investigate and
communicate findings.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Everything good with Lasers.
UNSYS:
- Issue with Quad mover in cell18/SA1 - Suren A. could address it without need of access but further investigations have to be done on-site.
SPB:
- Small injection incident over the weekend.
- Problem with one DAQ: number of pulses was OK but not the values. This was solved by the DOC and some data were wrongly saved for the users (attenuation values).
FXE:
- Timing tool tests at FXE during the night: crosstalk between SA1/SA3. When number of SA1 bunches were changed, the beam pointing was changing in SA3.
So FXE was requested to do not change bunch numbers what hindered the tests.
- LPD issue: progress has been made last week and is being accelerated.
MID:
- Quick warning: until Wednesday some Seeding tests will be ongoing at MID.
HED:
- Successfully experiment last week: HED thank all involved people.
- Addressing CTRL deployment: HED will contact CTRL after this meeting to discuss some details before the deployment action goes on.
- On last Friday at 13:00 one camera corrected configurated did not acquire data and DOC was contacted.
- On Saturday the timing drifted linearly for 0.5ps. HED contacted PRC and DRC and is not yet completely understood.
SCS:
- Nothing to report.
SQS:
- Confirm what FXE has said on the crosstalk with SA1/SA3.
- Sometimes MCP detector was inserted and trigger the EPS to beam stop. Jan G. says that there was a mistake in the interlock configurations for going above 30 bunches.
When SQS wanted to go to 67 bunches it triggered the beam OFF. |
Notes XO Integration Meeting 12.05.2021, posted by Arvid Gebhardt on 12 May 2021, 09:21
|
General Announcement:
- User Operations Meeting takes place on next Friday 09:00.
CTRL:
- CTRL would like to excuse itself to people receiving an automatic SMS tonight for notification. This was a test message and now it’s turned OFF (Good side: test for lasers went well).
- ADQ digitizers: two fixes are driven and based for SQS commissioning. CTRL encourage instruments to get in touch with contacts for making the update for the last version.
DA:
- Follow up from Monday / calibration update (on behalf of cal team): it worked as planned. AGIPD seems to run faster now.
- For SQS Remi detectors/digitizers can also be used.
EEE:
- nothing to report
VAC:
- nothing to report
XPD:
- are preparing for FD studies next week
DET:
- Follow-up on LPD online calibration pipeline pointed last Monday’s meeting: work is on progress and is expected to be ready for next week.
- DAQ processing for AGIPD: a minor bug in validation is reported and under investigation. But it is currently working.
- Issue reported for SPB for data aggregators: follow up with CTRL.
ITDM:
- nothing to report
XRO:
- nothing to report
LAS:
- nothing to report
UNSYS:
- nothing to report
SPB:
- not present
FXE:
- Thank to DET and DA for pipeline calibration. Initially it looks promising.
- Last week: digitizer went to wrong configuration. To be fixed with new updated.
- FXE will use the beam during the nights. D. Khakhulin will inform PRC and XO via e-mail.
MID:
- all fine
HED:
- not present
SCS:
- nothing to report
SQS:
- nothing to add |
Notes XO Integration Meeting 10.05.2021, posted by Frederik Wolff-Fabris on 10 May 2021, 10:05
|
General Announcement:
- Two smoke sensors affected in SA3 triggering fire alarm: The first one could be reset but for the second one the reset did not work and fire brigade was in.
Beam operation was recovered only after the External company came to reset it. The downtime of about 4 hours for SA1/SA3 operation happened.
CTRL:
- Nothing to report.
DA:
- On behalf of calibration team: this morning a major upgrade will be done. No access to calibration services during this time.
EEE:
- One loop in FXE was not working during the weekend. Issues with movs/motor loops and the cause has to be investigated (old scripts were in use). No devices were
damaged.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Deployment for HED calibration taking place shall fix last week's issue.
- A fix for DAQ on AGIPD to be done.
- FXE issue with LPD calibration: still open. This is not related to the DA deployment today. Jolanta S-D. will follow it inside DET.
ITDM:
- Nothing to report.
XRO:
- Not present.
LAS:
- Nothing to report.
UNSYS:
- Fire alarms in SA3 / cell#22-23 on Saturday evening: BKR informed that could be related to synchrotron radiation on the detector signal. Fire brigade was on-site but no
fire was found.
SPB:
- Nothing to report.
FXE:
- Thanks to EEE help with Gotthard.
- M2 Mirror pitch issues seems to be back to normal as before. Antje T. (XRO) made ZZ to correct loose gear box.
- Motor loops issues found and addressed by EEE.
MID:
- Nothing to report.
HED:
- Motor stop working and ticket has been written.
- Preparing for experiment this week.
SCS:
- Zoom problems last week has been taken care; Thanks to ITDM.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 05.05.2021, posted by Frederik Wolff-Fabris on 05 May 2021, 10:20
|
General Announcement:
-
CTRL:
- The device clone deployment went smoothly yesterday.
DA:
- Nothing to report.
EEE:
- Call from DOC referring to motor mirrors in SA1: XRO is investigating together with FXE. The issue is that pitch accuracy can't be reproduced as before.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Issue with offline calibration mention last Monday: the problem has been identified and a software fix is been worked (for HED detector). The calibration team is
working on it.
ITDM:
- Nothing to report.
XRO:
- Nothing to report besides the SA1 mirror motors.
LAS:
- Problem with Optics in SA3 PP-laser (either coating degradation or lack of clean) was found. The mirror will be rotated.
- DOC reported a synchronization issue in SA2. It is not a LAS issue, has to be fixed by DOC.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Chris Milne mentions the steering issues with M2 and M3 mirrors and XRO is looking into it.
MID:
- Synchronization issue with Laser is fixed.
- DAQ issue also fixed during the night.
HED:
- Things reported on Monday have been looked up.
- Train-ID issue: there is a walk around for it and test/fix to be done next week.
SCS:
- SCS asks for a solution to share screens with users in the control room (general solution also applicable to other instruments?).
K. Wrona says ITDM has solutions that were tested last week at FXE and was working properly.
An offline discussion between ITDM and SCS will take place to set the solution.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 03.05.2021, posted by Frederik Wolff-Fabris on 03 May 2021, 10:06
|
General Announcement:
- The deadline for submitting the summer maintenance tasks is ended. In case you have not done so, please submit it as soon as possible.
- Monica T. as DRC invites people to attend the DRC jour-fixe meeting.
CTRL:
- Bug fix for the device clone is prepared and will be deployed today. The clone devices will go down for a couple of minutes only.
- ADC 412 bug: there is a fix and instruments shall get in touch with their contacts for the deployment.
DA:
- One OCD call related to Karabo bridge and usage of online monitoring: the problem was not related to Karabo but with an environment conflict.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- All PP-lasers are up and running.
UNSYS:
- Waiting for the ZZ in SA1 and SA3.
SPB:
- User experiments last week went well again.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- HED run last week went well - quite successful with tests with Laser.
- Missing Train-ID issue was reported to expert groups; All the devices are not correctly seen (arriving some trains later);
Steffen Hauf says CTRL are working and will be ready to test. There is a chance for tests in the coming week or still this week.
An offline discussion will take place between CTRL and HED.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 28.04.2021, posted by Frederik Wolff-Fabris on 28 Apr 2021, 10:04
|
General Announcement:
- A. Galler points to the deadline for submitting summer maintenance task list on April 30th.
- XO would like to remind instruments that in case of operation failure related to Interlocks issue, please also include the PRCs (prc@xfel.eu) on the communications.
The Photon Run Coordinators are the first contact point that can help in such situations.
- Tomasz J. asks on the reason for the water-cooling problems with SA1 PP-laser last Monday. A. Galler says the origin was a power glitch that caused the interruption of
water systems.
Raul V. says that it's also seen in some tunnel devices but because of the changes in interlock logics this worked nicely.
- Zunaira A. asks why many power glitches are happening. Andreas G. says there were this year 3 events with different causes.
Last Monday's one was a real power glitch from distribution line; the one on Easter Sunday is under investigation via an 8D report and the one last week was caused by
human error.
CTRL:
- A bug was found in the devices for HED (clown system). A test in HED will be done prior a full deployment.
DA:
- Not present.
EEE:
- Status of ADQs in SQS: Bruno F. says that it worked yesterday and suspicion is that data were saved locally. DOC and CTRL are following with this issue.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- All fine for Detectors.
ITDM:
- New feature for data migration is set – main ADC.
XRO:
- Nothing to report.
LAS:
- All lasers are UP and running.
UNSYS:
- Issue with phase shifter in SA2/cell26 this week was solved (Doocs problem).
- Waiting for the ZZ access in SA1/SA3. A. Galler says that when time window opens this will be informed.
SPB:
- Experiment is going smoothly this week.
FXE:
- Last week HED had a few issues with brig brother server when scanning the undulators. The Big Brother was setting bunches back to zero. Issue will be followed up with
machine.
MID:
- All fine at MID.
HED:
- Nothing to add; thanks to CTRL for looking after the scan tool.
- Test with Relax synchronization will be done tonight.
SCS:
- Started using the PES and recorded runs - about 35% of missing train-IDs. A. Galler says that one point of contact is via DOC.
Steffen Hauf askes how many boards were using and Laurent M. informs 4 that run in the SCS topic.
SQS:
- Apart of the shift team in quarantine, everything seems to be going fine. |
Notes XO Integration Meeting 26.04.2021, posted by Frederik Wolff-Fabris on 26 Apr 2021, 11:08
|
General Announcement:
- The deadline to submit the maintenance task list is on 30.04. We thank all the groups that have already submitted. In case of any question and comments, please contact
us at xo-planning@xfel.eu
- Patrick Gessler reminds on the DRC meeting on Mondays 13:00. This meeting is done together with instruments to exchange information for the weekly plan. The e-mail for
contact is drc-jour-fixe@xfel.eu
- Andreas G. says that with the new 7-11 operation mode, XO is interested in knowing how longtime instruments need to be ready to take the beam in the early morning.
Please inform us for that XO checks the efficiency of this model.
CTRL:
- Remind on Beckhoff training next Wednesday (following e-mail to readiness).
- Upgrades of two Karaboo Beckhoff servers today (Karabo devices go down for 5-10 minutes, proposed to be done today during lunch time): SA2 master tunnel loop; and the
HED loop.
- Bug on ADQ for twelve devices, affecting SQS and others: EEE has discovered it and CTRL is analyzing to come up with a fix.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- PES in SA3 was provided and operated last week for instrument's usage.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Maurizio V. requests the SA2 experiments to provide information to XRO on issues related with mirror alignments or transport beamline.
LAS:
- Received info from Desy on a power glitch today at around 6AM. At the same time SA1 Amphos switched OFF due to not enough water.
It’s under investigation with TS. Jan G. points that also there was a beam loss in delivery around the same time.
- SA2 and SA3 PP-lasers are still in operation.
UNSYS:
- Issue with undulator SA1 /cell33 and cell1/SA3 needs ZZ. Currently the impact in operation is minimal and ZZ is postponed to the next available date.
SPB:
- Nothing to report.
FXE:
- Run with beam delivery went smooth.
MID:
- Quite successful beam week with small hick-ups to be addressed with experts.
HED:
- Commissioning beamtime this week; Data analysis will be made this week.
- Test of the PLC synchronization with Relax laser to be done this week.
SCS:
- Not present.
SQS:
- uTCA in ADQ had been mentioned already by CTRL. It has been tackled. |
Notes XO Integration Meeting 21.04.2021, posted by Frederik Wolff-Fabris on 21 Apr 2021, 10:50
|
General Announcement:
- XO reminds on the deadline of 30.04 for submitting the Summer maintenance task lists. For this, please do so sending the list to xo-planning@xfel.eu.
CTRL:
- Update on undulator Doocs side: CTRL needs to test for two or multi-colors schemes and asks instruments to CTRL if there are plans to use it.
- CTRL will schedule a short practical training on Beckhoff devices for next Wednesday. An announcement will be made through readiness mailing list.
DA:
- Nothing to report.
EEE:
- From the PLC side, EEE is following up the MID/CRLs interlock case.
- If groups submit a change request needed to be implemented during summer maintenance time, please contact and inform EEE beforehand.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- The network cable exchange mentioned last Monday was done.
XRO:
- Not present.
LAS:
- All 3 PP-lasers are in operation.
UNSYS:
- After power glitch on Monday in SA1, about half the undulator cells were half power cycled. Most could be recovered expect SA1/cell33 and SA3/cell1. This will require
ZZ accesses.
XO states the power failure was man-made caused.
- Quad mover sudden move: Lars F. from MXL does not have records on the command given for the sudden move and therefore the root cause can't be traced.
SPB:
- Nothing to report.
FXE:
- Beam since yesterday; one interlock issues.
Nerea J. says a ticket was received and the interlock issue was related to a Imager that always come up activated. This will be removed from the EPS.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Undulator photon energy couldn't be changed via Karabo. Jan-Torben D. will talk to Steffen H; David Hickin adds that the location was changed and the Karabo side is
done/corrected.
One needs the BKR connection/authorization and this will be organized by CTRL/Steffen H.
SQS:
- Issue with digitizer on work. It is needed for next week. |
Notes XO Integration Meeting 19.04.2021, posted by Frederik Wolff-Fabris on 19 Apr 2021, 11:39
|
General Announcement:
- Remind on the Summer maintenance period: deadline to submit task list is April 30th. XO invite all groups to submit by this date.
CTRL:
- Remind on the update of some specific Karabo installations to fix bugs (Data logger installations) to start today (e-mail was sent to the readiness).
- Bunch pattern statistics devices need to be fixed today at the same time as the data loggers (relates to lose of time stamps in lasers).
- Upgrade on instruments are coordinated by ICIs; Updates of Mc2 motors today.
- Z. Ansari asks if updates of MID will affect interlocks as well; Valerii B. answers this is a Karabo update and not PLCs, so no effect is expected.
- On recent days CTRL finds Karabo broker pileup and this was successfully understood; DOC was preventively monitoring it and fixed it beforehand.
DA:
- No OCD calls for DA;
- Collaboration and preparation for supporting SPB/SFX last week and for beamtime continuation in two weeks.
EEE:
- Today 08:30 it was found SA1 tunnel racks lost power. The uTCAs were also down but not the one in room 1.14. It affected SA1 tunnels loops such as Vacuum and motion.
The guess is on a power glitch. XO will bring up to the machine side and eventually correlated with the Easter Sunday's event.
VAC:
- Recovering the SA1 tunnel loops from 08:30 downtime. All seems to be fine right now.
- VAC asks if FXE is running cryo coolers. FXE tells they were running and will check the signals for possible issues.
XPD:
- HIREX motors in SA1 were OFF this morning.
DET:
- One OCD call was received and solved; DET is not aware of issues with Detectors.
ITDM:
- On the Wednesday problem with network (in SA1, the online calibration was down): Today a cable will be exchanged.
- ITDM was involved with experiments together with DA on offline calibrations.
- As DRC, K. Wrona reminds today that 1PM there is the DRC meeting and the participation of instruments is strongly encouraged.
XRO:
- Nothing to report.
LAS:
- SA1 PP-laser is fine.
- SA2 PP-lase with small issue with regulations. Tickets sent to CTRL.
- SA3 PP-laser with several small problems: damage to optics (need to exchange one AMPHOS stage); interlock tripped due to door problems and LAS went OFF.
LAS hopes to bring back laser and make available for instruments by tomorrow.
UNSYS:
- SA1/cell12 Quad mover sudden move last week: it was not related to a software issue and is under further investigation on Doocs side. A Doocs command was sent.
SPB:
- Experiment last week: quite successful and not many issues.
FXE:
- Nothing to add.
MID:
- All fine.
HED:
- A Motor on the CRL HED chamber was engaged and not moving (hardware issue). It may need a ZZ. It is not related to the CRL's arms.
- SEPS interlock tripped machine on Friday almost midnight (colleagues close shutter before moving CRLs). The beam permission was restored after lens removed from beam.
- ADC is offline. HED will contact Bruno F. offline.
SCS:
- Not present.
SQS:
- Commissioning for preparing next experiment.
- Issue with digitizers that can't be used: Bruno F. adds that action is scheduled for 14:00 as discussed with SQS's members. |
Notes XO Integration Meeting 14.04.2021, posted by Frederik Wolff-Fabris on 14 Apr 2021, 10:50
|
General Announcement:
- We would like to remind groups that the deadline for submitting to XO the feedback on the planned activities for the Summer/21 maintenance time is on April 30rd.
In case of questions please contact XO via xo-planning@xfel.eu
CTRL:
- Updates announced on last XO Int. meeting will take place next Monday (unless if a group protests). This means a short downtime on data loggers in tunnels. An e-mail
with a reminder will be sent.
- Upgrade for the bunch pattern statistics devices to take place (also will be offline for few minutes).
- Issues with MC2 motors: the workaround is to shutdown device and click/instantiate device. A fix will be deployed.
DA:
- DA involved and continuing with "Moonshot" coordination meeting.
EEE:
- Nothing to add.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Announcement with respect to Maxwell cluster: ITDM will change partition schemes in the next Summer maintenance.
Three new partitions will be introduced, for calibrations and a special one for reservations for beamtime (UPEX related);
A new feature is that if Maxwell is full of jobs, one can pre-empty it by killing jobs and putting back in the queue.
XRO:
- Yesterday beam going to HED: some problems were found regarding parameter configurations.
XRO reminds SA2 instruments to be careful and check devices positions and parameters are they may be changed in relation to April 1st due to the issue with PLCs during
Easter Sunday.
- Zunaira A. says there were problems with CRLs arms configurations. Some devices had to be restored but worked normally.
LAS:
- All lasers are fine.
- Shutters in SA1 PP-Laser were in maintenance mode - without defined reason.
- SA2 PP-laser is fine.
- SA3 PP-laser has degradation due to the cleaning. XO will address the issue with LAS.
UNSYS:
- All fine.
SPB:
- Running user experiment. Low photon energy yesterday (0.8mJ) and drop intensity over the full 352 bunch train but today it got better.
FXE:
- Nothing to add.
MID:
- All fine.
HED:
- All motors lost positions, e.g. mono, screens being driven (some in and not supposed to be in). CRL1 issue the position was lost at the axis; motors not being homed;
Zuzana K. will summarize main issues and forward to XO.
- Steffen H asks for a summary of encoders and Zunaira A. also asks on the use of interlocks and CRLs/shutter to evaluate if all is fine. Zuzana K. says HED did not work
yet on the CRL2 but will communicate as soon as experience is get.
SCS:
- Question to CTRL on the workaround of MC2 motors. Steffen H. says to shutdown device, check configuration and instantiate it again. An instruction e-mail will be sent
to readiness list.
SQS:
- Nothing to report. |