ID |
Date |
Author |
Location |
Severity |
Subject |
296
|
31 Jan 2022, 10:03 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 31.01.2022 |
General Announcement:
- On this current run, beam delivery to instruments will start on Tuesday at 07:00AM.
CTRL:
- New version of DeviceClone and Doocsml device will be updated today which contains a bug fix;
- Minor update on Karabo framework version on the datalogger machines will be performed today
- CTRL will proceed with the update today at 13:00 if they didn't hear any objection till 12:00. Intervention will not take longer than half an hour. No need to check afterwards.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Notification on power interruptions on the weekend: it seems to have no impact but please check the systems.
- Zunaira A. says the AIBS looses connections to network during glitches and has to be restarted if not masked. She will take a look.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- No big issues during the weekend; smaller ones were solved.
LAS:
- SA2 PP-laser is ON;
- SA3 PP-laser will need tomorrow to fix final things.
- SA1 PP-laser: LAS has negotiated with instruments to further investigate more.
UNSYS:
- SA1 issue on Saturday: problem with CPU;
- Ongoing tests with Apple-X this morning.
SPB:
- Aligning instrument to take beam tomorrow.
FXE:
- Nothing to report.
MID:
- All fine at MID.
- MID did not look yet in the PP-laser but will do so tomorrow or Wednesday.
HED:
- Waiting for DAQ rollout. In contact already with DA.
- Zunaira A. reminds on the reply for scheduling the hutch search training. HED will inform after their afternoon group meeting.
SCS:
- Nothing to report. SCS will also make a schedule and will contact Zunaira A.
SQS:
- Nothing to report. |
298
|
07 Feb 2022, 10:56 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 07.02.2022 |
General Announcement:
- Power glitch on Sunday morning at around 04:00AM seems to have no effect at machine or Schenefeld site.
CTRL:
- During winter shutdown the bunch arrival monitor was updated. An e-mail will be sent reflecting the changes.
DA:
- Single OCD call was solved: configuration of train match bridge to eXtra foam.
EEE:
- Two tickets from MID shall be followed this week.
- Tickets from DOC on SQS (solved) and PES (to be fixed by DA).
VAC:
- All fine with VAC.
XPD:
- All fine with Diagnostics.
DET:
- Nothing to report.
ITDM:
- Wednesday there will be shutdown of rack room in ground floor (604): details were sent by e-mail and there were no concerns/feedback from potentially affected people.
XRO:
- Nothing to report.
LAS:
- All 3 PP-lasers are stable and will be available by the end of the day.
UNSYS:
- Issue FRI/SAT: bad communication between balcony and SASE1 cells. It is solved but being further investigated.
- This morning there was a BKR request to go lower undulator gaps in SA2: gap can go as lower to 10.61mm.
SPB:
- Commissioning of AGIPD done; successful week.
FXE:
- Preparing the beamline; ready for beamtime.
MID:
- Issue with RF window: it was too short when they received the beam. Solved by PRC and instruments for fast solution.
- EEE had to solve an issue.
- Cameras in the tunnel monochromator may need a ZZ. This afternoon VAC will join for a ZZ and fix it.
- Week was successful.
HED:
- Quite successful beam usage last week.
- Issues with fast ADC devices not working until Wednesday. It worked for 24hs and then had to be restarted again and came out with wrong settings.
This needs further investigation and is needed for users this week. CTRL will follow up.
SCS:
- Preparing with UAC from tomorrow.
SQS:
- Safety problem with magnet interlock in one door: a loose screw was fixed (for the time beam).
- Issue with digitizer: SQS will clarify with CTRL.
- Recheck calibration of monochromator (needs cameras looking to the motor): it seems that the camera has to be discussed with XRO.
Martin D. says that currently Maurizio V. is in the tunnel and SQS shall contact for the possibility of a quick fix now.
- Zunaira A. says that if beam shutter closes without apparent reason: it can be related to emergency off, panic switch, air box or the maglock. It happens 3 times on the same day. If this happens again,
it may trip the entire accelerator.
People in safety group or laser interlock are responsible for fixing the maglocks. A. Galler asks Zunaira A. to send an e-mail to xo-info@xfel.eu.
Alberto Defanis says that Eric B. has fixed. Zunaira comments that this fix was momentarily for the weekend. Further discussion will follow. |
299
|
09 Feb 2022, 10:09 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 09.02.2022 |
General Announcement:
CTRL:
- GATT broker pile up: there is a fix available but not foreseen to be updated now. The middle layer will be fixed in cw09 (blue week). This will be coordinated with XO.
- Raul V. completes that 1. slowness of the GATT from last week can be fixed next Monday (which triggers a pile up); 2. More invasive changes mentioned by Wajid E. to be done in a later point.
- Wajid E. adds that if problems appear again (unresponsive middle layer), please call DOC.
DA:
- Nothing to report.
EEE:
- Finishing smaller tasks.
- Fast ADC issue is taken over by the fast electronics team. The cooling issue may be related to connection problems with the board. EEE monitors if last Monday fix was enough.
VAC:
- Nothing else apart of the GATT discussion.
XPD:
- XGM issue in XTD2: someone changes the settings leading to an overload in the XGM. A. Galler says that BKR operators did not touch it and one still looks to find who triggered the changes.
- Calorimeter and laser polarimeter to be installed at FXE on next days.
- A. Koch says there was an update of the DAQ in tunnels; that was tested on Monday afternoon and if problems are found, please contact A. Koch.
DET:
- Nothing to report.
ITDM:
- Nothing to report apart from ground floor room 604 work.
XRO:
- XRO contact control integrations to work on motor layer devices for March.
LAS:
- Nagios Karabo system still do not work.
UNSYS:
- Nothing apart of the smaller gap limitation in SA2 system for the 2-color test last Monday.
SPB:
- Nothing to report.
FXE:
- Hiccup with train-IDs not being synchronized: a ticket was sent to CTRL.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- Issues with middle layer devices that had to be restarted: Laurent M. will check on the status with SCS colleagues.
SQS:
- Issue with XTD10 PES digitizer: there is an open ticket in progress.
- Trigger of hutch shutter last week: the loose maglock has been addressed. |
300
|
14 Feb 2022, 10:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 14.02.2022 |
General Announcement:
- Thanks to instruments that posted at the Operations-2022 e-log. Please make regular entries that allow PRCs to recognize eventual issues that have to be tracked down.
CTRL:
- Nothing to report.
DA:
- Friday there was issues with Data Aggregators processing in several detectors. DOC found it was an issue with the new version. It was fixed by Saturday by working with a separate routing.
This was a bug in the new version (time of template creation). A final fix will be installed in a suitable date.
EEE:
- Nothing to report.
VAC:
- Intervention to solve issue GATT issues is postponed by CTRL (patch work is almost ready). To be discussed between Raul V., CTRL once deployment can be done only on Mondays.
XPD:
- Issue noticed that XGM data were showing missing trains at the start of last week. Tickets were issues and it was cured by end of the week – it affected only one XGM. Reason for missing trains must be
asked to Theo M.
- SA1 pulse energy was around 4.5mJ; a really good performance.
- Preparing calorimeter in SA1 hutch for experiment.
DET:
- Two OCD calls (one from HED) and one from SPB on a detector. Both fixed by DOC / OCDs.
ITDM:
- Incident on Saturday with influx DB cluster: happened by 3:00AM; started investigating during the morning. By 14:00-15:00 the cluster came back. Data synchronization is ongoing. Not sure yet if there is
lost data, under investigation.
XRO:
- Main BKR scene will contain a button to Graphana page.
LAS:
- Working with CTRL on Nagios issues.
UNSYS:
- Nothing to report.
SPB:
- Saturday problem with history of motor position at Karabo: solved by DOC.
- Broken cable for encoder of focusing mirror still has to be fixed (MKB focusing mirror in hutch).
FXE:
- Preparing for the week.
MID:
- All fine at MID.
HED:
- Week went good: successful user experiment.
- Beam was drifting quickly between left/right. Not sure if this is related to the new HED feedback loops. Under investigation.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
301
|
16 Feb 2022, 10:42 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 16.02.2022 |
General Announcement:
- XO launched a new XO Alfresco presence. Please check there for information related to operations and do not hesitate in giving suggestions or raising questions.
Adam R. asks for the schedule for the PRCs and DRCs. A. Galler says that the PRCs are included in the photon delivery tables. A discussion with Olivier M. on the DRC will take place.
Naresh K. asks on the status display: it is current a screenshot for the moment the page is loaded. It is not yet a live display.
- A. Galler reminds instruments to make entries to the Operations-2022 E-log to help the PRCs to track down and spot problems.
CTRL:
- Intervention on Monday GATT update: action to be done again next Monday to increase performance.
DA:
- Nothing to report.
EEE:
- Intervention on HED to replace a fast ADC card that lost communication to CPU. Replacement to take place today.
VAC:
- Nothing to report.
XPD:
- A. Koch reports that yesterday the Karabo fast-X was slow.
DET:
- Nothing to report.
- Olivier M. also observed slow fast-X connection to gateway.
ITDM:
- Janusz S. says the fast-X issues can be related to an overload yesterday.
- Login to Influx had intermittent connection for a half day yesterday n the afternoon. Investigation being taking place of it relates to fastX.
XRO:
- Nothing to report.
LAS:
- Issues with SA1 PP-laser: overshooting in the CM station. Maintenance this morning is taking place. Current timing is not as good as it could be.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Nothing to report apart of the ADC issue.
SCS:
- All fine at SCS.
SQS:
- SQS shutter issue from yesterday 15.02: problem was solved and it was due to a card misbehaving.
Zunaira A. says that the problem was with the Lasermet interlocks. Work done in the maintenance affected signals going to radiation protection interlocks. The particular signal was missing and after a fix
it worked again. |
302
|
21 Feb 2022, 11:57 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 21.02.2022 |
General Announcement:
- Philipp S. says a downtime for the calibration / GPFS update is to be done today at 12:00.
- Raul V. says that the biggest issue of last PRC week was related to the FXE beam pointing. In general, was a quiet and productive week. Please, instrument shall remind to make entries in the operation e-
log.
CTRL:
- In SA3 GATT activity will be performed today (fix to slowness) as beam is down and there is no activity.
DA:
- Announcement will come today: restart GPFS issue / Maxwell to be restarted.
EEE:
- One OCD on Saturday: issue with SDL motors in MID; to be followed up today.
VAC:
- Nothing else apart of the Gatt fix.
XPD:
- Shimadzu camera is installed in XTD6. A test has to be done and organized with instruments.
DET:
- Nothing to report.
ITDM:
- Fix to be deployed today: plan is to deploy Maxwell cluster done in the background; it shall not disturb any user of the system. All other storages with login to be done in March and still to be
confirmed.
- Slow data investigated: get wrong values with time stamp that could be corrected. Further investigation will take place this week.
XRO:
- FXE stability pointing issues on Friday afternoon: Raul V. placed a report in Operation E-log.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Pointing issue from Friday;
- In DAQ the BAM device name has changed and could not be used. Please keep the compatibility for names; A. Silenzi will check with CTRL.
MID:
- Motor problems: in touch with CTRL.
- Today morning the CRL2 lenses were retracted but not giving the proper signal. It has been under investigation.
- Issue with DAQ: slow devices were saving empty data (no entries); at least happening with two devices. It has been reported and one is fixed and the second is under investigation.
HED:
- Initial setup this afternoon for the two-color operation.
SCS:
- Not present.
SQS:
- Not present. |
303
|
23 Feb 2022, 10:30 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 23.02.2022 |
General Announcement:
- A. Galler reminds instruments and support groups to feel welcome in making regular entries on the Operations-2022 e-log.
CTRL:
- FastX of GATT middle layer devices was updated last Monday; the slowness issue is improved and fixed.
- Analyzing to upgrade the broker pile-up issue at Karabo: probably to be fixed by next Monday.
- On the BAM monitor: few properties were changed in the DOOCS side. Changes are transparent to DAQ and all the data must be able to be stored.
Problem appears when data shall be analyzed and then must be looked to the new DAQ. In case of issues please get in touch with CTRL.
DA:
- Maxwell GPFS access rights bug: patch was deployed on Monday and went successfully.
- Next week there will be an update of calibration (further announcement to be done).
EEE:
- Started installing Coax cables for SXP.
- LBM in MID: EEE contacted Beckhoff and will try to get a fix done still this week.
VAC:
- Nothing to report.
XPD:
- Issues with Desy mobile phones: not all phones were equally affected. Issue seems to be fixed this morning.
- In SA2 the SRA slits were too wide open (probably from tuning times). They were closed yesterday night after users finished work.
- Beam delivery to instruments is running fine and smooth.
DET:
- AGIPD detector at SPB: issue with one of 16 modules not sending data since yesterday. Investigation ongoing with DOC and experts.
Jolanta S.-D. says that this is a new issue which happened twice yesterday and Desy is contacted. An intervention for the hardware during the blue week shall take place. It may be related to electronics
outside detector.
ITDM:
- Issue with "phantom" data appearing at DAQ in SA3 instrument: investigation shows it can't be reproduced; maybe it relates to data arriving earlier. It was only visible at DOOCS devices (Undulators
device). Under further investigation by ITDM.
XRO:
- CRLs in SA2 fixed by ZZ.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing apart of the detector report.
FXE:
- Thanks to CTRL for addressing BAM issue.
MID:
- Thanks for solving CRL2 issue on Monday.
HED:
- Two-color mode working with nice energies. Investigation source shape from undulators with lenses.
SCS:
- Nothing to report.
SQS:
- Not present. |
304
|
28 Feb 2022, 10:35 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 28.02.2022 |
General Announcement:
- This week a facility development week takes place without user operation. XO thanks the groups for posting at the Operations-2022 e-log.
CTRL:
- Update on the SA3 GATT loop (middle layers devices) to be done following to find a good time slot.
DA:
- Today from 10:00-12:00 the calibration system will be down due to an update.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Nothing to report concerning XPD; SCS made an enter that XGM went OFF for a day.
- Regarding PRC, the machine was running well most of the time; problem with XTD2 Air conditioning due to a fire safety flap malfunctioning happened (downtime of several hours in SA1);
on Sunday magnet power supplies at the accelerator had a problem and took few hours to restore beam delivery as it required tunnel access.
- The two-color was not created in the right order but it was corrected afterwards. The bandwidths were adjusted a few times and HED said it went well, including their spectrometer.
- XO proposes that the info on the two-color (order) can be entered in the comment section of the delivery table.
DET:
- Issue with AGIPD at SPB did not happened again and currently there is no plan for any intervention.
ITDM:
- Janusz S. says the hardware monitoring needs to be rebooted for several servers (PC layers at SCS, MID) and a short downtime will happen. Proposition is to make today if no complains appear.
- Two control servers also need maintenance and proposal is to make next week.
XRO:
- Busy with activities in this blue week (e.g. Mirror calibrations); it may require CTRL support.
LAS:
- For next week a stage in SA1 will be replaced.
- A change of operation point in SA1/SPB and S3/SQS will happen this week. SCS also need a change working point to 1.1MHz.
UNSYS:
- Tests with the Apple-X to take place this week; preparation for commissioning and lasing attempt.
SPB:
- User time largely went OK; small troubles with DAQ and detector.
FXE:
- D3 is testing components in FXE hutch today.
- CTRL, ITDM and FXE will discuss on the previously mentioned updates.
MID:
- All fine at MID.
HED:
- A ticket send to EEE is being under work.
SCS:
- XGM high voltages went down to zero and was restarted by powering on again. Cause it is not clear and will be followed up by XPD and SCS members.
SQS:
- Continuing work of moving chamber to prepare for next week beamtime. Some PLCs for the chamber have to be reset motors and other things. EEE is helping with it. |
305
|
02 Mar 2022, 10:12 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 02.03.2022 |
General Announcement:
- There is a high probability that we will run out of Helium gas due to difficulties with stock. If an instrument needs He on next weeks please get in touch with TS to verify its availability.
CTRL:
- Last Monday intervention on the SA3 GATT middle layer servers was successful.
DA:
- Today a major upgrade of offline calibration is scheduled during which services won't be available.
EEE:
- Supporting different tasks at SQS (recommissioning motors), HED (motor encoders), SCS and FXE.
VAC:
- Ion pump in XTD9 shows a potential risk of tripping interlock: a ZZ on Friday 14:00 will take place to change parameters.
- Another ZZ in XTD2 to attempt to fix the solid attenuator in SA1 (hole in the arms reported by SPB / 0.5mm Si damaged); try to change the blocking mechanism.
- Dmitry K. asks on the Si intervention in XTD2: Martin D. says a change of Si is a major intervention as it involves need of clean tent etc… and will not be done this Friday.
The idea is to use a fresh spot on the attenuator. Conditions for interlocks have to be later checked with Liuba S.
XPD:
- ZZ on Monday to remove Shimadzu camera was successful.
DET:
- Nothing to report.
ITDM:
- Problem with Influx cluster which did not accepted data for 3 hours: search for fixes last days and today a new configuration will be applied, followed by restart afterwards (minutes downtime). No effect
in tunnels DAQ.
XRO:
- Commissioning beamtime for SA2 mirrors: available beamtime was short. The coating stripes were found in beamlines in respect to motor positions. More beamtime is needed to continue commissioning of M1,
M2 and M3 in XTD6.
LAS:
- Nothing to report.
UNSYS:
- Apple-X commissioning ongoing: working on Control systems commissioning. Last cell was controlled; more tests ongoing today. The phase shifter tables are not yet ready.
SPB:
- Working with different groups during the week.
FXE:
- D3 was present on Monday to test the BTM; it was successful.
MID:
- Radiation alarm triggered due to a false alarm at MID hutch. Thomas P. from HED helped by calling RPOs; W. Clement, Eric B. and M. Prollius helped to solve it.
HED:
- Preparing for the high intensity laser and also working on motors.
SCS:
- Not present.
SQS:
- Not present. |
306
|
07 Mar 2022, 10:39 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 07.03.2022 |
General Announcement:
- Accelerator is now running at 11.5GeV for the next 4 delivery weeks. The SA1 and SA3 systems show already good photon intensities and at SA2 the Self-seeding setup is taking place today.
CTRL:
- ITDM would like to reboot control machines in SA1 and FXE. It will proceed around lunch time.
DA:
- Update of offline calibration: it went fine, successful.
EEE:
- Replacement uTCA of HED optics today.
- Past week cables installation went fine.
- Motors in SA3 SQS laser setup and motor to be updated.
VAC:
- ZZ on solid attenuator XTD2: adjusted solid attenuator in-position (driving 2mm less in) to get a fresh spot on Si foil.
- ZZ on XTD9 ion vacuum pump: shows noble gas instability; adjusted set point; hope to survive until shutdown.
XPD:
- Setting up HIREX for HXRSS.
DET:
- Nothing major to report.
ITDM:
- Two servers to be updated (see CTRL report)
XRO:
- Nothing to report.
LAS:
- All fine with PP-laser.
UNSYS:
- Yesterday problem with cell 28/SA2: motor of undulator was stuck; after power cycle it worked.
- Issues with phase shifter in SA1: cell 26-31 is under investigation.
- Replacement of two phase shifter motors in SA3; possibly due to radiation damage.
SPB:
- Nothing to report.
FXE:
- Problem with LPD DAQ; ITDM solved last weekend.
MID:
- All fine.
HED:
- Exchange crate by EEE; busy preparing experiment.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
307
|
09 Mar 2022, 10:15 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 09.03.2022 |
General Announcement:
-
CTRL:
- Nothing major to report.
DA:
- Nothing to report.
EEE:
- Supporting instruments.
- Replacement uTCA in HED optics crate on Monday was done; being monitored during next two weeks.
VAC:
- Issue with SA3 broken ion pump controller in SA3; ZZ was done to replace controller.
- Issues when entering tunnel: XD2/4 PC turnstile crashed and D3had to reset software. Caused 30-min delay in entering tunnel.
XPD:
- Issue with FXE for XGM in their hutch: electronics did not work but was taking care by Theo / Fini. FXE has to confirm if this is fine now.
DET:
- Nothing to report.
ITDM:
- Intervention on Monday accordingly to the plan with reboot of 2 servers at SA1/FXE. Maybe another server to change in later date.
- Influx database issue on Monday: all data until 22:30 in the evening are not reliable (SA1 and SA3). Mid-term plan with CTRL to split SA1/SA3. Under investigation to find the cause.
XRO:
- FXE stability issue: last night beam was stable after the ZZ was done to switch off cooling systems of mirrors and clean tent.
LAS:
- 3 PP-lasers in operations: working
- Max L. asks for news on the winter maintenance 8D report.
- LAS send a ticket with TS concerning connection of front end UPS. Needs follow up.
- Announcement by TS on Saturday 25th June: concerning to LAS - what are the consequences? Max L. needs feedback to plan next summer maintenance period.
Tobias Clever says it is connected to construction of office building; needing new transformators and concerns power supply and not 35degrees water. XO will help with feedback.
UNSYS:
- SA1 PSs problems seemed to be solved at Desy side (new gap program done by Lars).
- Error signal from ambient field coils in cell14/SA1: Fixed by Suren A.
- In SA3 a problem with stuck quad mover: fixed by Suren A. with workaround;
- Cell 11 / SA3 phase shifter motor: could be another damaged device. It needs ZZ access.
SPB:
- Nothing to report.
FXE:
- Feedback on the XGM will be discussed internally.
- Beam jitter observed yesterday; thanks to all efforts.
- Energy change this morning took longtime.
MID:
- Monday afternoon: Karabo issue with control room due to broker pile-up; all fine now.
- No issues with HXRSS; used beam yesterday.
HED:
- Continuing prep for next week; also investigation on shielding.
SCS:
- Preparation for Liquid jet.
SQS:
- Nothing to report. |
308
|
14 Mar 2022, 10:41 |
Frederik Wolff-Fabris | General | Information | Notes XO Int. Meeting 14.03.2022 |
General Announcement:
- On the Summer-22 maintenance period: In about 2 weeks (by end of March) the call for the summer/22 maintenance period will be send to the groups.
CTRL:
- Deployment of new framework in order to fix an issue (this week): instruments shall agree with CTRL contacts on a suitable time. It affects cameras and/or similar devices that are installed.
DA:
- Report on last week issue with processing on Maxwell including offline calibration: people were not able to open files that were there; small selected notes were not present and were corrected on Friday.
Since then it seems no issues recurred.
EEE:
- Starting equipment installation of SXP racks in rack rooms.
- PLC side calls from SQS on weekend: issues with motors to be followed up; train-IDs not getting proper.
VAC:
- Not present.
XPD:
- XGMD in FXE hutch under investigation: new issue happened at the end of the week and may be again related to last Monday's one. People are working on it (XPD, FXE and Desy).
DET:
- One comment to DA: Jolanta S-D was on DOC and issue with offline for ePIX detector still occurring after Friday. Phillip S. says it was a different problem and it is fixed.
ITDM:
- Issue (2x) with Influx): still not understood; Data was not lost on last occurrence. Hick-ups and recovered. So far under investigation.
XRO:
- Jitter at SA1 in FXE last week: mostly likely the cooling in M1 and M2 or clean tent.
- No big issues after that.
LAS:
- Degradation in SA1 PP-laser with trigger software; maybe there is damage to optics. Being investigated today and can't guarantee when it will be working again.
- SA3 change of setpoint for SQS taking one-two days. Not sure if ready today.
- SA2 is running well.
UNSYS:
- Phase shifter cell11/S3: a ZZ is ongoing also fix with remotely procedure.
- Issue with quad mover cell18/SA1 is fixed remotely.
- Issue with Quad cell25/SA3: the ZZ is needed.
SPB:
- Issue with Agipd detector: fixed (one module not sending data).
- Agipd online preview devices: fixed;
- Laser timing issue: data was not collected.
FXE:
- Jitter problem already mentioned.
- XGMD issue under investigation.
- Laser problem: stop measuring on Saturday. Not sure if data was taken during the weekend.
MID:
- Difficult week.
- Broker pileup problem on Monday and Friday: Gabriele says it may be related to a memory leak - under investigation.
On Friday there were there sources identified to help getting rid of problems (Jolanta S.D); also GUI was a second source. Not fully understood the main source of Friday. It came back stable.
- Seeding beam had to be tuned on weekend as energy decreased by factor 5 when comparing to the best lasing at the week start. No scientific output could be achieved but feasibility with setup was
demonstrated.
HED:
- Wednesday optics hutch ADC was not working. Now it works fine.
- Full power with RELAX was tested and is being analyzed. Bassler cameras acquiring wrong train IDs. It shall be fixed.
SCS:
- Not present.
SQS:
- Successful week;
- Problem with motor being fixed by SQS, ME, EEE.
- Tool for online analysis was slow
- Heat load in the soft monochromator was seen and it has to be fixed in long term. |
309
|
16 Mar 2022, 10:39 |
Frederik Wolff-Fabris | General | Information | Notes XO Int. Meeting 16.03.2022 |
General Announcement:
- The Summer maintenance planning period is approaching and next Wednesday at the end of the XO Int. Meeting a presentation of the changes/improvements will be made.
Please communicate to the colleagues who are responsible for your group’s maintenance planning to join. This will be done after the regular round of discussions.
CTRL:
- Not present.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks to the groups.
VAC:
- Not present.
XPD:
- XGMD was fixed (ADC broken down twice last week): Desy people is also investigating; root cause not yet found.
- FEL Imager (YAG screen) in SA3 is found to be damaged. It was moved to a different spot for the time being.
- Ventilation in all tunnels caused beam jitter for 30-40 minutes yesterday.
It was caused by outside humidity (not common, 2-3 times per year) that filled up the filters and clogged it with water droplets leading to overrunning of the regulation schemes of pumps; then pressure
changed in tunnels.
DET:
- Nothing to report.
ITDM:
- InfluxDB: large issues yesterday (cluster crashed on MON evening); it took all day yesterday to brought up and data is now visible also at Karabo.
- ITDM has a root cause now and DAQ matrix devices were switched OFF yesterday which provides a fix. Last crash there was a lot of data that had to be synchronized and lacked resources.
- ITDM found a way to start without data from Monday evening to Tuesday 21:00. This data will be injected to InfluxDB / cluster in the coming days. In the meanwhile, one needs to use Graphana to look at
this data, not Karabo.
- Discussing future measures to trigger alarm to avoid the crash.
- Please don't start DAQ without the GO from ITDM via e-mail.
XRO:
- FXE jitter last week: not yet clear if it was the clean tent or the cooling water for M1/M2 the real cause; hard to find as cooling water was refilled and the clean tent was switched ON accidently, both
in parallel.
LAS:
- Issues with Sync system in SA1 persists: spurious 10Hz signal. PP-laser still do not have optical signal. LAS has a rough idea of the cause and is in contact with instruments.
UNSYS:
- Phase shifter in cell11/SA3 the motor was replaced and more 10 motors were ordered as spare parts.
- Quad mover in SA1#cell 18 was fixed.
- Quad mover in cell25/SA3 (wrong movement direction) is still to be addressed. MXL will work together with UNSYS.
- Tables for Phase shifters of Apple-X still to be uploaded.
- One new PS on the Apple-X is not working; Frank Brinker is informed and a ZZ is needed before the next blue week.
SPB:
- Influx issue but could use Graphana.
- Pulse energy is low this week: 2mJ while last week was 3.5mJ;
- Crosstalk with SA2: SA1 energy gets worse when changes happen in SA2 but recovers shortly after.
- SPB did not see any vibration on Tuesday 15.03. Also last week beam was stable.
FXE:
- Nothing to add.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- This morning a filter of RIXS spectrometer broken and has to be exchanged (filter gauge valve). VAC group is helping with the exchange.
SQS:
- Nothing to report. |
310
|
21 Mar 2022, 10:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 21.03.2022 |
General Announcement:
- A. Silenzi will be the DRC this week and reminds FXE, SQS and MID to be present at the jour-fixe meeting today at 13:00.
CTRL:
- Ongoing upgrade of Karabo framework on selected services to address issues with connection output signals: CTRL would like to do SA3 tunnel and SQS today (SCS already done); others next Monday.
- Alberto deFanis says that colleagues from SQS prefer to do in a week without users. Steffen H. says that this is fine to do later. The tunnel devices have to be coordinated with SCS also.
SQS will coordinate to a proposed date and inform CTRL.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks in between user experiments.
VAC:
- Issue with ion pump in rack 3.1 taking down SA1 and SA3 beam: ZZ by VAC/OCD and PRC to exchange controller yesterday. Another ZZ will take place this morning.
Downtime for the entire North Branch could be minimized (if V0 is closed, then at least SA1 could be running. Martin D. will look into optimizing the procedure).
XPD:
- Naresh K. as PRC says on the pump failure on Sunday. Downtime was longer for SA1 as machine could not decouple both SA1 and SA3.
- Naresh K. says the access the access ZZ is not so quick as it was done in dump area and machine has to ground magnets and power up again after the ZZ, making it take longer time (not only ZZ time).
- SCS turbo pump fail and the experiment stopped on Sunday around 15:30.
DET:
- Nothing to report.
ITDM:
- Following last Tuesday with InfluxDB break: since Tuesday things are going smoothly and looks optimistic after applying the fix. Nevertheless, more monitoring time is needed to certify it.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing major to report apart of the downtime yesterday.
- Some beam drift as usual was seen but not out of ordinary.
FXE:
- Problem with XGMD happened again and Theo M. is looking for spare parts.
MID:
- All fine at MID.
HED:
- Finished first of many experiments using RELAX laser. A minor problem is addressed by a ticket.
SCS:
- Turbo pump failure: have to be investigated and repaired again the whole vacuum until next week. Sunday at 15:00 it failed.
- SCS cameras and uTCA update is already done (see CTRL entry).
- XGM / HV inhibit problem is recurring once every week. SCS would like to be trained in how to recover it.
SQS:
- Nothing to add. |
311
|
23 Mar 2022, 10:04 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 23.03.2022 |
General Announcement and A.O.B.:
- Sandra Plett announces that a fire drill will take place next Saturday 26.03 involving many areas at Schenefeld. It starts at 10:00AM and around 6 persons will conduct the drill.
Please understand that this is only a training. An e-mail will be sent later.
- Theo M. says there is bridge problems in XTD2 XGM into the Doocs and Karabo side. Some trains are still missing. Other XGMs are fine. Stefen H. fears the solution lies more in the DOOCS side.
CTRL:
- The process to deploy Karabo 2.13.3 to mitigate the pipeline connection system is ongoing. Instruments will be managed in individual scheduling. Tunnels areas is proposed to proceed next Monday 28.03.
An e-mail will be sent.
DA:
- Nothing to report.
EEE:
- Fast electronics has a small intervention at HED to change a crate issue. HED is informed.
VAC:
- Nothing to report.
- A. Silenzi says he was informed by Martin D. that SA2 topics in e.g. InfluxDB or Graphana had an outage yesterday afternoon. Right now it is back to work but missed some data yesterday.
XPD:
- XGMD in FXE: the ADC was replaced and is again online.
DET:
- Nothing to report.
ITDM:
- Influx DB was down yesterday.
- Power glitch on Monday evening: cooling for 604 room at the XHQ ground floor was lost. A. Galler says XO will set a meeting with ITDM and TS to set an alarm when cooling fails.
XRO:
- Nothing to report.
LAS:
- Sandhya Venkatesan will join the XO. Int. meeting also.
- AMPHOS in SA1 and SA2 went down due to the lack of cooling water as consequence of Monday's power outage.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- XGMD is running again.
- Dmitry K. thanks Mattias Schoz for fixing an issue with beam. This is fixed. Operators loaded wrong file this morning and re-alignment had to be done.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Continuing with the clean of the chemistry chamber after an accident.
SQS:
- Nothing to report. |
312
|
28 Mar 2022, 10:09 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 28.03.2022 |
General Announcement and A.O.B.:
- BTM @ FXE was triggered on Saturday afternoon and only PRC was available to come onsite and fix the issue. Different procedures will be proposed to be put in place.
- SRI conference starts this week.
CTRL:
- Issue with data loggers last week: narrowed down to an issue with expected update of Telegraph. This time it caused a trigger in the broker pileup. Solved with help from ITDM.
- Upgrade in tunnels today (already agreed; see last meeting minutes).
- Issue with MID/SPB AGIPD middle layer: requires an upgrade of framework. Currently there is no agreed time with instruments but proposal is to make it on a Monday.
DA:
- Nothing to report.
EEE:
- Busy with FXE pump commissioning.
VAC:
- Nothing to report.
- Martin Dommach, as PRC, reports that energy change in SA1 was not permitted during the weekend (costing many lost shifts in SPB) due to crosstalk with SA3;
on Saturday an event with BTM monitor in XTD9/FXE shutter happened and brought down machine. SA2 and SA3 was restored after 1 hour and SA1 after 3,5 hours.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Problem with shutter in SQS instrument laser hutch during the weekend: it could continue apart of the error state. Under investigation.
UNSYS:
- Next Monday a ZZ is requested to 1. get phase shifter SA3/cell26 working; 2. Investigate cell 25/SA3 Quad mover (horizontal movement in wrong direction).
SPB:
- Lost shifts due to crosstalk (2 night-shifts on weekend). Apart of that other shifts went well.
FXE:
- Incident on Thursday (LPD motor not working): after restarting it lost motor positions. Stefen H. says that A. Silenzi is preparing a report and updates will follow.
MID:
- Quite good week with small hiccups in data acquisition.
HED:
- HED asks when EEE plans to update uTCA; EEE advices to please contact Bruno F.
SCS:
- Issue with replaced turbo pump: new one enters in error mode after going to 400Hz. SCS is in contact with VAC; to be fixed today.
SQS:
- Nothing to report. |
313
|
30 Mar 2022, 09:51 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 30.03.2022 |
General Announcement:
- A. Galler mentions that new Corona rules are in place. Please read the e-mail from Nicole E. from last Friday. In case of questions don’t hesitate in contacting SRP and/or taskforce.
CTRL:
- LPD motor issue under investigation. Report to follow later.
DA:
- Nothing to report.
EEE:
- Update on HED OPT uTCA crate: one last action was done and hopes to get it solved. Under monitoring to see if another intervention is still needed.
VAC:
- Nothing to report.
XPD:
- Issue after Karabo update: SA1 FEL Imager is not available for some time. Solved by CTRL/A. Parenti. For the BKR it took a little bit longer to get it ready. It may happen again.
DET:
- Nothing to report.
ITDM:
- Question if it is possible for next week to update the GPFS client to fix the recent bugs with permissions (Maxwell cluster calibration pipeline - solved at the time with a hot fix).
The complete version is ready to be deployed. One needs to update and restart service (matter of 1-2 hours) and the current proposal is to make it on Monday. An e-mail will be sent to XO-info.
XRO:
- Middle layer device of SRA in SA2 had the offset parameters placed to zero and as consequence it showed unreal numbers. M. Vannoni placed back old parameters and right now it shows the correct numbers.
It was positioned in the correct place but numbers were wrong. Wajid E. says parameters can be retrieved or please write a ticket to CTRL if it happens again.
- As PRC, machine had isolated problems yesterday night and this morning.
LAS:
- Synchronization for the PP-lasers went down. LbSync is informed.
- PP-laser in SA3 went fully down and SA2 is showing instabilities. LAS is working on them.
UNSYS:
- ZZ is planned for next Monday.
SPB:
- Problem with XTD2 XGM not solved and SPB/SFX is losing data. This seems to be a Doocs issue and needs to be addressed.
- Fast ADC issues under discussion with EEE.
- Machine performance is fine.
FXE:
- Nothing to report.
MID:
- Nothing to report.
HED:
- Nothing to report.
SCS:
- The turbo pumps problems were solved; Not clear if current activity is affected due to the issues with PP-laser. It will be asked to the instrument people.
SQS:
- Experiments from last week went well accordingly to the internal group meeting. |
318
|
20 Apr 2022, 10:07 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 20.04.2022 |
General Announcement:
- Steffen H. reminds on the Karabo developer training that will take place next week. Registrations are still possible.
- Martin Doering informs that next week Friday 29.04 is the deadline for submitting the summer maintenance lists. In case of needed help, please contact xo-planning@xfel.eu.
CTRL:
- Issue with Influx issue in cw14: data logging is now protected and a Karabo update in only the data loggers machines has to be done.
It is proposed to make it today at 13:00 (it is transparent for the users and takes about 30mins). An e-mail will be sent to the xo-info@xfel.eu.
- Beckoff updates in software level to be performed on next Monday 25.04 at 13:00. This affects mainly tunnels; with instruments the coordination will be done.
DA:
- Nothing to report.
EEE:
- Minor things being supported but nothing major to report.
VAC:
- Nothing to report.
XPD:
- A. Koch asks on the Beckhoff update: only hardware in tunnels will be updated and only Karabo software.
- J. Laksman says that during the Eastern beamtime at SQS a gas needle opened very fast and the pressure of chamber went very fast, posing a damage risk.
A Redmine ticket was placed for that it does not open too fast. An offline discussion will take place with EEE and CTRL.
DET:
- DSSC is back online at SCS after months of non-operation. There are issues to be fixed and investigations are in progress. The DSSC expert is on OCD this week and DOC can also help.
ITDM:
- Nothing to report.
- Marcin S. asks on the plan for limiting the raw data process per run in this current beamtime. SPB staff was told that they can't have more than 10 processes running;
K. Wrona informed this was talked to users and it was agreed with the limited number of 10. An offline discussion will take place and if more is needed this can be accommodated.
XRO:
- Nothing to report.
LAS:
- Not present.
UNSYS:
- Nothing to report.
SPB:
- See Marcin S. question above. Nothing else.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- Under operation and issue with DSSC was already mentioned.
SQS:
- Not present. |
319
|
25 Apr 2022, 10:09 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 25.04.2022 |
General Announcement:
- Regarding the use of special diagnostics devices: on THU XPD received a call for supporting the use of HIREX in the tunnels. XO would like to ask instruments to inform XPD or XO well in advance for such
requests, if possible.
Please include that in the "Comments" column of the X-ray delivery tables.
- Deadline for submitting the 2022 summer maintenance period is next Friday 29.04.
- A. Silenzi reminds instruments to join the Jour-Fixe meeting with DRC today at 13:00.
- Thomas Michelat reminds that during the night the instruments shall call the regular DOC number and not directly e.g. the CTRL OCD number.
- A. Galler inform the operation affair on the FXE mono is to be prepared in the next week or so, following the updated process for Operation Affairs.
CTRL:
- CTRL reminds that today at 13:00 there is an update of Karabo device in tunnels (for Beckhoff systems).
- Also at 13:00 an update of Aravis cameras in tunnels will take place.
DA:
- Nothing to report.
EEE:
- EEE is continuing the work with PnCCD issues.
- A ZZ in XTD6 will be done today for HED Mono.
- Investigating a motor in a CRL that showed issues.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Marcin S. says that one of the AGIPD central tiles was lost last week after a sudden beam jump. This will be discussed/investigated by DET / Olivier M.
ITDM:
- Sync&Share service is not accessible tomorrow 26.04.
- Today at 10AM the Redmine service will be updated.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Issue on Friday with SA2/cell33 not moving the gap. A remote reset solved the problem.
- Last night when moving SA3/cell21: it did not move and the cause was a strong applied taper. Now it is solved.
SPB:
- Experiments went well.
- AGIPD was affected due to beam movement.
FXE:
- Commissioned the Johann spectrometer last week and this week the plan is commissioning the Gotthard spectrometer.
MID:
- All fine at MID.
HED:
- Different settings for M1 and M2: changed positions have to be coordinated with MID, XPD and XRO when beam will be aligned again to HED. It caused problems for the HIREX alignment.
- A meeting/communication among these groups have to be organized. Angel R.-F. says that today and tomorrow morning work on the mirrors will work. XRO will lead and new positions to be communicated to HED.
SCS:
- Problems with cell21 last night.
- MDL devices for multi-pattern showed issues and will be discussed among SCS and CTRL.
SQS:
- Issue with two digitizers in the same uTCA board that can not be used in interleaved. Current understanding is that one has to ship device back to the manufacturer. |
321
|
02 May 2022, 07:27 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 02.05.2022 |
General Announcement:
- Martin Doering gives update on the maintenance planning progress: a Dashboard will be introduced and updated every week and almost all groups have submitted the lists.
CTRL:
- Reminder: today at 13:00 there is a deployment of Karabo Beckhoff in the tunnels (motors with newly features). No issues are expected; otherwise a reset may suffice.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- No incidents to report;
- Two ZZs in tunnels are taking place this morning.
DET:
- Feedback regarding AGIPD at SPB (cw16): one module has to be replaced during the summer maintenance period. Reason behind the damage has to be investigated together with SPB staff.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- ZZ taking place at SA3 cell 14 this morning.
SPB:
- Everything went smooth last week.
FXE:
- Couple of issues with Bassler cameras not connecting or acquiring data (DOC helped).
- Gun trips on the weekend means that one hour is needed to wait until running again Mono (using high heat load). Many gun trips were seen during the weekend.
Raul V. says that a possibility was discussed yesterday at around 09:45AM for an intervention to minimize the gun trips but after consultation with FXE staff it was decided to run in that condition. Peter
Z. says that calling 9-4800 was not reachable.
MID:
- Everything looks good for the last experiment with HXRSS.
- It lost the timing after a big trip on Sunday.
- Frequent broker Pile-ups happened and for mid-term has to be addressed (J. Hallmann). Gabrielle G. and Setffen H. say that a fix is known and in the mid-term it will be solved.
- Problem with multipatern of Laser happened; reported by Angel R.-F. LAS will look into it.
HED:
- CRL3 was tested and enabled it back.
- Tomorrow the mirror vertical motors will be checked together with XRO.
SCS:
- Nothing to report. Only smaller issues that were addressed.
SQS:
- Nothing to report. |