ID |
Date |
Author |
Location |
Severity |
Subject |
326
|
18 May 2022, 11:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 18.05.2022 |
General Announcement:
- The cold compressor of the accelerator failed yesterday night at around 23:15. The current expectation is that beam shall be back on Thursday morning.
Instruments and DRC were informed by the PRC via phone. A status entry can be found in the TTfinfo e-log, Operation-2022 e-log and XO Alfresco landing page.
- Update on the summer maintenance planning: Some groups have the review process finalized. Scheduling meetings are ongoing.
If a group thinks it is needed to have an scheduling meeting please feel free to contact Martin Doering.
CTRL:
- Nothing to report.
DA:
- Nothing to report.
EEE:
- Nothing to report.
- SA3 PP-L parts that were not behaving properly are under investigation with LAS.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Not present.
UNSYS:
- High doses observed in Apple-X are under investigation; A plan to move undulators in the last night did not take place.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- Yesterday a loop update cost time with loosing motor parameters but solved at the end.
- Broker pileup: root cause was the physical location of the board (configuration) was changed in between times. Location was changed when configuration was requested.
HED:
- Nothing to report.
SCS:
- Fast sample scanner has problems and needs to be fixed. Involved groups are informed.
SQS:
- Nothing to report. |
325
|
16 May 2022, 09:53 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 16.05.2022 |
General Announcement:
- Another cryo failure happened yesterday night due to an overpressure. Expected cryo OK shall be around 10AM. Plan to tune today holds.
- Steffen H. reminds that instruments shall participate in the Jour-fixe today at 13:00.
CTRL:
- Nothing to report.
DA:
- Announcement from the calibration team: offline calibration will have a deploy today – minor intervention – to correct e.g. bug fixes
EEE:
- SA3 PPL has a card that behaves not normally. Needs an access and is been repaired right now.
- Pouhneh S. says that Saturday reason maybe be related to laser interlock. The whole SA3 went down. Lasermet was onsite and a wire was shorted circuit. Needs to look closely if it was interlock .
It acted as if the emergency OFF was pressed. Zunaira A. says that this happened also in SA2 time ago: AMPHOS shutter has a sub-D connector and when this connects the fuse blows and power went down; LAS
and Klaus Witt solved at the time and reason shall be related to the AMPHOS shutter.
K. Wrona says as DRC he has got a call informing the power glitch. This morning the network switches in the room are still not functioning. Please call DRC using DOC number when power goes down.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
- On the power glitch of SA3: only SQS and SCS hutch devices were affected by the power glitch due to the short circuit on the interlock/PPL.
- Switch to the 18keV in SA2 went smooth on Saturday.
DET:
- AGIPD in SPB showed radiation damage. On May 7th the detector was exposed to high radiation and investigation still ongoing.
About 10% of pixels are damaged and need full characterization to find if still usable for users.
ITDM:
- Nothing to report.
XRO:
- Last week on Friday: moved Mirrors M1 and M2 of SA2. Mirror positions seem to be in the right positions.
When beam is handled over to MID the XRO group has to check alignment and it is already agreed with MID. Currently beam is in the Platinum stripe.
LAS:
- Power outage in SA3: seeder is back but interlock does not allow seeder shutter open (card to be changed). Many troubles with devices in the network.
UNSYS:
- ZZ for SA2/cell#2 to fix a problem.
SPB:
- Interesting program last week with stability investigation and beam usage on the weekend.
FXE:
- Got support from Data calibration: thanks!
MID:
- All fine at MID.
- Ulrike B. comments on the Karabo broker pile up over Easter: noticed now that there is a mismatch between what is written and triggered.
Wajid E. says that will contact offline to MID to further understand. It is corrected as of now. Bruno F. says that there is a ticket ongoing (wrong timer connection affecting the configuration location).
HED:
- Nothing to report.
- Long pulse train tests were successful and shall be used on cw22 - a meeting takes place with BKR today.
SCS:
- No full power outage in SCS hutch (beamline was checked and seem to be fine). XGM did not get interrupted.
SQS:
- Suffer from laser interlock during the weekend – could not take PP laser. Got a damaged scroll pump. |
324
|
11 May 2022, 10:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 11.05.2022 |
General Announcement:
- The accelerator is currently down. An electrical power issue lead to a cryo failure at 08:45AM and both cold and warm compressors went down.
The expected cryo OK shall come in the afternoon and by 15:30 we will have further information.
CTRL:
- Nothing to report.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Incident in MID branch/SA2 last evening with the V0 valve being closed: VAC received automatic message with spike around the CRL2 was detected, then closing the V0 valve. Solved in 5 mins.
Please report to VAC group (08:42PM) when such incidents happen. Reason behind this even is still unclear.
XPD:
- Nothing to add.
DET:
- Nothing to report.
ITDM:
- A Change in configuration of run controllers for the DAQ may profit today as machine is down. ITDM will discuss with groups and inform accordingly.
To be done in tunnels, laser, 5 instruments apart of SPB/SFX. One has to reload projects afterwards.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Apple-X: massive losses of encoders were found and the FD program is cancelled. A ZZ will be done today.
- Heidenheim linear encoders need to be replaced, quote is placed but not sure if it happens in summer maintenance.
SPB:
- Reconfiguring the instrument for the weekend.
FXE:
- Problem with XGMd: under work by Theo and Finni.
MID:
- It has now the PicoAmpmeter; in touch with EEE and CTRL.
HED:
- Monday long pulse train test: seems to be fine; also beam pointing stability went fine for HED.
- XRO was working with mirrors: Mikako M. says that Maurizio M. will address with HED the tests of vertical positions.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
323
|
09 May 2022, 10:30 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 09.05.2022 |
General Announcement:
- This week is a Facility development week: photon studies take place until Wednesday and then setup for 16.3GeV for the next weeks of operation.
- Martin Doering present a Dashboard for the Summer maintenance period. About 600 activities were submitted timely by the groups; currently the lists are being processed for checking for major error.
- Jan G. adds as PRC that there were no major issues during the weekend. The FXE XGMd gave wrong values and is under investigation.
CTRL:
- Few devices are saved in the projects with wrong classID: A script is to be applied today at 13:00 to fix this configuration; it is transparent.
- Update of data logger to be done today at 13:00 (15min downtime) also to improve Influx.
DA:
- No announcements; past OCD was quiet.
EEE:
- This week: intervention to be done in SA3 to allow better slits movement (together with XRO).
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing major to report.
ITDM:
- Nothing to report.
XRO:
- XRO will be busy next days with beamtime / FDs activities.
LAS:
- A water leak is found in SA1 laser hutch. TS is contacted and the suspicion lies in the humidifier from the roof. There is no estimative on time for repair.
- Maintenance works in SA3 PP-laser to be done until Wednesday.
UNSYS:
- Control system of Apple-X shows several encoders giving errors in several motors. Encoders were failing to start.
SPB:
- Experiments went well last week.
FXE:
- FXE has planned for PP-laser use.
- Martin K. asks if a ZZ for the Mono is happening today. A. Galler inform hat today there is ZZ and therefore the access request is not yet scheduled.
MID:
- MID colleagues still look for a picoampere meter.
HED:
- The Full train test is scheduled to take place today.
SCS:
- Not present.
SQS:
- Nothing to report. |
322
|
04 May 2022, 09:59 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 04.05.2022 |
General Announcement:
- Issues with machine on Monday (misaligned photocathode laser). Beam delivery started only Tuesday at 15:00 instead of 07:00AM. Cause is understood and mitigation strategy is put in place by MXL.
CTRL:
- Nothing to report.
- Monday's Karabo deployment went well.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Part of PES detectors were repaired on Monday's ZZ (14 out of 16 are now working).
DET:
- Nothing to report.
ITDM:
- SPB metro device causing Telegraph to stop sending data. Solved by DOC and a fix will be deployed today together with DA.
XRO:
- Not present.
LAS:
- Nothing to report.
UNSYS:
- Phase shifter cell 14 / SA3 is now fixed.
SPB:
- Few minor issues were solved.
- Gain were not closing - DOC is informed;
K. Wrona says some runs do not close (1-2 runs per shift): root cause of not understand but solution exist (close the run manually at MyMdc and restart; DOC can help with that).
FXE:
- Thank to SQS for helping with gas valve.
- Continuing with commissioning activities.
- CRLs were in the beam last Monday and were retracted.
MID:
- All fine at MID.
- Looking for a Keithley Pico-Ampere meter to borrow for cw20.
HED:
- Nothing to report.
- A. Galler informs a water pipe is blocking the construction of the pulsed room. It affects the cooling water. TS needs to coordinate with involved groups.
SCS:
- Experiment ongoing.
SQS:
- Nothing to report. |
321
|
02 May 2022, 07:27 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 02.05.2022 |
General Announcement:
- Martin Doering gives update on the maintenance planning progress: a Dashboard will be introduced and updated every week and almost all groups have submitted the lists.
CTRL:
- Reminder: today at 13:00 there is a deployment of Karabo Beckhoff in the tunnels (motors with newly features). No issues are expected; otherwise a reset may suffice.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- No incidents to report;
- Two ZZs in tunnels are taking place this morning.
DET:
- Feedback regarding AGIPD at SPB (cw16): one module has to be replaced during the summer maintenance period. Reason behind the damage has to be investigated together with SPB staff.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- ZZ taking place at SA3 cell 14 this morning.
SPB:
- Everything went smooth last week.
FXE:
- Couple of issues with Bassler cameras not connecting or acquiring data (DOC helped).
- Gun trips on the weekend means that one hour is needed to wait until running again Mono (using high heat load). Many gun trips were seen during the weekend.
Raul V. says that a possibility was discussed yesterday at around 09:45AM for an intervention to minimize the gun trips but after consultation with FXE staff it was decided to run in that condition. Peter
Z. says that calling 9-4800 was not reachable.
MID:
- Everything looks good for the last experiment with HXRSS.
- It lost the timing after a big trip on Sunday.
- Frequent broker Pile-ups happened and for mid-term has to be addressed (J. Hallmann). Gabrielle G. and Setffen H. say that a fix is known and in the mid-term it will be solved.
- Problem with multipatern of Laser happened; reported by Angel R.-F. LAS will look into it.
HED:
- CRL3 was tested and enabled it back.
- Tomorrow the mirror vertical motors will be checked together with XRO.
SCS:
- Nothing to report. Only smaller issues that were addressed.
SQS:
- Nothing to report. |
320
|
27 Apr 2022, 09:04 |
Andreas Galler | General | Information | Notes XO Integration Meeting 27.04.2022 |
General Announcement:
CTRL:
- bug noticed in BEckhoof karabo device --> already fixed from some topics, see xo-info comm from yesterday
DA:
EEE:
VAC:
- two devices went into error --> likely because of Beckhoff Karabo bug
XPD:
DET:
- AGIPD 1M at SPB partially vented yesterday --> detector went into interlock, no damage
- who should be called at SPB (or any instrument for that matter)
- Turbo pumps went into error state which triggered the partial vent --> karabo Beckhoff update?
- Wajid comments: karabo beckhoff update does not correlate with this incident, judging from time stamps
ITDM:
XRO:
LAS:
UNSYS:
SPB/SFX:
FXE:
- software limits of phase shifter of PP Laser
MID:
- all fine at MID
- HXRSS very much appreciated
HED:
SCS:
SQS:
DRC
PRC
- SCS incident on Monday, photon energy scanning script was left running --> had to be stopped manually
|
319
|
25 Apr 2022, 10:09 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 25.04.2022 |
General Announcement:
- Regarding the use of special diagnostics devices: on THU XPD received a call for supporting the use of HIREX in the tunnels. XO would like to ask instruments to inform XPD or XO well in advance for such
requests, if possible.
Please include that in the "Comments" column of the X-ray delivery tables.
- Deadline for submitting the 2022 summer maintenance period is next Friday 29.04.
- A. Silenzi reminds instruments to join the Jour-Fixe meeting with DRC today at 13:00.
- Thomas Michelat reminds that during the night the instruments shall call the regular DOC number and not directly e.g. the CTRL OCD number.
- A. Galler inform the operation affair on the FXE mono is to be prepared in the next week or so, following the updated process for Operation Affairs.
CTRL:
- CTRL reminds that today at 13:00 there is an update of Karabo device in tunnels (for Beckhoff systems).
- Also at 13:00 an update of Aravis cameras in tunnels will take place.
DA:
- Nothing to report.
EEE:
- EEE is continuing the work with PnCCD issues.
- A ZZ in XTD6 will be done today for HED Mono.
- Investigating a motor in a CRL that showed issues.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Marcin S. says that one of the AGIPD central tiles was lost last week after a sudden beam jump. This will be discussed/investigated by DET / Olivier M.
ITDM:
- Sync&Share service is not accessible tomorrow 26.04.
- Today at 10AM the Redmine service will be updated.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Issue on Friday with SA2/cell33 not moving the gap. A remote reset solved the problem.
- Last night when moving SA3/cell21: it did not move and the cause was a strong applied taper. Now it is solved.
SPB:
- Experiments went well.
- AGIPD was affected due to beam movement.
FXE:
- Commissioned the Johann spectrometer last week and this week the plan is commissioning the Gotthard spectrometer.
MID:
- All fine at MID.
HED:
- Different settings for M1 and M2: changed positions have to be coordinated with MID, XPD and XRO when beam will be aligned again to HED. It caused problems for the HIREX alignment.
- A meeting/communication among these groups have to be organized. Angel R.-F. says that today and tomorrow morning work on the mirrors will work. XRO will lead and new positions to be communicated to HED.
SCS:
- Problems with cell21 last night.
- MDL devices for multi-pattern showed issues and will be discussed among SCS and CTRL.
SQS:
- Issue with two digitizers in the same uTCA board that can not be used in interleaved. Current understanding is that one has to ship device back to the manufacturer. |
318
|
20 Apr 2022, 10:07 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 20.04.2022 |
General Announcement:
- Steffen H. reminds on the Karabo developer training that will take place next week. Registrations are still possible.
- Martin Doering informs that next week Friday 29.04 is the deadline for submitting the summer maintenance lists. In case of needed help, please contact xo-planning@xfel.eu.
CTRL:
- Issue with Influx issue in cw14: data logging is now protected and a Karabo update in only the data loggers machines has to be done.
It is proposed to make it today at 13:00 (it is transparent for the users and takes about 30mins). An e-mail will be sent to the xo-info@xfel.eu.
- Beckoff updates in software level to be performed on next Monday 25.04 at 13:00. This affects mainly tunnels; with instruments the coordination will be done.
DA:
- Nothing to report.
EEE:
- Minor things being supported but nothing major to report.
VAC:
- Nothing to report.
XPD:
- A. Koch asks on the Beckhoff update: only hardware in tunnels will be updated and only Karabo software.
- J. Laksman says that during the Eastern beamtime at SQS a gas needle opened very fast and the pressure of chamber went very fast, posing a damage risk.
A Redmine ticket was placed for that it does not open too fast. An offline discussion will take place with EEE and CTRL.
DET:
- DSSC is back online at SCS after months of non-operation. There are issues to be fixed and investigations are in progress. The DSSC expert is on OCD this week and DOC can also help.
ITDM:
- Nothing to report.
- Marcin S. asks on the plan for limiting the raw data process per run in this current beamtime. SPB staff was told that they can't have more than 10 processes running;
K. Wrona informed this was talked to users and it was agreed with the limited number of 10. An offline discussion will take place and if more is needed this can be accommodated.
XRO:
- Nothing to report.
LAS:
- Not present.
UNSYS:
- Nothing to report.
SPB:
- See Marcin S. question above. Nothing else.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- Under operation and issue with DSSC was already mentioned.
SQS:
- Not present. |
317
|
13 Apr 2022, 09:28 |
Andreas Galler | General | Information | Notes XO Integration Meeting 13.04.2022 |
General Announcement:
- No User operation meeting this Friday, instead cw 16 / cw 17 check will happen in Daily Operations Meeting at 15:30 on Thu Apr 14. Link available on ttfinfo in XFELelog
CTRL:
DA:
EEE:
VAC:
XPD:
DET:
- offline calib with JUNGFRAU --> tickets closed already
ITDM:
XRO:
- request to SA1 instruments
- please check the repaired XTD2 solid attenuators
- attenuators cannot sustain too much power, offline discussion follows
- SA2 mirror recalib
- roll of M1 and M2 was completely misaligned, recalibration successful
- geometrical center of the kinematic system roughly identified
LAS:
UNSYS:
SPB/SFX:
FXE:
MID:
- pump probe looks stable and getting ready for the user
- BAM doocsbridge stopped --> manual restart necessary
HED:
SCS:
SQS:
DRC
- one hour data lost in influxDB in SA2
PRC
|
316
|
11 Apr 2022, 09:10 |
Andreas Galler | General | Information | Notes XO integration meeting 11.04.2022 |
General Announcement:
CTRL:
- small intervention --> udpate of karabo-gui client exflagteway
DA:
- maxwell gpfs software maintenance --> offline calibration pipeline affected see ITDM
EEE:
- OCD call on rack 2.01 which was powered down. After fuse repaired
VAC:
XPD:
- Rack 2.01 --> Camera of SR imagare is not working
- situation not clear, being investigated --> IP conflict?
- Janusz suggests to powercycle the camera
DET:
ITDM:
- 10:00 maxfell gpfs update
XRO:
LAS:
- remainder change of set point for SCS to 112 kHz
UNSYS:
SPB/SFX:
FXE:
- preparing for user experiments
- concerned about optical sync of PP Laser, probelm of last week was fixed but no reason found
- test last thursday optical sync lost again while moving optical delay stage
- FXE
MID:
- alignment of optical laser alignment in joined hutch.
HED:
SCS:
SQS:
- last week Thursday night
- mono drift investigated --> preliminary analyisis points to the grating as the drifting component
|
315
|
06 Apr 2022, 09:15 |
Andreas Galler | General | Information | Notes XO integration meeting 06.04.2022 |
General Announcement:
CTRL:
- influxDB outage (telegraf issue)
- SA1 and SA3 affected
- currently fixed but linked to a device in SQS
DA:
EEE:
- PLC colleagues are working on CR and support requests
- pulling cables for SQS and SCS
- reported to XO uTCA in XTD1
- remote reboot planned for Friday, reserve ZZ slot for Friday
VAC:
- repair of XTD2 solid attenuator ongoing
XPD:
DET:
ITDM:
- nothing to report
- next Monday GPFS update will take place
XRO:
- mirror calibration started in SA2
- beam path through offset mirros not visible -> no beam on M3 pop in, but visible on M2 pop in MCP in beam?
- JG: no MCP in beam, how about PBLM?
- LS: no PBLM in beam
-
LAS:
- Sync issue from Monday cannot be replicated by Sebastian -->
- tests for SPB happening tomorrow
UNSYS:
- circular polratizaton lasing achievd in APPLE-X in left and right hand polarization, also linear horizontal and vertical and 45° measured at 900 and 700 eV
- thanks to UNSYS and DESY APPLE-X comm team.
SPB/SFX:
FXE:
MID:
HED:
SCS:
- reservation of nodes for jupyter hub.
- requested during drc hour fix to reserve 3 nodes
- is it possible to reserve 3 nodes by default?
- Olivier reports there was a misunderstanding and it got fixed very quickly
- Janusz reports that by standard 4 nodes are reserved for each experiment.
SQS:
|
314
|
04 Apr 2022, 09:26 |
Andreas Galler | General | Information | Notes XO integration meeting 04.04.2022 |
General Announcement:
CTRL:
- PP LAS restarting device servers for upgrading the device tag --> autoinstantiate some servers all topics
DA:
- Planned maintenance today for Cal and gpfs delayed at least until next Monday
EEE:
VAC:
- 1 ZZ ongoing to XTD4 to shiled an ion pump controller, suspicion of radiation damage.
XPD:
- similar Diamond cracks as in solid attenuator had been observed in the transmissive imagers
- Last weeek FEL imager issue had been clarified (image was rotated by 180 °, occured due to faultly config load)
DET:
- AGIPD1M (SPB/SFX), Friday morning some DAQ issues induced train losses. Didn't happen again after careful checking
ITDM:
- Issue at HED with data migration --> discussion will take place
XRO:
- Tomorrow and after tomorrow late shift some CTRL support might be required
LAS:
- PLL LAS SA1 sync issues (LbSYNC informed and arriving 10:00)
- PPL LAS3 NOPA setup
UNSYS:
SPB/SFX:
- Experiment went well, no major issue
- Run Controller crashd a few times over the weekend. Could be restarted but crashes some time later and has to be restarted
FXE:
- affecetd by LAS Sync issue
- issue with doocs panel for shutters (yifeng)
MID:
- all fine
- SSRY motor issue (ELMO)
- Nerea updated the ticket --> motor is working
HED:
- DA1 got stuck for a while (fixed by restarting)
- lot of short runs (1min) and only one train isneed from that -> requires fast data migration and analysis -> doesn't always work
- discussion with ITDM commencing
- PAM was missing from DAQ
SCS:
- no major issues
- RIXS on liquids went well
SQS:
DOC /DRC
issues with call forwarding |
313
|
30 Mar 2022, 09:51 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 30.03.2022 |
General Announcement:
- A. Galler mentions that new Corona rules are in place. Please read the e-mail from Nicole E. from last Friday. In case of questions don’t hesitate in contacting SRP and/or taskforce.
CTRL:
- LPD motor issue under investigation. Report to follow later.
DA:
- Nothing to report.
EEE:
- Update on HED OPT uTCA crate: one last action was done and hopes to get it solved. Under monitoring to see if another intervention is still needed.
VAC:
- Nothing to report.
XPD:
- Issue after Karabo update: SA1 FEL Imager is not available for some time. Solved by CTRL/A. Parenti. For the BKR it took a little bit longer to get it ready. It may happen again.
DET:
- Nothing to report.
ITDM:
- Question if it is possible for next week to update the GPFS client to fix the recent bugs with permissions (Maxwell cluster calibration pipeline - solved at the time with a hot fix).
The complete version is ready to be deployed. One needs to update and restart service (matter of 1-2 hours) and the current proposal is to make it on Monday. An e-mail will be sent to XO-info.
XRO:
- Middle layer device of SRA in SA2 had the offset parameters placed to zero and as consequence it showed unreal numbers. M. Vannoni placed back old parameters and right now it shows the correct numbers.
It was positioned in the correct place but numbers were wrong. Wajid E. says parameters can be retrieved or please write a ticket to CTRL if it happens again.
- As PRC, machine had isolated problems yesterday night and this morning.
LAS:
- Synchronization for the PP-lasers went down. LbSync is informed.
- PP-laser in SA3 went fully down and SA2 is showing instabilities. LAS is working on them.
UNSYS:
- ZZ is planned for next Monday.
SPB:
- Problem with XTD2 XGM not solved and SPB/SFX is losing data. This seems to be a Doocs issue and needs to be addressed.
- Fast ADC issues under discussion with EEE.
- Machine performance is fine.
FXE:
- Nothing to report.
MID:
- Nothing to report.
HED:
- Nothing to report.
SCS:
- The turbo pumps problems were solved; Not clear if current activity is affected due to the issues with PP-laser. It will be asked to the instrument people.
SQS:
- Experiments from last week went well accordingly to the internal group meeting. |
312
|
28 Mar 2022, 10:09 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 28.03.2022 |
General Announcement and A.O.B.:
- BTM @ FXE was triggered on Saturday afternoon and only PRC was available to come onsite and fix the issue. Different procedures will be proposed to be put in place.
- SRI conference starts this week.
CTRL:
- Issue with data loggers last week: narrowed down to an issue with expected update of Telegraph. This time it caused a trigger in the broker pileup. Solved with help from ITDM.
- Upgrade in tunnels today (already agreed; see last meeting minutes).
- Issue with MID/SPB AGIPD middle layer: requires an upgrade of framework. Currently there is no agreed time with instruments but proposal is to make it on a Monday.
DA:
- Nothing to report.
EEE:
- Busy with FXE pump commissioning.
VAC:
- Nothing to report.
- Martin Dommach, as PRC, reports that energy change in SA1 was not permitted during the weekend (costing many lost shifts in SPB) due to crosstalk with SA3;
on Saturday an event with BTM monitor in XTD9/FXE shutter happened and brought down machine. SA2 and SA3 was restored after 1 hour and SA1 after 3,5 hours.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Problem with shutter in SQS instrument laser hutch during the weekend: it could continue apart of the error state. Under investigation.
UNSYS:
- Next Monday a ZZ is requested to 1. get phase shifter SA3/cell26 working; 2. Investigate cell 25/SA3 Quad mover (horizontal movement in wrong direction).
SPB:
- Lost shifts due to crosstalk (2 night-shifts on weekend). Apart of that other shifts went well.
FXE:
- Incident on Thursday (LPD motor not working): after restarting it lost motor positions. Stefen H. says that A. Silenzi is preparing a report and updates will follow.
MID:
- Quite good week with small hiccups in data acquisition.
HED:
- HED asks when EEE plans to update uTCA; EEE advices to please contact Bruno F.
SCS:
- Issue with replaced turbo pump: new one enters in error mode after going to 400Hz. SCS is in contact with VAC; to be fixed today.
SQS:
- Nothing to report. |
311
|
23 Mar 2022, 10:04 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 23.03.2022 |
General Announcement and A.O.B.:
- Sandra Plett announces that a fire drill will take place next Saturday 26.03 involving many areas at Schenefeld. It starts at 10:00AM and around 6 persons will conduct the drill.
Please understand that this is only a training. An e-mail will be sent later.
- Theo M. says there is bridge problems in XTD2 XGM into the Doocs and Karabo side. Some trains are still missing. Other XGMs are fine. Stefen H. fears the solution lies more in the DOOCS side.
CTRL:
- The process to deploy Karabo 2.13.3 to mitigate the pipeline connection system is ongoing. Instruments will be managed in individual scheduling. Tunnels areas is proposed to proceed next Monday 28.03.
An e-mail will be sent.
DA:
- Nothing to report.
EEE:
- Fast electronics has a small intervention at HED to change a crate issue. HED is informed.
VAC:
- Nothing to report.
- A. Silenzi says he was informed by Martin D. that SA2 topics in e.g. InfluxDB or Graphana had an outage yesterday afternoon. Right now it is back to work but missed some data yesterday.
XPD:
- XGMD in FXE: the ADC was replaced and is again online.
DET:
- Nothing to report.
ITDM:
- Influx DB was down yesterday.
- Power glitch on Monday evening: cooling for 604 room at the XHQ ground floor was lost. A. Galler says XO will set a meeting with ITDM and TS to set an alarm when cooling fails.
XRO:
- Nothing to report.
LAS:
- Sandhya Venkatesan will join the XO. Int. meeting also.
- AMPHOS in SA1 and SA2 went down due to the lack of cooling water as consequence of Monday's power outage.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- XGMD is running again.
- Dmitry K. thanks Mattias Schoz for fixing an issue with beam. This is fixed. Operators loaded wrong file this morning and re-alignment had to be done.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Continuing with the clean of the chemistry chamber after an accident.
SQS:
- Nothing to report. |
310
|
21 Mar 2022, 10:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 21.03.2022 |
General Announcement:
- A. Silenzi will be the DRC this week and reminds FXE, SQS and MID to be present at the jour-fixe meeting today at 13:00.
CTRL:
- Ongoing upgrade of Karabo framework on selected services to address issues with connection output signals: CTRL would like to do SA3 tunnel and SQS today (SCS already done); others next Monday.
- Alberto deFanis says that colleagues from SQS prefer to do in a week without users. Steffen H. says that this is fine to do later. The tunnel devices have to be coordinated with SCS also.
SQS will coordinate to a proposed date and inform CTRL.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks in between user experiments.
VAC:
- Issue with ion pump in rack 3.1 taking down SA1 and SA3 beam: ZZ by VAC/OCD and PRC to exchange controller yesterday. Another ZZ will take place this morning.
Downtime for the entire North Branch could be minimized (if V0 is closed, then at least SA1 could be running. Martin D. will look into optimizing the procedure).
XPD:
- Naresh K. as PRC says on the pump failure on Sunday. Downtime was longer for SA1 as machine could not decouple both SA1 and SA3.
- Naresh K. says the access the access ZZ is not so quick as it was done in dump area and machine has to ground magnets and power up again after the ZZ, making it take longer time (not only ZZ time).
- SCS turbo pump fail and the experiment stopped on Sunday around 15:30.
DET:
- Nothing to report.
ITDM:
- Following last Tuesday with InfluxDB break: since Tuesday things are going smoothly and looks optimistic after applying the fix. Nevertheless, more monitoring time is needed to certify it.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing major to report apart of the downtime yesterday.
- Some beam drift as usual was seen but not out of ordinary.
FXE:
- Problem with XGMD happened again and Theo M. is looking for spare parts.
MID:
- All fine at MID.
HED:
- Finished first of many experiments using RELAX laser. A minor problem is addressed by a ticket.
SCS:
- Turbo pump failure: have to be investigated and repaired again the whole vacuum until next week. Sunday at 15:00 it failed.
- SCS cameras and uTCA update is already done (see CTRL entry).
- XGM / HV inhibit problem is recurring once every week. SCS would like to be trained in how to recover it.
SQS:
- Nothing to add. |
309
|
16 Mar 2022, 10:39 |
Frederik Wolff-Fabris | General | Information | Notes XO Int. Meeting 16.03.2022 |
General Announcement:
- The Summer maintenance planning period is approaching and next Wednesday at the end of the XO Int. Meeting a presentation of the changes/improvements will be made.
Please communicate to the colleagues who are responsible for your group’s maintenance planning to join. This will be done after the regular round of discussions.
CTRL:
- Not present.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks to the groups.
VAC:
- Not present.
XPD:
- XGMD was fixed (ADC broken down twice last week): Desy people is also investigating; root cause not yet found.
- FEL Imager (YAG screen) in SA3 is found to be damaged. It was moved to a different spot for the time being.
- Ventilation in all tunnels caused beam jitter for 30-40 minutes yesterday.
It was caused by outside humidity (not common, 2-3 times per year) that filled up the filters and clogged it with water droplets leading to overrunning of the regulation schemes of pumps; then pressure
changed in tunnels.
DET:
- Nothing to report.
ITDM:
- InfluxDB: large issues yesterday (cluster crashed on MON evening); it took all day yesterday to brought up and data is now visible also at Karabo.
- ITDM has a root cause now and DAQ matrix devices were switched OFF yesterday which provides a fix. Last crash there was a lot of data that had to be synchronized and lacked resources.
- ITDM found a way to start without data from Monday evening to Tuesday 21:00. This data will be injected to InfluxDB / cluster in the coming days. In the meanwhile, one needs to use Graphana to look at
this data, not Karabo.
- Discussing future measures to trigger alarm to avoid the crash.
- Please don't start DAQ without the GO from ITDM via e-mail.
XRO:
- FXE jitter last week: not yet clear if it was the clean tent or the cooling water for M1/M2 the real cause; hard to find as cooling water was refilled and the clean tent was switched ON accidently, both
in parallel.
LAS:
- Issues with Sync system in SA1 persists: spurious 10Hz signal. PP-laser still do not have optical signal. LAS has a rough idea of the cause and is in contact with instruments.
UNSYS:
- Phase shifter in cell11/SA3 the motor was replaced and more 10 motors were ordered as spare parts.
- Quad mover in SA1#cell 18 was fixed.
- Quad mover in cell25/SA3 (wrong movement direction) is still to be addressed. MXL will work together with UNSYS.
- Tables for Phase shifters of Apple-X still to be uploaded.
- One new PS on the Apple-X is not working; Frank Brinker is informed and a ZZ is needed before the next blue week.
SPB:
- Influx issue but could use Graphana.
- Pulse energy is low this week: 2mJ while last week was 3.5mJ;
- Crosstalk with SA2: SA1 energy gets worse when changes happen in SA2 but recovers shortly after.
- SPB did not see any vibration on Tuesday 15.03. Also last week beam was stable.
FXE:
- Nothing to add.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- This morning a filter of RIXS spectrometer broken and has to be exchanged (filter gauge valve). VAC group is helping with the exchange.
SQS:
- Nothing to report. |
308
|
14 Mar 2022, 10:41 |
Frederik Wolff-Fabris | General | Information | Notes XO Int. Meeting 14.03.2022 |
General Announcement:
- On the Summer-22 maintenance period: In about 2 weeks (by end of March) the call for the summer/22 maintenance period will be send to the groups.
CTRL:
- Deployment of new framework in order to fix an issue (this week): instruments shall agree with CTRL contacts on a suitable time. It affects cameras and/or similar devices that are installed.
DA:
- Report on last week issue with processing on Maxwell including offline calibration: people were not able to open files that were there; small selected notes were not present and were corrected on Friday.
Since then it seems no issues recurred.
EEE:
- Starting equipment installation of SXP racks in rack rooms.
- PLC side calls from SQS on weekend: issues with motors to be followed up; train-IDs not getting proper.
VAC:
- Not present.
XPD:
- XGMD in FXE hutch under investigation: new issue happened at the end of the week and may be again related to last Monday's one. People are working on it (XPD, FXE and Desy).
DET:
- One comment to DA: Jolanta S-D was on DOC and issue with offline for ePIX detector still occurring after Friday. Phillip S. says it was a different problem and it is fixed.
ITDM:
- Issue (2x) with Influx): still not understood; Data was not lost on last occurrence. Hick-ups and recovered. So far under investigation.
XRO:
- Jitter at SA1 in FXE last week: mostly likely the cooling in M1 and M2 or clean tent.
- No big issues after that.
LAS:
- Degradation in SA1 PP-laser with trigger software; maybe there is damage to optics. Being investigated today and can't guarantee when it will be working again.
- SA3 change of setpoint for SQS taking one-two days. Not sure if ready today.
- SA2 is running well.
UNSYS:
- Phase shifter cell11/S3: a ZZ is ongoing also fix with remotely procedure.
- Issue with quad mover cell18/SA1 is fixed remotely.
- Issue with Quad cell25/SA3: the ZZ is needed.
SPB:
- Issue with Agipd detector: fixed (one module not sending data).
- Agipd online preview devices: fixed;
- Laser timing issue: data was not collected.
FXE:
- Jitter problem already mentioned.
- XGMD issue under investigation.
- Laser problem: stop measuring on Saturday. Not sure if data was taken during the weekend.
MID:
- Difficult week.
- Broker pileup problem on Monday and Friday: Gabriele says it may be related to a memory leak - under investigation.
On Friday there were there sources identified to help getting rid of problems (Jolanta S.D); also GUI was a second source. Not fully understood the main source of Friday. It came back stable.
- Seeding beam had to be tuned on weekend as energy decreased by factor 5 when comparing to the best lasing at the week start. No scientific output could be achieved but feasibility with setup was
demonstrated.
HED:
- Wednesday optics hutch ADC was not working. Now it works fine.
- Full power with RELAX was tested and is being analyzed. Bassler cameras acquiring wrong train IDs. It shall be fixed.
SCS:
- Not present.
SQS:
- Successful week;
- Problem with motor being fixed by SQS, ME, EEE.
- Tool for online analysis was slow
- Heat load in the soft monochromator was seen and it has to be fixed in long term. |
307
|
09 Mar 2022, 10:15 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 09.03.2022 |
General Announcement:
-
CTRL:
- Nothing major to report.
DA:
- Nothing to report.
EEE:
- Supporting instruments.
- Replacement uTCA in HED optics crate on Monday was done; being monitored during next two weeks.
VAC:
- Issue with SA3 broken ion pump controller in SA3; ZZ was done to replace controller.
- Issues when entering tunnel: XD2/4 PC turnstile crashed and D3had to reset software. Caused 30-min delay in entering tunnel.
XPD:
- Issue with FXE for XGM in their hutch: electronics did not work but was taking care by Theo / Fini. FXE has to confirm if this is fine now.
DET:
- Nothing to report.
ITDM:
- Intervention on Monday accordingly to the plan with reboot of 2 servers at SA1/FXE. Maybe another server to change in later date.
- Influx database issue on Monday: all data until 22:30 in the evening are not reliable (SA1 and SA3). Mid-term plan with CTRL to split SA1/SA3. Under investigation to find the cause.
XRO:
- FXE stability issue: last night beam was stable after the ZZ was done to switch off cooling systems of mirrors and clean tent.
LAS:
- 3 PP-lasers in operations: working
- Max L. asks for news on the winter maintenance 8D report.
- LAS send a ticket with TS concerning connection of front end UPS. Needs follow up.
- Announcement by TS on Saturday 25th June: concerning to LAS - what are the consequences? Max L. needs feedback to plan next summer maintenance period.
Tobias Clever says it is connected to construction of office building; needing new transformators and concerns power supply and not 35degrees water. XO will help with feedback.
UNSYS:
- SA1 PSs problems seemed to be solved at Desy side (new gap program done by Lars).
- Error signal from ambient field coils in cell14/SA1: Fixed by Suren A.
- In SA3 a problem with stuck quad mover: fixed by Suren A. with workaround;
- Cell 11 / SA3 phase shifter motor: could be another damaged device. It needs ZZ access.
SPB:
- Nothing to report.
FXE:
- Feedback on the XGM will be discussed internally.
- Beam jitter observed yesterday; thanks to all efforts.
- Energy change this morning took longtime.
MID:
- Monday afternoon: Karabo issue with control room due to broker pile-up; all fine now.
- No issues with HXRSS; used beam yesterday.
HED:
- Continuing prep for next week; also investigation on shielding.
SCS:
- Preparation for Liquid jet.
SQS:
- Nothing to report. |