Notes XO Integration Meeting 13.04.2022, posted by Andreas Galler on 13 Apr 2022, 09:28
|
General Announcement:
- No User operation meeting this Friday, instead cw 16 / cw 17 check will happen in Daily Operations Meeting at 15:30 on Thu Apr 14. Link available on ttfinfo in XFELelog
CTRL:
DA:
EEE:
VAC:
XPD:
DET:
- offline calib with JUNGFRAU --> tickets closed already
ITDM:
XRO:
- request to SA1 instruments
- please check the repaired XTD2 solid attenuators
- attenuators cannot sustain too much power, offline discussion follows
- SA2 mirror recalib
- roll of M1 and M2 was completely misaligned, recalibration successful
- geometrical center of the kinematic system roughly identified
LAS:
UNSYS:
SPB/SFX:
FXE:
MID:
- pump probe looks stable and getting ready for the user
- BAM doocsbridge stopped --> manual restart necessary
HED:
SCS:
SQS:
DRC
- one hour data lost in influxDB in SA2
PRC
|
Notes XO integration meeting 11.04.2022, posted by Andreas Galler on 11 Apr 2022, 09:10
|
General Announcement:
CTRL:
- small intervention --> udpate of karabo-gui client exflagteway
DA:
- maxwell gpfs software maintenance --> offline calibration pipeline affected see ITDM
EEE:
- OCD call on rack 2.01 which was powered down. After fuse repaired
VAC:
XPD:
- Rack 2.01 --> Camera of SR imagare is not working
- situation not clear, being investigated --> IP conflict?
- Janusz suggests to powercycle the camera
DET:
ITDM:
- 10:00 maxfell gpfs update
XRO:
LAS:
- remainder change of set point for SCS to 112 kHz
UNSYS:
SPB/SFX:
FXE:
- preparing for user experiments
- concerned about optical sync of PP Laser, probelm of last week was fixed but no reason found
- test last thursday optical sync lost again while moving optical delay stage
- FXE
MID:
- alignment of optical laser alignment in joined hutch.
HED:
SCS:
SQS:
- last week Thursday night
- mono drift investigated --> preliminary analyisis points to the grating as the drifting component
|
Notes XO integration meeting 06.04.2022, posted by Andreas Galler on 06 Apr 2022, 09:15
|
General Announcement:
CTRL:
- influxDB outage (telegraf issue)
- SA1 and SA3 affected
- currently fixed but linked to a device in SQS
DA:
EEE:
- PLC colleagues are working on CR and support requests
- pulling cables for SQS and SCS
- reported to XO uTCA in XTD1
- remote reboot planned for Friday, reserve ZZ slot for Friday
VAC:
- repair of XTD2 solid attenuator ongoing
XPD:
DET:
ITDM:
- nothing to report
- next Monday GPFS update will take place
XRO:
- mirror calibration started in SA2
- beam path through offset mirros not visible -> no beam on M3 pop in, but visible on M2 pop in MCP in beam?
- JG: no MCP in beam, how about PBLM?
- LS: no PBLM in beam
-
LAS:
- Sync issue from Monday cannot be replicated by Sebastian -->
- tests for SPB happening tomorrow
UNSYS:
- circular polratizaton lasing achievd in APPLE-X in left and right hand polarization, also linear horizontal and vertical and 45° measured at 900 and 700 eV
- thanks to UNSYS and DESY APPLE-X comm team.
SPB/SFX:
FXE:
MID:
HED:
SCS:
- reservation of nodes for jupyter hub.
- requested during drc hour fix to reserve 3 nodes
- is it possible to reserve 3 nodes by default?
- Olivier reports there was a misunderstanding and it got fixed very quickly
- Janusz reports that by standard 4 nodes are reserved for each experiment.
SQS:
|
Notes XO integration meeting 04.04.2022, posted by Andreas Galler on 04 Apr 2022, 09:26
|
General Announcement:
CTRL:
- PP LAS restarting device servers for upgrading the device tag --> autoinstantiate some servers all topics
DA:
- Planned maintenance today for Cal and gpfs delayed at least until next Monday
EEE:
VAC:
- 1 ZZ ongoing to XTD4 to shiled an ion pump controller, suspicion of radiation damage.
XPD:
- similar Diamond cracks as in solid attenuator had been observed in the transmissive imagers
- Last weeek FEL imager issue had been clarified (image was rotated by 180 °, occured due to faultly config load)
DET:
- AGIPD1M (SPB/SFX), Friday morning some DAQ issues induced train losses. Didn't happen again after careful checking
ITDM:
- Issue at HED with data migration --> discussion will take place
XRO:
- Tomorrow and after tomorrow late shift some CTRL support might be required
LAS:
- PLL LAS SA1 sync issues (LbSYNC informed and arriving 10:00)
- PPL LAS3 NOPA setup
UNSYS:
SPB/SFX:
- Experiment went well, no major issue
- Run Controller crashd a few times over the weekend. Could be restarted but crashes some time later and has to be restarted
FXE:
- affecetd by LAS Sync issue
- issue with doocs panel for shutters (yifeng)
MID:
- all fine
- SSRY motor issue (ELMO)
- Nerea updated the ticket --> motor is working
HED:
- DA1 got stuck for a while (fixed by restarting)
- lot of short runs (1min) and only one train isneed from that -> requires fast data migration and analysis -> doesn't always work
- discussion with ITDM commencing
- PAM was missing from DAQ
SCS:
- no major issues
- RIXS on liquids went well
SQS:
DOC /DRC
issues with call forwarding |
Notes XO Integration Meeting 30.03.2022, posted by Frederik Wolff-Fabris on 30 Mar 2022, 09:51
|
General Announcement:
- A. Galler mentions that new Corona rules are in place. Please read the e-mail from Nicole E. from last Friday. In case of questions don’t hesitate in contacting SRP and/or taskforce.
CTRL:
- LPD motor issue under investigation. Report to follow later.
DA:
- Nothing to report.
EEE:
- Update on HED OPT uTCA crate: one last action was done and hopes to get it solved. Under monitoring to see if another intervention is still needed.
VAC:
- Nothing to report.
XPD:
- Issue after Karabo update: SA1 FEL Imager is not available for some time. Solved by CTRL/A. Parenti. For the BKR it took a little bit longer to get it ready. It may happen again.
DET:
- Nothing to report.
ITDM:
- Question if it is possible for next week to update the GPFS client to fix the recent bugs with permissions (Maxwell cluster calibration pipeline - solved at the time with a hot fix).
The complete version is ready to be deployed. One needs to update and restart service (matter of 1-2 hours) and the current proposal is to make it on Monday. An e-mail will be sent to XO-info.
XRO:
- Middle layer device of SRA in SA2 had the offset parameters placed to zero and as consequence it showed unreal numbers. M. Vannoni placed back old parameters and right now it shows the correct numbers.
It was positioned in the correct place but numbers were wrong. Wajid E. says parameters can be retrieved or please write a ticket to CTRL if it happens again.
- As PRC, machine had isolated problems yesterday night and this morning.
LAS:
- Synchronization for the PP-lasers went down. LbSync is informed.
- PP-laser in SA3 went fully down and SA2 is showing instabilities. LAS is working on them.
UNSYS:
- ZZ is planned for next Monday.
SPB:
- Problem with XTD2 XGM not solved and SPB/SFX is losing data. This seems to be a Doocs issue and needs to be addressed.
- Fast ADC issues under discussion with EEE.
- Machine performance is fine.
FXE:
- Nothing to report.
MID:
- Nothing to report.
HED:
- Nothing to report.
SCS:
- The turbo pumps problems were solved; Not clear if current activity is affected due to the issues with PP-laser. It will be asked to the instrument people.
SQS:
- Experiments from last week went well accordingly to the internal group meeting. |
Notes XO Integration Meeting 28.03.2022, posted by Frederik Wolff-Fabris on 28 Mar 2022, 10:09
|
General Announcement and A.O.B.:
- BTM @ FXE was triggered on Saturday afternoon and only PRC was available to come onsite and fix the issue. Different procedures will be proposed to be put in place.
- SRI conference starts this week.
CTRL:
- Issue with data loggers last week: narrowed down to an issue with expected update of Telegraph. This time it caused a trigger in the broker pileup. Solved with help from ITDM.
- Upgrade in tunnels today (already agreed; see last meeting minutes).
- Issue with MID/SPB AGIPD middle layer: requires an upgrade of framework. Currently there is no agreed time with instruments but proposal is to make it on a Monday.
DA:
- Nothing to report.
EEE:
- Busy with FXE pump commissioning.
VAC:
- Nothing to report.
- Martin Dommach, as PRC, reports that energy change in SA1 was not permitted during the weekend (costing many lost shifts in SPB) due to crosstalk with SA3;
on Saturday an event with BTM monitor in XTD9/FXE shutter happened and brought down machine. SA2 and SA3 was restored after 1 hour and SA1 after 3,5 hours.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Problem with shutter in SQS instrument laser hutch during the weekend: it could continue apart of the error state. Under investigation.
UNSYS:
- Next Monday a ZZ is requested to 1. get phase shifter SA3/cell26 working; 2. Investigate cell 25/SA3 Quad mover (horizontal movement in wrong direction).
SPB:
- Lost shifts due to crosstalk (2 night-shifts on weekend). Apart of that other shifts went well.
FXE:
- Incident on Thursday (LPD motor not working): after restarting it lost motor positions. Stefen H. says that A. Silenzi is preparing a report and updates will follow.
MID:
- Quite good week with small hiccups in data acquisition.
HED:
- HED asks when EEE plans to update uTCA; EEE advices to please contact Bruno F.
SCS:
- Issue with replaced turbo pump: new one enters in error mode after going to 400Hz. SCS is in contact with VAC; to be fixed today.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 23.03.2022, posted by Frederik Wolff-Fabris on 23 Mar 2022, 10:04
|
General Announcement and A.O.B.:
- Sandra Plett announces that a fire drill will take place next Saturday 26.03 involving many areas at Schenefeld. It starts at 10:00AM and around 6 persons will conduct the drill.
Please understand that this is only a training. An e-mail will be sent later.
- Theo M. says there is bridge problems in XTD2 XGM into the Doocs and Karabo side. Some trains are still missing. Other XGMs are fine. Stefen H. fears the solution lies more in the DOOCS side.
CTRL:
- The process to deploy Karabo 2.13.3 to mitigate the pipeline connection system is ongoing. Instruments will be managed in individual scheduling. Tunnels areas is proposed to proceed next Monday 28.03.
An e-mail will be sent.
DA:
- Nothing to report.
EEE:
- Fast electronics has a small intervention at HED to change a crate issue. HED is informed.
VAC:
- Nothing to report.
- A. Silenzi says he was informed by Martin D. that SA2 topics in e.g. InfluxDB or Graphana had an outage yesterday afternoon. Right now it is back to work but missed some data yesterday.
XPD:
- XGMD in FXE: the ADC was replaced and is again online.
DET:
- Nothing to report.
ITDM:
- Influx DB was down yesterday.
- Power glitch on Monday evening: cooling for 604 room at the XHQ ground floor was lost. A. Galler says XO will set a meeting with ITDM and TS to set an alarm when cooling fails.
XRO:
- Nothing to report.
LAS:
- Sandhya Venkatesan will join the XO. Int. meeting also.
- AMPHOS in SA1 and SA2 went down due to the lack of cooling water as consequence of Monday's power outage.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- XGMD is running again.
- Dmitry K. thanks Mattias Schoz for fixing an issue with beam. This is fixed. Operators loaded wrong file this morning and re-alignment had to be done.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Continuing with the clean of the chemistry chamber after an accident.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 21.03.2022, posted by Frederik Wolff-Fabris on 21 Mar 2022, 10:02
|
General Announcement:
- A. Silenzi will be the DRC this week and reminds FXE, SQS and MID to be present at the jour-fixe meeting today at 13:00.
CTRL:
- Ongoing upgrade of Karabo framework on selected services to address issues with connection output signals: CTRL would like to do SA3 tunnel and SQS today (SCS already done); others next Monday.
- Alberto deFanis says that colleagues from SQS prefer to do in a week without users. Steffen H. says that this is fine to do later. The tunnel devices have to be coordinated with SCS also.
SQS will coordinate to a proposed date and inform CTRL.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks in between user experiments.
VAC:
- Issue with ion pump in rack 3.1 taking down SA1 and SA3 beam: ZZ by VAC/OCD and PRC to exchange controller yesterday. Another ZZ will take place this morning.
Downtime for the entire North Branch could be minimized (if V0 is closed, then at least SA1 could be running. Martin D. will look into optimizing the procedure).
XPD:
- Naresh K. as PRC says on the pump failure on Sunday. Downtime was longer for SA1 as machine could not decouple both SA1 and SA3.
- Naresh K. says the access the access ZZ is not so quick as it was done in dump area and machine has to ground magnets and power up again after the ZZ, making it take longer time (not only ZZ time).
- SCS turbo pump fail and the experiment stopped on Sunday around 15:30.
DET:
- Nothing to report.
ITDM:
- Following last Tuesday with InfluxDB break: since Tuesday things are going smoothly and looks optimistic after applying the fix. Nevertheless, more monitoring time is needed to certify it.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing major to report apart of the downtime yesterday.
- Some beam drift as usual was seen but not out of ordinary.
FXE:
- Problem with XGMD happened again and Theo M. is looking for spare parts.
MID:
- All fine at MID.
HED:
- Finished first of many experiments using RELAX laser. A minor problem is addressed by a ticket.
SCS:
- Turbo pump failure: have to be investigated and repaired again the whole vacuum until next week. Sunday at 15:00 it failed.
- SCS cameras and uTCA update is already done (see CTRL entry).
- XGM / HV inhibit problem is recurring once every week. SCS would like to be trained in how to recover it.
SQS:
- Nothing to add. |
Notes XO Int. Meeting 16.03.2022, posted by Frederik Wolff-Fabris on 16 Mar 2022, 10:39
|
General Announcement:
- The Summer maintenance planning period is approaching and next Wednesday at the end of the XO Int. Meeting a presentation of the changes/improvements will be made.
Please communicate to the colleagues who are responsible for your group’s maintenance planning to join. This will be done after the regular round of discussions.
CTRL:
- Not present.
DA:
- Nothing to report.
EEE:
- Continuing with support tasks to the groups.
VAC:
- Not present.
XPD:
- XGMD was fixed (ADC broken down twice last week): Desy people is also investigating; root cause not yet found.
- FEL Imager (YAG screen) in SA3 is found to be damaged. It was moved to a different spot for the time being.
- Ventilation in all tunnels caused beam jitter for 30-40 minutes yesterday.
It was caused by outside humidity (not common, 2-3 times per year) that filled up the filters and clogged it with water droplets leading to overrunning of the regulation schemes of pumps; then pressure
changed in tunnels.
DET:
- Nothing to report.
ITDM:
- InfluxDB: large issues yesterday (cluster crashed on MON evening); it took all day yesterday to brought up and data is now visible also at Karabo.
- ITDM has a root cause now and DAQ matrix devices were switched OFF yesterday which provides a fix. Last crash there was a lot of data that had to be synchronized and lacked resources.
- ITDM found a way to start without data from Monday evening to Tuesday 21:00. This data will be injected to InfluxDB / cluster in the coming days. In the meanwhile, one needs to use Graphana to look at
this data, not Karabo.
- Discussing future measures to trigger alarm to avoid the crash.
- Please don't start DAQ without the GO from ITDM via e-mail.
XRO:
- FXE jitter last week: not yet clear if it was the clean tent or the cooling water for M1/M2 the real cause; hard to find as cooling water was refilled and the clean tent was switched ON accidently, both
in parallel.
LAS:
- Issues with Sync system in SA1 persists: spurious 10Hz signal. PP-laser still do not have optical signal. LAS has a rough idea of the cause and is in contact with instruments.
UNSYS:
- Phase shifter in cell11/SA3 the motor was replaced and more 10 motors were ordered as spare parts.
- Quad mover in SA1#cell 18 was fixed.
- Quad mover in cell25/SA3 (wrong movement direction) is still to be addressed. MXL will work together with UNSYS.
- Tables for Phase shifters of Apple-X still to be uploaded.
- One new PS on the Apple-X is not working; Frank Brinker is informed and a ZZ is needed before the next blue week.
SPB:
- Influx issue but could use Graphana.
- Pulse energy is low this week: 2mJ while last week was 3.5mJ;
- Crosstalk with SA2: SA1 energy gets worse when changes happen in SA2 but recovers shortly after.
- SPB did not see any vibration on Tuesday 15.03. Also last week beam was stable.
FXE:
- Nothing to add.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- This morning a filter of RIXS spectrometer broken and has to be exchanged (filter gauge valve). VAC group is helping with the exchange.
SQS:
- Nothing to report. |
Notes XO Int. Meeting 14.03.2022, posted by Frederik Wolff-Fabris on 14 Mar 2022, 10:41
|
General Announcement:
- On the Summer-22 maintenance period: In about 2 weeks (by end of March) the call for the summer/22 maintenance period will be send to the groups.
CTRL:
- Deployment of new framework in order to fix an issue (this week): instruments shall agree with CTRL contacts on a suitable time. It affects cameras and/or similar devices that are installed.
DA:
- Report on last week issue with processing on Maxwell including offline calibration: people were not able to open files that were there; small selected notes were not present and were corrected on Friday.
Since then it seems no issues recurred.
EEE:
- Starting equipment installation of SXP racks in rack rooms.
- PLC side calls from SQS on weekend: issues with motors to be followed up; train-IDs not getting proper.
VAC:
- Not present.
XPD:
- XGMD in FXE hutch under investigation: new issue happened at the end of the week and may be again related to last Monday's one. People are working on it (XPD, FXE and Desy).
DET:
- One comment to DA: Jolanta S-D was on DOC and issue with offline for ePIX detector still occurring after Friday. Phillip S. says it was a different problem and it is fixed.
ITDM:
- Issue (2x) with Influx): still not understood; Data was not lost on last occurrence. Hick-ups and recovered. So far under investigation.
XRO:
- Jitter at SA1 in FXE last week: mostly likely the cooling in M1 and M2 or clean tent.
- No big issues after that.
LAS:
- Degradation in SA1 PP-laser with trigger software; maybe there is damage to optics. Being investigated today and can't guarantee when it will be working again.
- SA3 change of setpoint for SQS taking one-two days. Not sure if ready today.
- SA2 is running well.
UNSYS:
- Phase shifter cell11/S3: a ZZ is ongoing also fix with remotely procedure.
- Issue with quad mover cell18/SA1 is fixed remotely.
- Issue with Quad cell25/SA3: the ZZ is needed.
SPB:
- Issue with Agipd detector: fixed (one module not sending data).
- Agipd online preview devices: fixed;
- Laser timing issue: data was not collected.
FXE:
- Jitter problem already mentioned.
- XGMD issue under investigation.
- Laser problem: stop measuring on Saturday. Not sure if data was taken during the weekend.
MID:
- Difficult week.
- Broker pileup problem on Monday and Friday: Gabriele says it may be related to a memory leak - under investigation.
On Friday there were there sources identified to help getting rid of problems (Jolanta S.D); also GUI was a second source. Not fully understood the main source of Friday. It came back stable.
- Seeding beam had to be tuned on weekend as energy decreased by factor 5 when comparing to the best lasing at the week start. No scientific output could be achieved but feasibility with setup was
demonstrated.
HED:
- Wednesday optics hutch ADC was not working. Now it works fine.
- Full power with RELAX was tested and is being analyzed. Bassler cameras acquiring wrong train IDs. It shall be fixed.
SCS:
- Not present.
SQS:
- Successful week;
- Problem with motor being fixed by SQS, ME, EEE.
- Tool for online analysis was slow
- Heat load in the soft monochromator was seen and it has to be fixed in long term. |
Notes XO Integration Meeting 09.03.2022, posted by Frederik Wolff-Fabris on 09 Mar 2022, 10:15
|
General Announcement:
-
CTRL:
- Nothing major to report.
DA:
- Nothing to report.
EEE:
- Supporting instruments.
- Replacement uTCA in HED optics crate on Monday was done; being monitored during next two weeks.
VAC:
- Issue with SA3 broken ion pump controller in SA3; ZZ was done to replace controller.
- Issues when entering tunnel: XD2/4 PC turnstile crashed and D3had to reset software. Caused 30-min delay in entering tunnel.
XPD:
- Issue with FXE for XGM in their hutch: electronics did not work but was taking care by Theo / Fini. FXE has to confirm if this is fine now.
DET:
- Nothing to report.
ITDM:
- Intervention on Monday accordingly to the plan with reboot of 2 servers at SA1/FXE. Maybe another server to change in later date.
- Influx database issue on Monday: all data until 22:30 in the evening are not reliable (SA1 and SA3). Mid-term plan with CTRL to split SA1/SA3. Under investigation to find the cause.
XRO:
- FXE stability issue: last night beam was stable after the ZZ was done to switch off cooling systems of mirrors and clean tent.
LAS:
- 3 PP-lasers in operations: working
- Max L. asks for news on the winter maintenance 8D report.
- LAS send a ticket with TS concerning connection of front end UPS. Needs follow up.
- Announcement by TS on Saturday 25th June: concerning to LAS - what are the consequences? Max L. needs feedback to plan next summer maintenance period.
Tobias Clever says it is connected to construction of office building; needing new transformators and concerns power supply and not 35degrees water. XO will help with feedback.
UNSYS:
- SA1 PSs problems seemed to be solved at Desy side (new gap program done by Lars).
- Error signal from ambient field coils in cell14/SA1: Fixed by Suren A.
- In SA3 a problem with stuck quad mover: fixed by Suren A. with workaround;
- Cell 11 / SA3 phase shifter motor: could be another damaged device. It needs ZZ access.
SPB:
- Nothing to report.
FXE:
- Feedback on the XGM will be discussed internally.
- Beam jitter observed yesterday; thanks to all efforts.
- Energy change this morning took longtime.
MID:
- Monday afternoon: Karabo issue with control room due to broker pile-up; all fine now.
- No issues with HXRSS; used beam yesterday.
HED:
- Continuing prep for next week; also investigation on shielding.
SCS:
- Preparation for Liquid jet.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 07.03.2022 , posted by Frederik Wolff-Fabris on 07 Mar 2022, 10:39
|
General Announcement:
- Accelerator is now running at 11.5GeV for the next 4 delivery weeks. The SA1 and SA3 systems show already good photon intensities and at SA2 the Self-seeding setup is taking place today.
CTRL:
- ITDM would like to reboot control machines in SA1 and FXE. It will proceed around lunch time.
DA:
- Update of offline calibration: it went fine, successful.
EEE:
- Replacement uTCA of HED optics today.
- Past week cables installation went fine.
- Motors in SA3 SQS laser setup and motor to be updated.
VAC:
- ZZ on solid attenuator XTD2: adjusted solid attenuator in-position (driving 2mm less in) to get a fresh spot on Si foil.
- ZZ on XTD9 ion vacuum pump: shows noble gas instability; adjusted set point; hope to survive until shutdown.
XPD:
- Setting up HIREX for HXRSS.
DET:
- Nothing major to report.
ITDM:
- Two servers to be updated (see CTRL report)
XRO:
- Nothing to report.
LAS:
- All fine with PP-laser.
UNSYS:
- Yesterday problem with cell 28/SA2: motor of undulator was stuck; after power cycle it worked.
- Issues with phase shifter in SA1: cell 26-31 is under investigation.
- Replacement of two phase shifter motors in SA3; possibly due to radiation damage.
SPB:
- Nothing to report.
FXE:
- Problem with LPD DAQ; ITDM solved last weekend.
MID:
- All fine.
HED:
- Exchange crate by EEE; busy preparing experiment.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 02.03.2022, posted by Frederik Wolff-Fabris on 02 Mar 2022, 10:12
|
General Announcement:
- There is a high probability that we will run out of Helium gas due to difficulties with stock. If an instrument needs He on next weeks please get in touch with TS to verify its availability.
CTRL:
- Last Monday intervention on the SA3 GATT middle layer servers was successful.
DA:
- Today a major upgrade of offline calibration is scheduled during which services won't be available.
EEE:
- Supporting different tasks at SQS (recommissioning motors), HED (motor encoders), SCS and FXE.
VAC:
- Ion pump in XTD9 shows a potential risk of tripping interlock: a ZZ on Friday 14:00 will take place to change parameters.
- Another ZZ in XTD2 to attempt to fix the solid attenuator in SA1 (hole in the arms reported by SPB / 0.5mm Si damaged); try to change the blocking mechanism.
- Dmitry K. asks on the Si intervention in XTD2: Martin D. says a change of Si is a major intervention as it involves need of clean tent etc… and will not be done this Friday.
The idea is to use a fresh spot on the attenuator. Conditions for interlocks have to be later checked with Liuba S.
XPD:
- ZZ on Monday to remove Shimadzu camera was successful.
DET:
- Nothing to report.
ITDM:
- Problem with Influx cluster which did not accepted data for 3 hours: search for fixes last days and today a new configuration will be applied, followed by restart afterwards (minutes downtime). No effect
in tunnels DAQ.
XRO:
- Commissioning beamtime for SA2 mirrors: available beamtime was short. The coating stripes were found in beamlines in respect to motor positions. More beamtime is needed to continue commissioning of M1,
M2 and M3 in XTD6.
LAS:
- Nothing to report.
UNSYS:
- Apple-X commissioning ongoing: working on Control systems commissioning. Last cell was controlled; more tests ongoing today. The phase shifter tables are not yet ready.
SPB:
- Working with different groups during the week.
FXE:
- D3 was present on Monday to test the BTM; it was successful.
MID:
- Radiation alarm triggered due to a false alarm at MID hutch. Thomas P. from HED helped by calling RPOs; W. Clement, Eric B. and M. Prollius helped to solve it.
HED:
- Preparing for the high intensity laser and also working on motors.
SCS:
- Not present.
SQS:
- Not present. |
Notes XO Integration Meeting 28.02.2022, posted by Frederik Wolff-Fabris on 28 Feb 2022, 10:35
|
General Announcement:
- This week a facility development week takes place without user operation. XO thanks the groups for posting at the Operations-2022 e-log.
CTRL:
- Update on the SA3 GATT loop (middle layers devices) to be done following to find a good time slot.
DA:
- Today from 10:00-12:00 the calibration system will be down due to an update.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Nothing to report concerning XPD; SCS made an enter that XGM went OFF for a day.
- Regarding PRC, the machine was running well most of the time; problem with XTD2 Air conditioning due to a fire safety flap malfunctioning happened (downtime of several hours in SA1);
on Sunday magnet power supplies at the accelerator had a problem and took few hours to restore beam delivery as it required tunnel access.
- The two-color was not created in the right order but it was corrected afterwards. The bandwidths were adjusted a few times and HED said it went well, including their spectrometer.
- XO proposes that the info on the two-color (order) can be entered in the comment section of the delivery table.
DET:
- Issue with AGIPD at SPB did not happened again and currently there is no plan for any intervention.
ITDM:
- Janusz S. says the hardware monitoring needs to be rebooted for several servers (PC layers at SCS, MID) and a short downtime will happen. Proposition is to make today if no complains appear.
- Two control servers also need maintenance and proposal is to make next week.
XRO:
- Busy with activities in this blue week (e.g. Mirror calibrations); it may require CTRL support.
LAS:
- For next week a stage in SA1 will be replaced.
- A change of operation point in SA1/SPB and S3/SQS will happen this week. SCS also need a change working point to 1.1MHz.
UNSYS:
- Tests with the Apple-X to take place this week; preparation for commissioning and lasing attempt.
SPB:
- User time largely went OK; small troubles with DAQ and detector.
FXE:
- D3 is testing components in FXE hutch today.
- CTRL, ITDM and FXE will discuss on the previously mentioned updates.
MID:
- All fine at MID.
HED:
- A ticket send to EEE is being under work.
SCS:
- XGM high voltages went down to zero and was restarted by powering on again. Cause it is not clear and will be followed up by XPD and SCS members.
SQS:
- Continuing work of moving chamber to prepare for next week beamtime. Some PLCs for the chamber have to be reset motors and other things. EEE is helping with it. |
Notes XO Integration Meeting 23.02.2022, posted by Frederik Wolff-Fabris on 23 Feb 2022, 10:30
|
General Announcement:
- A. Galler reminds instruments and support groups to feel welcome in making regular entries on the Operations-2022 e-log.
CTRL:
- FastX of GATT middle layer devices was updated last Monday; the slowness issue is improved and fixed.
- Analyzing to upgrade the broker pile-up issue at Karabo: probably to be fixed by next Monday.
- On the BAM monitor: few properties were changed in the DOOCS side. Changes are transparent to DAQ and all the data must be able to be stored.
Problem appears when data shall be analyzed and then must be looked to the new DAQ. In case of issues please get in touch with CTRL.
DA:
- Maxwell GPFS access rights bug: patch was deployed on Monday and went successfully.
- Next week there will be an update of calibration (further announcement to be done).
EEE:
- Started installing Coax cables for SXP.
- LBM in MID: EEE contacted Beckhoff and will try to get a fix done still this week.
VAC:
- Nothing to report.
XPD:
- Issues with Desy mobile phones: not all phones were equally affected. Issue seems to be fixed this morning.
- In SA2 the SRA slits were too wide open (probably from tuning times). They were closed yesterday night after users finished work.
- Beam delivery to instruments is running fine and smooth.
DET:
- AGIPD detector at SPB: issue with one of 16 modules not sending data since yesterday. Investigation ongoing with DOC and experts.
Jolanta S.-D. says that this is a new issue which happened twice yesterday and Desy is contacted. An intervention for the hardware during the blue week shall take place. It may be related to electronics
outside detector.
ITDM:
- Issue with "phantom" data appearing at DAQ in SA3 instrument: investigation shows it can't be reproduced; maybe it relates to data arriving earlier. It was only visible at DOOCS devices (Undulators
device). Under further investigation by ITDM.
XRO:
- CRLs in SA2 fixed by ZZ.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing apart of the detector report.
FXE:
- Thanks to CTRL for addressing BAM issue.
MID:
- Thanks for solving CRL2 issue on Monday.
HED:
- Two-color mode working with nice energies. Investigation source shape from undulators with lenses.
SCS:
- Nothing to report.
SQS:
- Not present. |
Notes XO Integration Meeting 21.02.2022, posted by Frederik Wolff-Fabris on 21 Feb 2022, 11:57
|
General Announcement:
- Philipp S. says a downtime for the calibration / GPFS update is to be done today at 12:00.
- Raul V. says that the biggest issue of last PRC week was related to the FXE beam pointing. In general, was a quiet and productive week. Please, instrument shall remind to make entries in the operation e-
log.
CTRL:
- In SA3 GATT activity will be performed today (fix to slowness) as beam is down and there is no activity.
DA:
- Announcement will come today: restart GPFS issue / Maxwell to be restarted.
EEE:
- One OCD on Saturday: issue with SDL motors in MID; to be followed up today.
VAC:
- Nothing else apart of the Gatt fix.
XPD:
- Shimadzu camera is installed in XTD6. A test has to be done and organized with instruments.
DET:
- Nothing to report.
ITDM:
- Fix to be deployed today: plan is to deploy Maxwell cluster done in the background; it shall not disturb any user of the system. All other storages with login to be done in March and still to be
confirmed.
- Slow data investigated: get wrong values with time stamp that could be corrected. Further investigation will take place this week.
XRO:
- FXE stability pointing issues on Friday afternoon: Raul V. placed a report in Operation E-log.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Pointing issue from Friday;
- In DAQ the BAM device name has changed and could not be used. Please keep the compatibility for names; A. Silenzi will check with CTRL.
MID:
- Motor problems: in touch with CTRL.
- Today morning the CRL2 lenses were retracted but not giving the proper signal. It has been under investigation.
- Issue with DAQ: slow devices were saving empty data (no entries); at least happening with two devices. It has been reported and one is fixed and the second is under investigation.
HED:
- Initial setup this afternoon for the two-color operation.
SCS:
- Not present.
SQS:
- Not present. |
Notes XO Integration Meeting 16.02.2022, posted by Frederik Wolff-Fabris on 16 Feb 2022, 10:42
|
General Announcement:
- XO launched a new XO Alfresco presence. Please check there for information related to operations and do not hesitate in giving suggestions or raising questions.
Adam R. asks for the schedule for the PRCs and DRCs. A. Galler says that the PRCs are included in the photon delivery tables. A discussion with Olivier M. on the DRC will take place.
Naresh K. asks on the status display: it is current a screenshot for the moment the page is loaded. It is not yet a live display.
- A. Galler reminds instruments to make entries to the Operations-2022 E-log to help the PRCs to track down and spot problems.
CTRL:
- Intervention on Monday GATT update: action to be done again next Monday to increase performance.
DA:
- Nothing to report.
EEE:
- Intervention on HED to replace a fast ADC card that lost communication to CPU. Replacement to take place today.
VAC:
- Nothing to report.
XPD:
- A. Koch reports that yesterday the Karabo fast-X was slow.
DET:
- Nothing to report.
- Olivier M. also observed slow fast-X connection to gateway.
ITDM:
- Janusz S. says the fast-X issues can be related to an overload yesterday.
- Login to Influx had intermittent connection for a half day yesterday n the afternoon. Investigation being taking place of it relates to fastX.
XRO:
- Nothing to report.
LAS:
- Issues with SA1 PP-laser: overshooting in the CM station. Maintenance this morning is taking place. Current timing is not as good as it could be.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Nothing to report apart of the ADC issue.
SCS:
- All fine at SCS.
SQS:
- SQS shutter issue from yesterday 15.02: problem was solved and it was due to a card misbehaving.
Zunaira A. says that the problem was with the Lasermet interlocks. Work done in the maintenance affected signals going to radiation protection interlocks. The particular signal was missing and after a fix
it worked again. |
Notes XO Integration Meeting 14.02.2022, posted by Frederik Wolff-Fabris on 14 Feb 2022, 10:02
|
General Announcement:
- Thanks to instruments that posted at the Operations-2022 e-log. Please make regular entries that allow PRCs to recognize eventual issues that have to be tracked down.
CTRL:
- Nothing to report.
DA:
- Friday there was issues with Data Aggregators processing in several detectors. DOC found it was an issue with the new version. It was fixed by Saturday by working with a separate routing.
This was a bug in the new version (time of template creation). A final fix will be installed in a suitable date.
EEE:
- Nothing to report.
VAC:
- Intervention to solve issue GATT issues is postponed by CTRL (patch work is almost ready). To be discussed between Raul V., CTRL once deployment can be done only on Mondays.
XPD:
- Issue noticed that XGM data were showing missing trains at the start of last week. Tickets were issues and it was cured by end of the week – it affected only one XGM. Reason for missing trains must be
asked to Theo M.
- SA1 pulse energy was around 4.5mJ; a really good performance.
- Preparing calorimeter in SA1 hutch for experiment.
DET:
- Two OCD calls (one from HED) and one from SPB on a detector. Both fixed by DOC / OCDs.
ITDM:
- Incident on Saturday with influx DB cluster: happened by 3:00AM; started investigating during the morning. By 14:00-15:00 the cluster came back. Data synchronization is ongoing. Not sure yet if there is
lost data, under investigation.
XRO:
- Main BKR scene will contain a button to Graphana page.
LAS:
- Working with CTRL on Nagios issues.
UNSYS:
- Nothing to report.
SPB:
- Saturday problem with history of motor position at Karabo: solved by DOC.
- Broken cable for encoder of focusing mirror still has to be fixed (MKB focusing mirror in hutch).
FXE:
- Preparing for the week.
MID:
- All fine at MID.
HED:
- Week went good: successful user experiment.
- Beam was drifting quickly between left/right. Not sure if this is related to the new HED feedback loops. Under investigation.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 09.02.2022, posted by Frederik Wolff-Fabris on 09 Feb 2022, 10:09
|
General Announcement:
CTRL:
- GATT broker pile up: there is a fix available but not foreseen to be updated now. The middle layer will be fixed in cw09 (blue week). This will be coordinated with XO.
- Raul V. completes that 1. slowness of the GATT from last week can be fixed next Monday (which triggers a pile up); 2. More invasive changes mentioned by Wajid E. to be done in a later point.
- Wajid E. adds that if problems appear again (unresponsive middle layer), please call DOC.
DA:
- Nothing to report.
EEE:
- Finishing smaller tasks.
- Fast ADC issue is taken over by the fast electronics team. The cooling issue may be related to connection problems with the board. EEE monitors if last Monday fix was enough.
VAC:
- Nothing else apart of the GATT discussion.
XPD:
- XGM issue in XTD2: someone changes the settings leading to an overload in the XGM. A. Galler says that BKR operators did not touch it and one still looks to find who triggered the changes.
- Calorimeter and laser polarimeter to be installed at FXE on next days.
- A. Koch says there was an update of the DAQ in tunnels; that was tested on Monday afternoon and if problems are found, please contact A. Koch.
DET:
- Nothing to report.
ITDM:
- Nothing to report apart from ground floor room 604 work.
XRO:
- XRO contact control integrations to work on motor layer devices for March.
LAS:
- Nagios Karabo system still do not work.
UNSYS:
- Nothing apart of the smaller gap limitation in SA2 system for the 2-color test last Monday.
SPB:
- Nothing to report.
FXE:
- Hiccup with train-IDs not being synchronized: a ticket was sent to CTRL.
MID:
- All fine at MID.
HED:
- Not present.
SCS:
- Issues with middle layer devices that had to be restarted: Laurent M. will check on the status with SCS colleagues.
SQS:
- Issue with XTD10 PES digitizer: there is an open ticket in progress.
- Trigger of hutch shutter last week: the loose maglock has been addressed. |
Notes XO Integration Meeting 07.02.2022, posted by Frederik Wolff-Fabris on 07 Feb 2022, 10:56
|
General Announcement:
- Power glitch on Sunday morning at around 04:00AM seems to have no effect at machine or Schenefeld site.
CTRL:
- During winter shutdown the bunch arrival monitor was updated. An e-mail will be sent reflecting the changes.
DA:
- Single OCD call was solved: configuration of train match bridge to eXtra foam.
EEE:
- Two tickets from MID shall be followed this week.
- Tickets from DOC on SQS (solved) and PES (to be fixed by DA).
VAC:
- All fine with VAC.
XPD:
- All fine with Diagnostics.
DET:
- Nothing to report.
ITDM:
- Wednesday there will be shutdown of rack room in ground floor (604): details were sent by e-mail and there were no concerns/feedback from potentially affected people.
XRO:
- Nothing to report.
LAS:
- All 3 PP-lasers are stable and will be available by the end of the day.
UNSYS:
- Issue FRI/SAT: bad communication between balcony and SASE1 cells. It is solved but being further investigated.
- This morning there was a BKR request to go lower undulator gaps in SA2: gap can go as lower to 10.61mm.
SPB:
- Commissioning of AGIPD done; successful week.
FXE:
- Preparing the beamline; ready for beamtime.
MID:
- Issue with RF window: it was too short when they received the beam. Solved by PRC and instruments for fast solution.
- EEE had to solve an issue.
- Cameras in the tunnel monochromator may need a ZZ. This afternoon VAC will join for a ZZ and fix it.
- Week was successful.
HED:
- Quite successful beam usage last week.
- Issues with fast ADC devices not working until Wednesday. It worked for 24hs and then had to be restarted again and came out with wrong settings.
This needs further investigation and is needed for users this week. CTRL will follow up.
SCS:
- Preparing with UAC from tomorrow.
SQS:
- Safety problem with magnet interlock in one door: a loose screw was fixed (for the time beam).
- Issue with digitizer: SQS will clarify with CTRL.
- Recheck calibration of monochromator (needs cameras looking to the motor): it seems that the camera has to be discussed with XRO.
Martin D. says that currently Maurizio V. is in the tunnel and SQS shall contact for the possibility of a quick fix now.
- Zunaira A. says that if beam shutter closes without apparent reason: it can be related to emergency off, panic switch, air box or the maglock. It happens 3 times on the same day. If this happens again,
it may trip the entire accelerator.
People in safety group or laser interlock are responsible for fixing the maglocks. A. Galler asks Zunaira A. to send an e-mail to xo-info@xfel.eu.
Alberto Defanis says that Eric B. has fixed. Zunaira comments that this fix was momentarily for the weekend. Further discussion will follow. |