ID |
Date |
Author |
Location |
Severity |
Subject |
Draft
|
12 Oct 2020, 09:12 |
Andreas Galler | General | Information | Notes XO Integration Meeting 12.10.2020 |
General Announcement:
CTRL:
- issues SA3 vacuum loop
- gets more and more diffcult to recover
- problem not clear, PLC comminucation is lost without obvious reasons
- no software changes took place from either CTRL nor EEE ... component issue?
- fastadc issues --> EEE
- mirror in SA3 not always following a move ommand XTD10 M1
DA:
EEE:
- OCD at HED --> hexapod
- fineDelay
VAC:
- preventive action in XTD10 for GATT scroll pump
XPD:
- SA2 imager XTD6 pop in 45 after M3 small mark on the imager --> imager still usable
DET:
ITDM:
- OCD call data aggregator at MID
- need to understand better how to avoid the situation
- users complained to maxwell service (DESY directly) about limited resources becuase of high load by other users
- there are reserved nodes --> use them!
- issues with scratch
- more than 300 TB had been accumulated within a relatively short time
- it seems that data was calibrated to scratch not to proc
- MID comments --> a special calib mode was used that could not save to proc which is why it was saved to scratch
- problem with one of the proxy servers --> degraded performance of data migration
XRO:
LAS:
- issue at SA1 with Galvo
- after shift change to FXE, laser was not working --> Galvo was out of phase
- FXE resetted something (not clear) and the laser came back
- LAS investigating the issue
UNSYS:
- nothing special
- a couple of times the undulators opened suddenly without any command issued.
- O. Henssler from DESY proveds the logs
SPB/SFX:
- coupling with SA1 and SA2 (SA2 using train on demand) is an issue
- beam walks off the XTD9 XGM aperture
- offset mirrors M2
- smaract cotnrollers
- most fixed by increasing timeout windows
FXE:
- shutter can be opened despite closed vavales
MID:
HED:
- damage on M3 mirror
- due to focusing scheme --> 9keV and intermediate focus --> during alignment unfortunately the focusing point came too close to M3
- meeting with XPD and XRO at 11:30
- hexapod issue reported to EEE is not quite fixed
- ´status is not reporting the correct status of the hexapod
- hexapod started working at some point and fell back to weird operation later again
- CRL arm7 seems to be at the limit of the x-axis
- if FEL stays where it is, XTD1 chamber might need to be moved
- JUNGFRAU issue
- up to a point the data is fine in the preview
- after some runs data seen in the preview is not written to file
SCS:
SQS:
|
169
|
14 Oct 2020, 09:22 |
Andreas Galler | General | Information | Notes XO Integration Meeting 14.10.2020 |
General Announcement:
CTRL:
- problems in SA3
- VAC loop unstable, devices cannot be brought into a stable state --> what used to be a restart every two weeks, cannot be recovered these days
- issues with motors developing a short time out
- somewhere there is a beavy load on Beckhoff loops
- TM comments: at times the movements happen very late after issuing the command
- influxDB sees high load
- any modification of or running a new macro?
- Please let CTRL know
- the issue started with VAC --> devugging shoud start there
DA:
EEE:
- working together with CTRL on SA3 issue
- OCD call from SCS but call was not correctly frowarded
VAC:
XPD:
DET:
- OCD call from SCS --> was fixed
ITDM:
XRO:
LAS:
UNSYS:
SPB/SFX:
FXE:
- people on shift could not initiate a hutch search
- GABS access right missing (Gebietsabsuche)
- introduced without informing FXE
MID:
- waiting for invitation AGIPD trainID solution
HED:
- JUNGFRAU issue presists
- HexaPOD is working again
SCS:
SQS:
- DAQ issue
- no data was recorded without
- vac is an issue but not endangering
- motor issue is vital to the instrument
|
213
|
07 Apr 2021, 08:52 |
Andreas Galler | General | Information | Notes XO Integration Meeting 07.04.2021 |
General Announcement:
CTRL:
- recurrent error reprted by LAS --> drift compensation in SA2 --> need feedback
- minor issues reported instruments during startup
DA:
EEE:
- Issue on April 4th --> 4 PLCs shut down without a cause, (looks like power cut)
- MID crate went down at 9:00
- SA3 utca shut down (could be voltage drop, could be ehternet) (9:35)
VAC:
- VAC-OCD recovered SA2 vacuum state (valves, and ystem) (see EEE SA2 incident above)
- VAC obseved that the MID Mono warmed up, the cryo warmed up presumably after a PLCs update which interrupted an interloop communication
XPD:
DET:
ITDM:
- SA1 baclony room checked
- some racks lost power --> investifation going on (possibly happened on Sat. Apr 3rd 8:00 am)
XRO:
- incorrect motor velocity in SA3 --> fixed
-
LAS:
- small issue in SA2 with slow feedback --> see CTRL report
UNSYS:
- problem on SA1 central control node, after a power cycle aircoils and phase shifters were unresponsive
SPB/SFX:
- Laser Interlock issue was solved, power spike involved that reported by MEA3 ( klaus Witt).
FXE:
MID:
- M1 mirror was moved completely out on 1st of Apr.
- --> Naresh Kujala moved M1 out.
HED:
- Laser interlock issue on Tuesday
- with regards to the MID Mono cryo, there had been HED PLC updates
- calibration pipeline issue on AGIPD
- HED utca in optics hutch went off
SCS:
SQS:
|
239
|
07 Jul 2021, 09:07 |
Andreas Galler | General | Information | Notes XO Integration Meeting 05.07.2021 |
General Announcement:
- User Operations Meeting will resume on Friday
CTRL:
- Last week found small bug for the fastADC – short down time requested
DA:
- Nothing to report.
EEE:
- Fast electronics – are planning a ZZ
- Fuses issue – MID & HED medium priority crates fixed
- SASE 3 vacuum loop has to be updated – today afternoon or tomorrow around lunch timewill be coordinated by XO with EEE and MXL
VAC:
- Nothing to report
XPD:
- Start up of equipment
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Alignment Laser interlock issue at SA3, being investigated
UNSYS:
- Nothing to report.
SPB:
- some motor issues after crate restarts in connection with but not caused by the works adressing the fuse issue
FXE:
- Nothing to report.
MID:
- all fine
HED:
- nothing.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
240
|
07 Jul 2021, 09:14 |
Andreas Galler | General | Information | Notes XO Integration Meeting 07.07.2021 |
General Announcement:
CTRL:
DA:
EEE:
VAC:
ME:
XPD:
- within next three days XPD equipment to be checked with x-rays
- DAQ test next Monday
-
DET:
- LPD
- communication with RAL and STFC, no sign of corrosion --> single module to be taken out and test in the lab
ITDM:
XRO:
LAS:
UNSYS:
SPB/SFX:
FXE:
- nothing --> hutch being prepared
- PLC loops 4 and 7 are down due to hardware issues on the PLC --> EEE on it.
- sample stack --> loop4 preparation delayed
MID:
- pre is going on, everything
HED:
SCS:
SQS:
AOB:
- Zunaira --> beam shutter keys
|
241
|
12 Jul 2021, 09:29 |
Andreas Galler | General | Information | Notes XO Integration Meeting 12.07.2021 |
General Announcement:
CTRL:
- InFlux, still experiencing delay with querying the database (20min, karaba; 5min Grafana)
- Jan reports that in the XGM trendlines a similar effect is seen
DA:
EEE:
- Friday call from BKR, because of MPS blocking
- FXE SHUT MPS signal was not reset
- ZA comments, that the reason is not yet clear --> ZA is contact with Rathjen and MPS experts
VAC:
ME:
XPD:
- For the FXE SHUT it seems MPS and karabo data are differeing
-
DET:
- LPD
- Hazem is back on Friday --> Supermodules will be tested then in the Lab
- AGIPD
- special dark procedur currently not working well
ITDM:
XRO:
- This week, component check in all SASE tunnels
-
LAS:
UNSYS:
- Friday
- SA1 cell 10, Suren exchanged the coupler
SPB/SFX:
FXE:
- nothing to report
- PLC issues loop4 and loop7 solved
MID:
HED:
SCS:
SQS:
AOB:
- Zunaira --> beam shutter keys
|
242
|
14 Jul 2021, 09:28 |
Andreas Galler | General | Information | Notes XO Integration Meeting 14.07.2021 |
General Announcement:
CTRL:
- InlFlux logging, some statisitics had been gathered --> but no root cause had been found.
- Load Balancer would needed to be restarted --> data inside buffer would be
DA:
EEE:
- PLC team is still busy with check up of the "fuse" issue
- MID cabling going
VAC:
ME:
XPD:
- one XGM didn't ecevice a signal
- downstream XGM's are running as well
DET:
- AGIPD
- dark procedure fixed and seems to be working now at SPB/SFX, has to be installed at HED and MID
- LPD
- 3 supermodules had been moved to the DET LAB --> so far visual inspection does not show major issues
- no module powered yet
ITDM:
XRO:
LAS:
- humidty in PPLAS
- TS working on it
- PKG after maintenance did not come back to specs
UNSYS:
SPB/SFX:
FXE:
- nothing to report --> shutter porblem solved
MID:
HED:
SCS:
SQS:
AOB:
|
265
|
04 Oct 2021, 09:30 |
Andreas Galler | General | Information | Notes XO Integration Meeting 04.10.2021 |
General Announcements
none
CTRL
- late train IDs at XGM, being sorted
DA
- nothing to report
EEE
DET
ITDM
- nothing to report
XRO
-
LAS
- Failure of Power SUpply on AMPHOS in SA3, had been replaced over the weekend,
- observed jump on ODL in SA2 around midnight --> being investigated
UNSYS
- nothing to report
- PKG status to be discussed
SPB/SFX
- nothing major to report
FXE
MID
- Feedback for M2 stopped working, started calculating wrong values
HED
- nothing to report
SCS
- problems with DAQ: DA1 stuck --> DOC and ITDM are aware
SQS
- nothing to report
AOB
|
266
|
06 Oct 2021, 09:28 |
Andreas Galler | General | Information | Notes XO Integration Meeting 06.10.2021 |
General Announcement:
CTRL:
DA:
EEE:
VAC:
XPD:
DET:
- nothing to report (SPB reports JF1M issue --> suspected readout board issue)
ITDM:
- issues with Nagios
- misconfig by puppet mangement (DESY)
XRO:
LAS:
UNSYS:
- Phaseshifter tables for higher energies in blue week had been uploaded
- two quad movers from SA2 moved to 0
- triggered by doocs --> quad movers moved back but it is unclear if they actually returned the BBA optimized position
- clarification with M. Scholz confirms that quad movers did not move.
SPB/SFX:
FXE:
- User Experiment starts
- C&C update requested until next week for LPD (uTCA)
MID:
HED:
SCS:
SQS:
AOB:
|
267
|
11 Oct 2021, 09:27 |
Andreas Galler | General | Information | Notes XO Integration Meeting 11.10.2021 |
General Announcement:
CTRL:
DA:
EEE:
- several motors in SPB/SFX had trouble, all originating from the same crate
VAC:
- VAC problems at HED last Wed
- monochromator section in the tunnel, two unidentified vacuum bursts
XPD:
DET:
ITDM:
XRO:
- FXE (PZ) reported a large jump in the bender setting, sebsequently Ry and pitch drifted for days and stabilized later
- M1 M2 alignment changed as well
- currently stable and being investigated
LAS:
UNSYS:
SPB/SFX:
- unstable pulse energy jittering ~ 35% over days.
FXE:
- Experiment went well, apart from minor issue with calibration pipeline with JF
MID:
HED:
SCS:
SQS:
AOB:
|
268
|
13 Oct 2021, 09:26 |
Andreas Galler | General | Information | Notes XO Integration Meeting 13.10.2021 |
General Announcement:
- This meeting is a tool to share information/experiences across scientific and technical groups
CTRL:
DA:
EEE:
- next week
- update on GATT and EPS loop in SA3 in roder to allow for new protection of GATT for the whole beaemtime
- SA2 on Thursday (maybe Friday), try to correct backlash issue --> loop update
VAC:
- all is running
- HED VAC event
- miscommunication between operators led to this issue
XPD:
- no issue to report
- next week, during the whole week there'll be a high photon energy test at SA1 and SA2 see a big involvement of XPD
- 24 keV @SA1 (Tue)
- ~30 keVC @ SA2 (Wed/Thu)
- PI/Planner is Frank Brinker
DET:
- investigating JF1M issue of 4-6Hz instead of 10 Hz operation
ITDM:
XRO:
- SA1 M1/M2 drifts under investigation
- XRO is involved as well with next weeks high photon energy effort as well as EEE loop updates
LAS:
UNSYS:
SPB/SFX:
- user exp this week, 24 keV next week
FXE:
MID:
HED:
- next week high photon energy
SCS:
SQS:
- next week material tests, maybe also detector group.
AOB:
|
288
|
20 Dec 2021, 09:25 |
Andreas Galler | General | Information | Notes XO integration meeting 20.12.2021 |
General Announcement:
CTRL:
DA:
EEE:
VAC:
XPD:
- nothing special, shtudown tasks are in schedule, HXPES / XTED2 cable work ongoing
DET:
ITDM:
XRO:
LAS:
UNSYS:
- APPLEXv inv place, VAC to be closed next year
SPB/SFX:
- working on mirrors
- not clear what is the status with the movement of racks wrt SXP (SPB/SFX optics hutch racks currently powered down)
- electrical installation?
- PG comments: Brad won't allow operation until 230 V power distribution is calrified, once Brad gives the OK, the racks can be switched on
racks are being finalized by EEE, but some checks need electrical power
- RB: SPB/SFX were promised that the move of the racks were finished before X-Mas, but it seems the electrical work is not up to satandards
FXE:
- shutdown task in schedule
MID:
- all fine
- some loop updates planned for Today
HED:
SCS:
SQS:
AOB:
- light OCD over X-Mas for XPD in case of floowings, power outages.lists will be
|
289
|
05 Jan 2022, 09:29 |
Andreas Galler | General | Information | Notes XO Integration Meeting 05.01.2022 |
General Announcement:
CTRL:
DA:
EEE:
- 1 MID plC dead
- cable work planned for next week
VAC:
- In general excellent progress
- closing of M2 XTD2 (SA1) in progress, sealing issues mitigated
XPD:
- XTD10, XTD6
- improvements, commissioning ongoing
- clean room tent water leak reported on 23.12.
DET:
ITDM:
- Jan 19: dCachce maintenance
XRO:
- progress as planned
- changes to existing components will be posted to respective elogs
LAS:
UNSYS:
SPB/SFX:
FXE:
MID:
- PLC outage on Jan 1 st caused VAC issues (reported by EEE)
- no access to MID optics hutch due to interlock test
HED:
- everything on schedule
- leak on a hose to humidifier device in A12. This leaked a few litres on the floor.
SCS:
SQS:
|
297
|
02 Feb 2022, 08:39 |
Andreas Galler | General | Information | Notes XO Integration Meeting 02.02.2022 |
General Announcement:
CTRL:
- observation from DOC
- macros some times after the upgrade to Python 3.8 --> please contact CTRL?
DA:
- significant upgrade to Offline Analysis -->
- proc data won't inlcude a copy of raw data any longer
EEE:
VAC:
XPD:
- no incidents
- 2 tunnel accesses on Monday --> succesful
DET:
- DET commissioning at SPB (AGIPD JUNGFRAU 1M), MID (AGIPD), HED (ePIX, JUNGFRAU)
ITDM:
- E0.604 rack room, Wed next week
- fixing power distribution units in every rack --> TS and external company will need a full day
- affected services are: Lab DAQ, LAB control servers (EEE; CTRL, DET), GitLab runners, one machine for data analysis, PLC database, karabo project database (?)
- Yesterday short interruption with influx reading note --> test requested for next Monday causing a 1 hour interruption
XRO:
- SA1 / SA2 M3 PBLMs have been equipped with diamond plates and can be used for beam stability measurements
- Backlash Controls implemented last week --> values
- report of lost positions of vertical slit in SA3
- in case of similar issues please contact DOC prior to self fixing.
LAS:
- SA2 PPL is good
- SA3 PPL planned to be up today
- SA1 PPL being investigated as agreed with SA1 instruments
UNSYS:
- transparency test for APPLE X went successful
SPB/SFX:
- SPB/SFX aligned since yesterday, no issues
FXE:
- blown fuses in crates --> solved with EEE
MID:
HED:
- Com yesterday
- outstanding issue: fastADC not working EEE aware of that
SCS:
SQS:
- GATT always worked but very slowly responding
- OCD contacted
- Raul comments that this is a known issue and should be communicated to CTRL
- SH nesures that Raul creates the ticket
- XTD10_PES
- digitizer display frozen almost all time (connected to 2 DAQs) --> reported to DOC
- SH comments that when connecting to two DAQs (tunnel and SQS), should be connected to only one DAQ.
|
314
|
04 Apr 2022, 09:26 |
Andreas Galler | General | Information | Notes XO integration meeting 04.04.2022 |
General Announcement:
CTRL:
- PP LAS restarting device servers for upgrading the device tag --> autoinstantiate some servers all topics
DA:
- Planned maintenance today for Cal and gpfs delayed at least until next Monday
EEE:
VAC:
- 1 ZZ ongoing to XTD4 to shiled an ion pump controller, suspicion of radiation damage.
XPD:
- similar Diamond cracks as in solid attenuator had been observed in the transmissive imagers
- Last weeek FEL imager issue had been clarified (image was rotated by 180 °, occured due to faultly config load)
DET:
- AGIPD1M (SPB/SFX), Friday morning some DAQ issues induced train losses. Didn't happen again after careful checking
ITDM:
- Issue at HED with data migration --> discussion will take place
XRO:
- Tomorrow and after tomorrow late shift some CTRL support might be required
LAS:
- PLL LAS SA1 sync issues (LbSYNC informed and arriving 10:00)
- PPL LAS3 NOPA setup
UNSYS:
SPB/SFX:
- Experiment went well, no major issue
- Run Controller crashd a few times over the weekend. Could be restarted but crashes some time later and has to be restarted
FXE:
- affecetd by LAS Sync issue
- issue with doocs panel for shutters (yifeng)
MID:
- all fine
- SSRY motor issue (ELMO)
- Nerea updated the ticket --> motor is working
HED:
- DA1 got stuck for a while (fixed by restarting)
- lot of short runs (1min) and only one train isneed from that -> requires fast data migration and analysis -> doesn't always work
- discussion with ITDM commencing
- PAM was missing from DAQ
SCS:
- no major issues
- RIXS on liquids went well
SQS:
DOC /DRC
issues with call forwarding |
315
|
06 Apr 2022, 09:15 |
Andreas Galler | General | Information | Notes XO integration meeting 06.04.2022 |
General Announcement:
CTRL:
- influxDB outage (telegraf issue)
- SA1 and SA3 affected
- currently fixed but linked to a device in SQS
DA:
EEE:
- PLC colleagues are working on CR and support requests
- pulling cables for SQS and SCS
- reported to XO uTCA in XTD1
- remote reboot planned for Friday, reserve ZZ slot for Friday
VAC:
- repair of XTD2 solid attenuator ongoing
XPD:
DET:
ITDM:
- nothing to report
- next Monday GPFS update will take place
XRO:
- mirror calibration started in SA2
- beam path through offset mirros not visible -> no beam on M3 pop in, but visible on M2 pop in MCP in beam?
- JG: no MCP in beam, how about PBLM?
- LS: no PBLM in beam
-
LAS:
- Sync issue from Monday cannot be replicated by Sebastian -->
- tests for SPB happening tomorrow
UNSYS:
- circular polratizaton lasing achievd in APPLE-X in left and right hand polarization, also linear horizontal and vertical and 45° measured at 900 and 700 eV
- thanks to UNSYS and DESY APPLE-X comm team.
SPB/SFX:
FXE:
MID:
HED:
SCS:
- reservation of nodes for jupyter hub.
- requested during drc hour fix to reserve 3 nodes
- is it possible to reserve 3 nodes by default?
- Olivier reports there was a misunderstanding and it got fixed very quickly
- Janusz reports that by standard 4 nodes are reserved for each experiment.
SQS:
|
316
|
11 Apr 2022, 09:10 |
Andreas Galler | General | Information | Notes XO integration meeting 11.04.2022 |
General Announcement:
CTRL:
- small intervention --> udpate of karabo-gui client exflagteway
DA:
- maxwell gpfs software maintenance --> offline calibration pipeline affected see ITDM
EEE:
- OCD call on rack 2.01 which was powered down. After fuse repaired
VAC:
XPD:
- Rack 2.01 --> Camera of SR imagare is not working
- situation not clear, being investigated --> IP conflict?
- Janusz suggests to powercycle the camera
DET:
ITDM:
- 10:00 maxfell gpfs update
XRO:
LAS:
- remainder change of set point for SCS to 112 kHz
UNSYS:
SPB/SFX:
FXE:
- preparing for user experiments
- concerned about optical sync of PP Laser, probelm of last week was fixed but no reason found
- test last thursday optical sync lost again while moving optical delay stage
- FXE
MID:
- alignment of optical laser alignment in joined hutch.
HED:
SCS:
SQS:
- last week Thursday night
- mono drift investigated --> preliminary analyisis points to the grating as the drifting component
|
317
|
13 Apr 2022, 09:28 |
Andreas Galler | General | Information | Notes XO Integration Meeting 13.04.2022 |
General Announcement:
- No User operation meeting this Friday, instead cw 16 / cw 17 check will happen in Daily Operations Meeting at 15:30 on Thu Apr 14. Link available on ttfinfo in XFELelog
CTRL:
DA:
EEE:
VAC:
XPD:
DET:
- offline calib with JUNGFRAU --> tickets closed already
ITDM:
XRO:
- request to SA1 instruments
- please check the repaired XTD2 solid attenuators
- attenuators cannot sustain too much power, offline discussion follows
- SA2 mirror recalib
- roll of M1 and M2 was completely misaligned, recalibration successful
- geometrical center of the kinematic system roughly identified
LAS:
UNSYS:
SPB/SFX:
FXE:
MID:
- pump probe looks stable and getting ready for the user
- BAM doocsbridge stopped --> manual restart necessary
HED:
SCS:
SQS:
DRC
- one hour data lost in influxDB in SA2
PRC
|
320
|
27 Apr 2022, 09:04 |
Andreas Galler | General | Information | Notes XO Integration Meeting 27.04.2022 |
General Announcement:
CTRL:
- bug noticed in BEckhoof karabo device --> already fixed from some topics, see xo-info comm from yesterday
DA:
EEE:
VAC:
- two devices went into error --> likely because of Beckhoff Karabo bug
XPD:
DET:
- AGIPD 1M at SPB partially vented yesterday --> detector went into interlock, no damage
- who should be called at SPB (or any instrument for that matter)
- Turbo pumps went into error state which triggered the partial vent --> karabo Beckhoff update?
- Wajid comments: karabo beckhoff update does not correlate with this incident, judging from time stamps
ITDM:
XRO:
LAS:
UNSYS:
SPB/SFX:
FXE:
- software limits of phase shifter of PP Laser
MID:
- all fine at MID
- HXRSS very much appreciated
HED:
SCS:
SQS:
DRC
PRC
- SCS incident on Monday, photon energy scanning script was left running --> had to be stopped manually
|
338
|
04 Jul 2022, 09:30 |
Andreas Galler | General | Information | Notes XO Integration Meeting 04.07.2022 |
General Announcement:
- M. Döring:
- almost all of the activities that should have been done until last Friday had been done
CTRL:
- last week karabo topic updates had finishedexcept for LAS
-
DA:
EEE:
VAC:
- leak search on M6 (SA3) chamber
- leak search in XTD9
- all in all well
XPD:
- almost all tasks have been completed
- XTD1 new crate connected on Friday,
DET:
ITDM:
- all updates are finsihed
- some activities continuing on the offsite --> should be comüpletely transparent
- after the prox change there are some PCs which are using the old host names --> relavant people will be contacted by Janusz Szuba
XRO:
- M6 mirror installation beign finished
- end switch checks shoudl be finished by today
- ramp up procedure being written down
- instruments are asked to submit any additional requests for ramp up
LAS:
- fastADCs should be finished by Wed this wek
UNSYS:
- control system has to be adjusted after removal of APPLE X, i.e. the control system looks for motors and
SPB/SFX:
FXE:
- motor commissioning ongoing
MID:
HED:
- when modifying cables downstairs, make sure modules upstairs are switched off
SCS:
SQS:
DRC
PRC
- SCS incident on Monday, photon energy scanning script was left running --> had to be stopped manually
|