ID |
Date |
Author |
Location |
Severity |
Subject |
161
|
16 Sep 2020, 10:20 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 16.09.2020 |
General Announcement:
- Yesterday evening a ZZ in XTD10 was planned but it turned unsuccessful due to no access in the turnstiles. D3 solve the problem in the middle of the night and since
then ZZ is possible again.
- Eric Boyd mention on the SQS shutter: a XTD10 ZZ was planned for 8:30AM. He has currently no news so far.
CTRL:
- XGM Pulse pattern decoder issue is under investigation.
- Vacuum issues (giving unknown state in Karabo): also under investigation.
- Next Monday CTRL plans to update Karabo login system. Groups shall inform CTRL if there is any issue against this.
- Last Influx training will be next Friday An e-mail to participants will be sent.
DA:
- No issues to be reported.
EEE:
- SA1/ Solid Attenuator Interlock problems: EEE is investigating and news will follow. FXE found an issue within the motion loop last night and XRO also helps.
It is the Interlock on the attenuator itself and not in the EPS.
VAC:
- Nothing to report.
XPD:
- XGM signals were monitored when SA1 went to 5keV for a short time yesterday night. No issues were observed.
DET:
- SCS made an OCD call on the DSSC which has been fixed.
- Agipd issue: under investigation together with ITDM (a ticket was opened to ITDM). This is not a calibration issue.
ITDM:
- ITDM will follow on the Maxwell issue with Agipd.
- CTRL and ITDM working and progressing on the bunch patter issue.
XRO:
- Issue with SA1 solid attenuator: same behavior can be expected also in SA2.
- Mirror inspection in XTD2: it shows no big damage on it. A full service will be made later. Next Monday inspection on XTD10 mirror to understand situation.
- Meeting to investigate possible upgrades to drive motors faster will take place. Participation of CTRL, DA and instruments.
LAS:
- Nothing to report.
UNSYS:
- Suren K. informs Beckhoff Hardware for cell#31/SA2 was received and is under test. As soon UNSYS is ready for a ZZ it will inform PRCs for scheduling a date.
Mostly likely ZZs will occur on Tuesday and not Monday due to continuation of 11.5 GeV studies.
- Communication problem with one undulator on Monday was quickly fixed.
SPB:
- Working on nano KB mirrors.
- 5 keV last night: gives right trajectory. It looks like the beam performance was sufficient for SPB alignment. Fine tuning has to come.
FXE:
- Affected by the shutter problem during the tentative ZZ access yesterday.
- Solid Attenuator issue in discussion with EEE.
- Sampling rate of ADC has changed. Peter Z. contacted EEE and it has been accessed.
MID:
- Not present.
HED:
- User experiment ongoing.
- Some stages worked intermittently and has to be investigated (related to crystal alignments).
- Pulse picker is not behaving correctly. It may be a Hardware issue and has to be investigated. Then HED will communicate to EEE.
SCS:
- PP laser was down last night and the OCD was contacted. There is the suspicion the crystal is damaged and the OCD informed it will be fixed this morning.
Daniel Kane will try to get some info inside LAS and report back t SCS.
SQS:
- It has users this week and shutter becomes a serious issue. In Karabo the shutter state changed but it is disconnected. It is not clear if the ZZ team is progressing
right now.
- Problem is that when trying to close the shutter via JDDD interface it went to an intermediate state but shutter did not close.
This cause interlock not opening the hutch and the beam could not be sent to the hutch. No beam since 5PM yesterday. Also locally operated in the panel it does not work.
The JDDD interface worked nicely the whole last week. Eric Boyd asks if problem appeared at the first operation. Tommaso M. says they did a lot of times prior having this
issue.
- Waiting for updates from CTRL mentioned. |
162
|
21 Sep 2020, 16:06 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 21.09.2020 |
General Announcement:
- Last week issue with Oracle database upgrade was fixed by effort during the night. People entered tunnel by ZZ after this communication issue was fixed.
CTRL:
- It is planned to happen a Data loggers update today.
- Time server update: related to a bug which happened twice in five years. It will be carried out today afternoon (after XPD beamtime) or tomorrow morning.
- A. Galler asks on updates on pulse pattern decoder: Steffen H. says it is ready to be deployed and tested.
DA:
- DA received only one OCD call last week: no specific issues during the week.
EEE:
- XO informs that the ATT interlock PLC update will be done tomorrow (22.09) morning.
VAC:
- OCD call on Friday from MID related to the cryocooler of monochromator: it was clear issue was related to the power supply. TS was involved to solve the problem by
exchanging cryocooler.
VAC informs the OCD number to be dialed is 9-4900 and not 4900.
XPD:
- It was observed problems with SA2/XGM spinning motor gauges.
DET:
- Nothing special: quiet week.
ITDM:
- There is a problem with SASE2 online data storage. It happens data were not migrated from Schenefeld to Desy or it was very slow which in turn affected MID. Partially
solved by making system live.
- ITDM needs to diagnose the GPFS filesystem in SASE2 in order to understand the issues we experienced over the weekend. This may lead to some instabilities or even
unavailability of the SASE2 online data storage.
- XO suggests ITDM to coordinate separately with MID and HED.
XRO:
- Nothing to report.
LAS:
- SA3 had power drops and today LAS makes tests regarding Daniel Rivas info.
- SA1 currently replacing laser diode and updating uTCA system.
- SA2 nothing to report.
UNSYS:
- Terminals for SA2/cell31 have arrived and are ready to be installed in next ZZ access.
- Problem with air coil in SA3/cell12 this morning will be taken care.
SPB:
- Commissioning last week using 5 keV beam.
FXE:
- Mono commissioning.
MID:
- Recovered vacuum issue on cryocooler on SAT morning with help from TS. Beam was back on Saturday morning (+-10AM).
- Timing tests were running on Friday by EEE. MID waits for results to be analyzed. Train-ID issue seems to be progressing and many people work on it.
HED:
- User beamtime was very successful. Stability was very good.
- Pulse on demand test aimed to be able to have single PP shot synchronized with FEL. Today from 10AM CTRL and LAS will temporarily change bunch pattern (communicated to
the PRC).
No need for X-ray beam to the hutch.
SCS:
- No special issues to report.
SQS:
- Two weeks ago the monochromator was used. X-ray beam was twinkling and not reproducible. Seems to be connected to higher heat load when using 300 pulses.
SQS says that this needs to be addressed when beam is back again. |
163
|
23 Sep 2020, 11:05 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 23.09.2020 |
General Announcement:
- XO is anticipating in one week the delivery table preparations to gain more time to spot possible problems related to beam delivery. The cw42 table was sent to the
instruments last Monday.
CTRL:
- CTRL confirms but updates on the dataloggers and time servers were done successfully.
- LAS brought some issues to CTRL that are under investigations.
DA:
- Nothing to report.
EEE:
- EEE conducted yesterday the SA1/SA2 ATT interlock logic modification. Summary can be found in these links:
https://in.xfel.eu/elog/SASE1+technical+commissioning/1982
https://in.xfel.eu/elog/SASE2+technical+commissioning/830
VAC:
- VAC reminds in general that EuXFEL staff shall check their access rights in the DACHS database. To have trainings up to date is not enough.
In case your rights are expired, please contact Martin Dommach via e-mail.
XPD:
- FD week had a good start with GATT bleaching studies. Today there is the FD K-mono 4-Bounce ongoing.
- Machine switched back to 14 GeV on Monday; XGM/SA1 gas was exchanged back; graphite screen in SA1 is inserted.
DET:
- Nothing to report.
ITDM:
- The SASE2 online data storage was diagnosed and reconfigured.
XRO:
- A. Trapp adds on the solid attenuators interlock modification for SA1/SA2: from now it is not possible to remove the diamonds if the silicon is in.
For more info please check the e-log or contact M. Vannoni:
https://in.xfel.eu/elog/SASE1+technical+commissioning/1982
https://in.xfel.eu/elog/SASE2+technical+commissioning/830
- Yesterday there was a ZZ in XTD9 for SPB to inspect BPLM: seems to be OK but minor issues to be resolved for the winter shutdown.
LAS:
- SA1 in maintenance: no risk for FXE during the weekend. It must be operational tomorrow morning.
- SA2: issue with ADC cards - the NOPA is lost. Reported to CTRL.
- SA3: no issues.
UNSYS:
- ZZ in SA2/Cell31 is scheduled for tomorrow (24.09) morning at 09:00AM.
SPB:
- Involved in the update of attenuator.
- Helping the stability task force activities this week.
FXE:
- FXE informs the second Gotthard is back: Karabo can't start or reset. CTRL suggests that when there is an urgent case one shall call the OCD. Otherwise sending a CTRL
ticket is most suitable.
- Last week issue with digitizer: A new card was installed but has different sampling frequency. EEE is aware and is looking into it.
- FXE comments it is hard for the instrument staff to attend the two Graphana trainings due to night shifts/compensation, etc. FXE asks on possibility of additional
trainings.
Steffen H. says one can organize an extra training. A blue week is more suitable for the instrument staff. The extra training will be organized for the coming days/weeks.
MID:
- Train-ID issue is progressing.
- MID asks for help from technical and engineering staff to find parts with M20 holes adaptors in order to lift granite stone.
** Post meeting update: VAC group helps MID.
HED:
- Not present.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
164
|
28 Sep 2020, 11:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 28.09.2020 |
General Announcement:
- Andreas G. informs the winter maintenance period will happen in two months approximately. The call for tasks will go out this week to the support groups and
instruments.
- XO informs Desy and TS are investigating the Thursday's event of power cut but do not yet have a clear reason for it.
- Jan G. informs that SCS hutch will be closed and searched this afternoon for the FD on Gain Studies.
CTRL:
- Notification service informed on a broken event. After a Karabo shutdown the FXE topic is fine again.
- Power cut: CTRL sees no issues as leftovers.
- Investigation of missing data in DOOCS (10 seconds outdated): CTRL has two colleagues looking into it.
- Supporting EEE and LAS on the ongoing working with PP-lasers.
- Update of dataloggers has to be made again. It means a 2 minutes downtime during the course of today/afternoon if nobody has something against.
DA:
- Question to CTRL: a notification service was forward to us. DA needs further description on that. Steffen H. says it was related to the python server and this is not
deployed by CTRL.
This is the reason why was forward to DA.
EEE:
- Follow up from Friday recover on the power glitch of Thursday: some cards in SA1 balcony rooms were not reachable anymore on Friday. These lead to a complete restart of
the whole crate (means interruption of timing). After additional local intervention communication to the MPS was restored.
- Changed the fast ADC of PP laser in SA1.
VAC:
- Nothing to report.
XPD:
- No issues to report.
DET:
- Agipd was power up after the power cut and had no issues.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- SA3 PP-laser: problems with SQS site. It misses the power energy (dropped down) and is under investigation. Maybe it is related to some optics assembling.
- SA2 PP-laser: power supply for AMPHOS was lost on Friday. The spare part arrives today at 11:00 and will be changed this afternoon.
- SA1 PP-laser: problem with uTCA (no ADC running and not possibility to login). Crate hangs after every few hours. Working together with EEE (100% in EEE side).
UNSYS:
- The problems with cell1/SA2 not moving were fixed by Suren K.
- Cell2/SA3 hardware problems (couplers) found this morning relates to an Ethernet card which needs to be replaced by ZZ. This cell is not operational.
- Phase shifters in SA3, cell4 and cell5 are not moving. The hardware works and it may be some higher level controls.
- Couplers were sent to Beckhoff for further investigation in order to find solution for decreasing the number of failures in future.
SPB:
- Nothing to report.
FXE:
- Due to issues with PP-laser the FXE did not test the timing tool during the weekend.
MID:
- All fine: still minor vacuum issue in the optical room but MID hopes to be solved soon.
HED:
- Pulse on demand tool was tested last Monday.
SCS:
- Nothing to report.
SQS:
- Nothing to add. |
165
|
30 Sep 2020, 12:08 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 30.09.2020 |
General Announcement:
- XO informs the Call for the Winter Maintenance tasks will be sent to the support groups and Instruments today.
Please note it is not named "shutdown" anymore are critical services such as basic infrastructure must be maintained. At least in this round it will be called as "Winter
Maintenance" to avoid confusion among the staff.
The tasks list shall be sent to xo-planning@xfel.eu.
- In case of need to go to tunnel via ZZ, please contact the PRC at 9-4800.
- Gerd W. informs on the SXP construction work that starts on Nov. 23rd. The plan aims for a minimal impact in SA3 area at XHEXP1.
There will be new connections to the cooling water system in SA3and civil constructions.
Cooling water will be cut from Nov. 23rd until Christmas. This means rack rooms and laser will be affected.
- Gerd W. answers Raul V.: the cut cooling water supply does not involve tunnels.
- Tommaso M. says instruments need to have permanent cooling water supply.
Gerd W. states the electronics must be shut down and will not be able to re-start before Christmas as it takes time to empty the system, weld, do pressure tests and fill
it up again.
- XO suggests that SA3 instruments must coordinate closely with Gerd and also involve MB (Serguei M.).
- Nerea J. asks if this applies to SA3 balcony room also. Gerd W. says that the rack rooms will be affected.
- Gerd informs there is a series of meetings taking place on Thursdays at 14:00 and this construction work can be addressed. The SCS and SQS group leaders have been
invited to this series.
If someone else wants to participate, please contact Gerd W. for getting the Zoom link,
- Monica T. says that this lack of cooling water may impact the DSSC move to SQS by delaying its test.
CTRL:
- CTRL experiences some login hiccups related to information sent to InfluxDB. This will be fixed Sunday night (Monday at 02:00AM). There is no reason this issue will
affect devices until the fix is done.
DA:
- Train-ID investigation for AGIPD offset is ongoing. DA understands from now on DET is responsible for the leading efforts.
- Monica T. comments it has been identified the issue related to the AGIPD firmware (2019 version, accordingly to Jolanta). This will require a significant change and to
do it now is not an option.
The good news is that this is a systematic issue and DET know how to take care of it.
- A. Galler asks the small task force on the train-ID to make a small presentation during next XO Integration Meeting.
EEE:
- ADC in PP-laser (SA1) is confirmed to be solved.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Issue with DSSC yesterday: one Interlock signal was permanently enabled. EEE solved after restarting the device. It will be further investigated.
ITDM:
- Desy/IT requested ITDM a time slot to fix redundant parts of the routers between "ours" and "accelerators". To do so Desy/IT and MXL suggests to apply the fix (do not
expect side effects) today before lunch time.
No relation to the timing system of lasers is expected (low risk). None in this meeting's audience raises an issue against the proposed time.
XRO:
- Nothing to report.
LAS:
- All 3 PP-lasers are up. Currently LAS still make some studies and instruments must send a message in case the need of use. Lasers are ready to be provided to
instruments.
UNSYS:
- A ZZ is currently ongoing to fix issues with failed couplers in SA3 control cabinets.
- Beckhoff replies the cause of failure is either an overload of current or relations to temperature. Temperature seems not to be the issue (accordingly to UNSYS).
SPB:
- Nothing to report.
FXE:
- Plan to test digitizer (HQ412): issue is that sampling rate is not reproducible as in the lab. Probably it will happen tomorrow together with PP-laser use.
MID:
- The vacuum issue with the monochromator is under control.
HED:
- Preparing the user next week (that uses the pp-laser).
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
166
|
06 Oct 2020, 10:36 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 05.10.2020 |
General Announcement:
- XO has sent to the support groups and instruments the call for the winter maintenance tasks. Deadline for submission is Oct. 16th.
CTRL:
- Friday night incident with signals of SA3 vacuum loop. It took hours to recover the system.
- A deployment/restart of online cluster nodes will be done today meaning 10 minutes downtime. Devices / clone devices shortly unavailable during the
deployment.
- CTRL reminds that tickets that may require Doocs bridge are better worked if sent during the week time.
DA:
- No major issues to report.
- DA will carry an offline discussion with CTRL on the cluster nodes deployment. CTRL says this is for the CPU nodes.
EEE:
- Nothing major to report. No incidents during the weekend.
- Looking into PL/fast ADC mTCA issue with SA1 PP-laser. To be discussed separately with LAS, XO after the meeting: next solution is to exchange CPU -
needing coordination with ITDM and CTRL.
VAC:
- No major issues. Karabo SA3 VAC loops look fine now.
XPD:
- PES: ZZ on Friday to fix issue with a broken gauge, mostly likely related to the power failure. Cables had to be switched. PES works again.
- Tests next week using PES: Joakim L. will set the system.
- Change of SA3 XGM settings during this week: Theo M. will change setup during weekdays.
DET:
- Nothing major to report.
ITDM:
- No OCD calls during the weekend.
XRO:
- Nothing to report.
LAS:
- Shutdown AMPHOS in SA3 PP-laser (SCS/SQS): LAS asks for one single point of communication with the instruments; This laser needs work today morning to do
not compromise the week.
- E-mail for communication can be scs@xfel.eu and sqs@xfel.eu where all group members receive the message.
- SA2 PP-laser is ready for next week.
- SA1 PP-laser still show problems with ADC cards. CPU will be exchanged - to be discussed after this meeting.
UNSYS:
- ZZ intervention last week.
- Couple of hiccups remotely fixed.
SPB:
- Nothing to report, besides issues with PP-laser.
FXE:
- ADQ with different sampling rates issue was solved by EEE.
- Plan is to work on the Jungfrau commissioning.
MID:
- MID requests if there is planned a meeting to discuss the train-ID issue. Steve A. comments that yes, a meeting will be organized next week.
HED:
- Asks on the request for a pulse on demand test during today. F. Wolff-Fabris says this has been sent to the machine RCs and will be discussed during the
RC meeting at 10:00.
As soon as a decision is done this will be communicated.
SCS:
- Nothing to report.
SQS:
- Nothing to add apart of PP-laser issue. |
167
|
08 Oct 2020, 08:52 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 07.10.2020 |
General Announcement:
- Discussion on access rights based on FXE issue:
- The new Dachs system does not send notification that access rights will expire. Every staff member has to identify at the system when his/her rights will
expire at each location.
- Thomas T. says EuXFEL has made a request to Desy in order to modify this and currently it is at the pipeline at Desy.
- Jan G. mentions that for the labs in his responsibilities he can see the access rights are being expired soon and he manually notifies the people.
CTRL:
- Two OCDs on Tuesday night: 1. SCS motor not moving (was fixed); 2. One SPB smarACT device server crashed (fixed).
- CTRL is following up both incidents as well as a motor in HED/SA2.
DA:
- Agreement with CTRL (see last Monday's minutes) was done.
EEE:
- Nothing to reports on EDS, PLC.
- CPU for SA1 PP-laser was replaced last Monday. No issues observed since then.
VAC:
- Nothing to report.
XPD:
- No news to report.
- SA3/XGM operation this week (change of settings due to different photon energies range) is less problematic than initially thought.
Procedure has been integrated to the BKR operators who will make the adjustments at each shift change.
DET:
- Nothing special.
ITDM:
- Nothing new.
XRO:
- All ongoing, nothing to report.
LAS:
- SA1 PP-laser is running again after exchange of CPU.
- All PP-lasers running.
UNSYS:
- Tests with quad movers: helping with STAB task force.
SPB:
- SmarACT issue: communication error between local control and Karabo.
- SPB reports beam instability issue when SA3 changes, for example, the photon energy or bunch number. BKR was contacted and is aware of it.
FXE:
- Issue with Interlock yesterday: new access rights for all staff in FXE. GABS has blocked the search prevention.
Zunaira A. mentions that Frederico L., Dmitry K., Martin K. and Peter Z. can grant access to the staff members. Zunaira A. is not allowed to provide this
access grants.
MID:
- All fine.
HED:
- User experiment this week.
- The CRL1 in XTD1 had issue while moving arm in y-direction. A ticket was sent to CTRL. Contact for this ticket is Thomas Preston.
SCS:
- User beamtime this week.
SQS:
- Two color operation with magnetic chicane ongoing this week during day time, making use of the PES and XGM.
- One issue with saving digitizer settings. We open a ticket to ITDM. |
170
|
19 Oct 2020, 15:29 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 19.10.2020 |
General Announcement:
- A Vacuum issue in one of the dumps brought machine down at around 1:30AM. Only around 7AM machine could re-start running and tuning again.
- XO thanks the groups who have already sent the maintenance lists. For the remaining groups, please contact at the xo-planning@xfel.eu.
CTRL:
- Few minor OCD calls during the weekend: One relates to a camera triggering.
- Issue with SA3 vacuum loop seems to be currently fixed but needs further investigations.
- Update of Beckhoff motors is planned for this week. CTRL will get in touch with Instruments.
DA:
- Nothing special to report.
EEE:
- Nothing to report.
- Question to XO on Maintenance: Could we have a draft of tasks list before Nov. 6th?
Andreas G. answers that XO is working on it and as soon as lists are finished this will be published. In case info on partial parts is already needed, may be provided
depending on the progress.
VAC:
- SA3 vacuum loop issue was mentioned by CTRL (see above).
XPD:
- Nothing to report.
DET:
- Nothing special to report.
ITDM:
- Update will be done on DA acquisition of HED. This has been communicated with instrument personnel.
XRO:
- No bad news to report.
LAS:
- SA3 PP-laser presents small power drop in the laser beam: LAS will take care this week.
- SA1 PP-laser has a problem with ADC and stop acquiring data.
- Need to update firmware of ADC cards to get train ID's in SA3 PP-laser. This was requested to EEE. This may bring laser down again and may not be ready by Thursday. XO,
LAS and EEE will discuss offline.
UNSYS:
- Nothing to report.
SPB:
- Last week was smooth.
FXE:
- Problems with tunnels last night at around 1:00AM. More than half the shift was lost with no beam delivery: main user data was therefore not acquired.
MID:
- Nothing to report.
HED:
- Beam was lost between 1AM to 7AM. Last data has been taken between 8:00 and 9:00AM.
- During the weekend, beam drift was observed at Pop-in at the end of the tunnel. There are data and has to be analyzed.
- Discussion on implementation of Dynamic trigger will take place.
SCS:
- Not present.
SQS:
- Minor issue with Data: one data aggregator was not working properly. OCD and SQS personnel have solved it.
K. Wrona comments that too many properties overloaded the aggregator. A long term solution will be worked on. |
171
|
21 Oct 2020, 11:11 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 21.10.2020 |
General Announcement:
- XO thank the effort made by preparing the activity lists for the maintenance period. In few cases XO is contacting groups back with questions and/or comments.
- The Winter maintenance period was adjusted yesterday. The ending period is now on January 13th, 2021, instead of January 4th.
CTRL:
- CTRL is reaching conclusions on recent Beckhoff problems:
1. CTRL would like to upgrade most of installations in instruments next week (upgrade of motors software). This upgrade is in general a step forward for the PLC
connections when it is dropped (re-starting).
2. For SASE installations, CTRL will make next week during machine days.
3. The GATT will not be upgraded now – this will start during Winter maintenance period. Upgrade of GATT is a priority for CTRL (has been added to the maintenance list
submitted to XO).
DA:
- Nothing to report.
EEE:
- Intervention in SA3 PP-laser was done.
VAC:
- No new issues to report.
- Maintenance on SA3 vacuum loop: in case of problems, one has to restart in PLC level with EEE staff and may result a bit longer recovery.
Steffen H. comments it has been worked on and may be solved in the next day or so. The restart procedure is easy but lengthy. CTRL/OCD knows what to do.
XPD:
- One of vacuum pumps lost Beckhoff connection.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Not present.
FXE:
- Nothing to report.
MID:
- All is super at MID.
HED:
- All sounds great at the moment.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
172
|
26 Oct 2020, 11:18 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 26.10.2020 |
General Announcement:
-
CTRL:
- SA3 vacuum loops problems: seems to compromise some experiments. Beckhoff loops are changing states very quickly and compromises the whole system.
CTRL ask for the instruments to give info on the type of command given that makes motors change states. CTRL investigates together with EEE as more data has been produced
as compared to the past.
This problem does not happen in SA1/SA2 and may be related to the PES which will be checked this afternoon.
- Please call the new OCD numbers: 6541 and 6542. Specially on weekends it is the fast way to contact. This number is valid since last spring.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Nothing to report except of SA3 vacuum loops.
XPD:
- Not present.
DET:
- XO asks on the train-ID progress of Agipd. Monica T. says that the plan is to correct in the DAQ level during the next shutdown.
K. Wrona adds that the fixing takes long and the current investigation focus on doing in the DAQ level during January.
ITDM:
- Maxwell nodes used to login in the Maxwell will be rebooted today, causing access limitations.
XRO:
- Nothing to report.
LAS:
- Bunch pattern in demand: LAS assume XO will take care of this topic. Andreas G. says XO will take care but need to be informed on what is needed.
Tomasz J. mentions it needs some coordination between instruments and machine.
UNSYS:
- Couple of issues in SA3 undulator cell: it got stuck, mostly during machine development times by running out of range. This was corrected during the weekend.
SPB:
- Moving on with WP8. Continuing today.
FXE:
- Nothing to add.
MID:
- MID makes a general reminder on the train-ID development with AGIPD: MID expects to have a meeting to discuss a long term solution plus the test procedure for
implementation after the quick fix.
This has to be planned in advance.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- SQS has suffered during the weekend with the VAC issue.
- CTRL side can look at the PES this afternoon. |
173
|
29 Oct 2020, 08:11 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 28.10.2020 |
General Announcement:
-
CTRL:
- SA3 VAC loop - 2 approaches to find out the root cause (discussed also with EEE):
1. Thursday 8-11:00AM to test the whole loop by partitioning batches of devices. This may pinpoint the single device that cause the problem.
2. Reproduce the problem at lab level. Based on the reproducibility this approach will be continued in parallel.
DA:
- Nothing to report.
EEE:
- PLC colleagues are troubleshooting SA3 VAC loop.
- DET reported that measurements are carrying on wrong train-IDs. Tests are being made in the lab (not reproducible) but EEE is working on it.
VAC:
- SA3 VAC loop is main activity - interaction with Michaela P.
XPD:
- Group mainly busy with Photon Diag Conference this week.
- Supporting STAB and not heard any major issues in this effort.
DET:
- Report on the train-ID. This affects all the AGIPDs including the mini-half.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Preparing for next week operation.
UNSYS:
- Nothing to report.
SPB:
- Collected lot of data last few days: SPB/SFX thanks LAS and accelerator teams for support.
FXE:
- Nothing to report.
MID:
- All fine.
HED:
- All fine.
SCS:
- Issue with door interlock: Eric B. has investigated this morning. Problem seems to be with the maglocks and has been solved for now but a long term has to be worked on.
SQS:
- Nothing to report. |
174
|
03 Nov 2020, 08:39 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 02.11.2020 |
General Announcement:
- A. Galler reminds the attendance that this meeting is also an opportunity for making comments/questions on the new Covid-19 rules that start to take place.
CTRL:
- Main issue last days was related to the SA3 VAC PLC. The applied fix was left running during the weekend. So far it works faster and reacts faster for diagnostics.
CTRL is getting some statistics in order to get the final solution. If something goes wrong, please call OCD.
- No OCD calls during the weekend.
DA:
- No issues or news.
EEE:
- uTCA crate of SA2 balcony room is not reachable. A quick fix on Friday was not successful and EEE needs to power cycle or replace the whole crate.
This will interrupt timing for PP-laser and the signals sent to MPS. Meeting between EEE, lasers, instruments and XO will take place.
- (Note after meeting: the power cycle of the actual crate will be done today after lunch. If this does not helps then immediately exchange of crate will be done
afterwards.
PP-laser will be brought to a safe state and takes about 4-5 hours to restart it).
VAC:
- Nothing to report.
XPD:
- Worked at BKR with MCP detectors. XPD reported to CTRL an issue with ADC - some channels see no signal or not standard ones.
EEE knows and runs some scripts - this happens from time to time and may affect recording data.
- In general, from XGM side, Theo M. has prepared a way to switch settings to extreme changes of photon energies.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Motor problem with SA1/cell31 last week was repaired and undulator should be working fine.
SPB:
- STAB campaign last week went fine and collected data that has to be analyzed.
FXE:
- Nothing to add.
MID:
- Waiting for invitation to a meeting on the AGIPD train-ID.
HED:
- Requests the beam delivery schedule in order to know if has to schedule people for Tuesday night.
SCS:
- Not present.
SQS:
- Nothing to report. |
175
|
04 Nov 2020, 10:55 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 04.11.2020 |
General Announcement:
- XO reminds on the 4.5MHz operation: MXL tested a 12hs switching mode and this will be used along the week.
The night shift operates at 4.5 MHz and day shift at 2.25MHz to avoid sending unnecessary bunches to the dumps.
CTRL:
- Issues with MC2 motors (bug in moving, mainly in SA3). CTRL has a fix but it is hard to find a proper time slot this week to apply it. The idea is to deploy next Monday
only in SA3 first.
If issues are found beforehand, please contact CTRL.
- SQS mentions a bug in the mirror observed yesterday which relates to the above described issue. SQS has to observe it to verify it moves all the way and then one can
wait until Monday for the fix.
DA:
- Nothing to report.
EEE:
- Crate issue in SA2: it was not needed to exchange the crate. A reboot on Monday afternoon helped to bring system back.
Crate works as normal. During winter shutdown EEE will replace crate and power modules.
- MID and LAS reported issues with train-ID / timing servers. Triggers have not been delivered in 10Hz. This is in discussion with DESY.
One solution is an update of the DOOCS timing serves that do not show errors which does not interrupt timing servers. In MID this may be related to data recording.
For the PP Laser further discuss with Desy colleagues will be made. This affects only MID because the use of old version of Doocs server.
- MCP problems reported by XPD: mismatch on the firmware version and EEE plans to upgrade this week.
- Preparation for the winter maintenance is ongoing and installations can start on 23/11.
VAC:
- VAC would like to thank EEE and CTRL for the SA3 loop vacuum fix.
XPD:
- XPD found another issue with MCP: wrong assigned triggers. A ticket will be created.
DET:
- Update of MID AGIPD is still an issue but this is not critical at the moment.
- CTRL has accessed the LPD detector issue.
ITDM:
- K. Wrona informs ITDM has the reservation of Maxwell for data analysis during next two weeks.
The user groups have to be informed that one has to use Maxwell reservation to quick analyze data.
XRO:
- Nothing to report.
LAS:
- SA1 SPB PP-laser had to increase energy and is on work.
- SA2 PP-laser missing trigger from around 7PM yesterday. Laser went OFF due to this and luckily LAS was watching it and was able to switch ON quickly. LAS is in touch
with EEE and Desy.
A fallback solution could be to generate triggers by hardware but there is no progress from Desy side. LAS will prepare specifications for this module.
UNSYS:
- Small hiccup in one SA2 undulator that was remotely corrected.
SPB:
- Trying to align beam: SPB sees elliptical rings. Andreas G. suggests if this continues please communicate to the PRC.
FXE:
- No further information on issues with the LPD.
MID:
- Tomorrow it takes place the AGIPD train-ID meeting.
HED:
- Vacuum valves in differential pumping system were strangely closed on Friday 2AM. HED discovered this only yesterday. Pressure slowly went up. HED will further
investigate.
SCS:
- Not present.
SQS:
- 4.5MHz operation had gun trips due to the overheating of the dump.
F. Wolff-Fabris comments that was due to the radiation alarms on the dumps that removed beam permission by about one hour around 4AM. |
176
|
10 Nov 2020, 10:53 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 09.11.2020 |
General Announcement:
- Maintenance summary task list has been made available at the S:\ folder as well as in the XO Alfresco page.
CTRL:
- On Thursday laser1 went down: error on deployment related to fix zero train ID.
- Lasers safety deployment is fixed.
- SA3 vac loop: progress was made and new version of Karabo needs re-start this morning.
- SA3 fixed MC2 motor issues. Coordinated with instruments to do tests today.
DA:
- 1 OCD was received and solved during the weekend.
EEE:
- Announces that on tomorrow's PBS there will be a EEE presentation on the Digitizer instruments.
VAC:
- Nothing to report.
XPD:
- No issues. The switch of XGM settings at different energies went well.
DET:
- Nothing to report.
ITDM:
- Meeting with MID on train-ID is set. There is a mitigation error plan which will be implemented.
XRO:
- Nothing to report. Minor issues on CTRLs but are not stoppers.
LAS:
- PP-laser for SPB run was interrupted for 45 minutes.
- Issue with synchronization / motor becomes unstable, relating to motor slippage. Hardware will be replaced during the shutdown.
UNSYS:
- Nothing to report.
SPB:
- Finished user experiment: went OK but machine not very stable. Intensity and pointing very unstable. After beam losses timing came quite different.
Data for pointing quantification to be analyzed.
FXE:
- Unstable beam the whole week.
- Crosstalk between SA3 and SA1: sometimes BKR could fix/work on it, sometimes not. This happens when FXE runs at 150 bunches and SA3 varying 1-100.
SA1 intensity goes down by factor 2.
MID:
- DET and ITDM contacted during the weekend.
HED:
- Not present.
SCS:
- Not present.
SQS:
- Crosstalk SA1/SA3: night shift crew said it is not a problem for next week if run without Interleaved. |
177
|
11 Nov 2020, 10:49 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 11.11.2020 |
General Announcement:
- XO is in contact with some groups regarding the winter shutdown, in order to get additional information or solve potential conflicts.
If there are any question, please contact us in this meeting or via e-mail.
- Steffen adds on the idea of using FXE for initial deployment of updated Karabo version. It will take place after 23/Nov.
- Jan asks how XO needs to be informed for tasks that groups deal with each other. A. Galler says XPD is on the list for setting a date to discuss this.
Groups are encouraged to communicate with other support groups beforehand.
CTRL:
- Misbehaving timing server at MID yesterday was initially solved by EEE. EEE replaced the hardware of the timing system itself but problem continued.
System seems now to be healthy but from time to time there are small timing interruptions. Investigation is ongoing and will be discussed in the digitizer meeting of Data
department.
- One CTRL ticket on Doocs device at SCS: it was a problem with custom installation. CTRL asks to be informed in case groups have custom installations. This may pose a
risk when deploying new updates.
DA:
- AGIPD train-ID issue: a small workaround until shutdown is in place. One has to set the individual channel and seems to be now working. Next intervention to take place
during Winter maintenance time.
EEE:
- Nothing else to report aside of the MID timing system.
VAC:
- Nothing to report.
XPD:
- SA3 XGM was switched to Ne as requested by instruments.
DET:
- Preparation for activities on AGIPD at SPB/SFX during the winter time is ongoing.
ITDM:
- Pre-announcement: at the start of the maintenance, ITDM has activity on the computer cluster. A separate announcement will be sent.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- No issues to report.
- Intervention date of AGIPD: SPB requires due to highest priority work that between 16/11-18/12 no interruptions of electrical power or the need of EQP may happen.
FXE:
- Crosstalk SA1/SA3 was not seen yesterday as SQS used only 1 bunch.
MID:
- Nothing to add besides what CTRL and EEE have mentioned.
HED:
- Next week HED does not use X-ray beam: commissioning of the high power laser and test of pulse on demand are planned. A. Galler says it pulse on demand was fixed by
CTRLs and can be used.
- Question on use of ADCs: Bruno says some trains will not be recorder or be repeated. This may be related to the timing issue like in MID.
But data can be acquired. It is under investigation accordingly to Bruno, with CTRL, DA, ITDM.
SCS:
- Not present.
SQS:
- Nothing to report. |
178
|
16 Nov 2020, 13:14 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 16.11.2020 |
General Announcement:
-
CTRL:
- Further improvements on the SA3 vacuum loop will continue during the shutdown.
DA:
- No issues to report.
EEE:
- 1 OCD call from SCS call related to PLC: hardware issue.
- Fast electronics side: MID timing system problem is still present. Frequency that this issue happens is reduced.
- For the Winter maintenance, EEE will ship some crates to Poland. Due to Covid-19 situation, only crates with non-critical changes for next run will be sent. EEE is in
closed contacts with instruments.
VAC:
- 1 OCD during the weekend was related to GATT and promptly solved.
XPD:
- Nothing to report.
DET:
- Last Friday problem with offline jobs was solved (it had to be followed by IT).
- SPB activity for AGIPD baseline intervention/correction to start today.
ITDM:
- Maintenance on the offline storage will happen next Monday 23.11. An e-mail was sent this morning. The system will be OFF and all data will not be accessible that day.
- ITDM will also start the maintenance of online cluster on 23.11. They will not be available until the end of maintenance (about 3 days).
XRO:
- Nothing to report.
LAS:
- OCD calls from SCS: 1. observed crystal degradation in SCS side; 2. laser pattern server could not be changed (LAS forward the message to CTRL).
- All other PP-lasers are fine.
UNSYS:
- BKR tested air coils in the last SA2 undulators cells for performing work on Wednesday for photon beam alignment (there was a problem with Cell37/SA2 not responding).
SPB:
- Instrument and beam delivery were running successfully last week;
- This week SPB starts working with AGIPD: one aim to work as fast as possible to finish before Christmas break.
FXE:
- D3 is currently preparing the tests.
MID:
- Continues to use beam this week for detector tests.
HED:
- Tested AGIPD during last weekend. Currently HED is changing setup to use High Intensity laser this week.
SCS:
- Not present.
SQS:
- Nothing to report. |
179
|
18 Nov 2020, 11:23 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 18.11.2020 |
General Announcement:
- Updates on the maintenance schedule will be made available in Alfresco. Guidelines in how to integrate calendars will be sent to readiness mailing list, for example.
- Please feel free to contact XO by to share any thoughts related to the Maintenance.
CTRL:
- Successful tests of HED pulser were done on Monday;
- CTRL is preparing Karabo 2.10 deployment: a schedule will be sent around per e-mail. FXE starts on 25/11. Remaining areas take place on Tue/Wed/Thu of week 49.
SPB will be discussed later due to the AGIPD work.
DA:
- DA has received two messages from CTRL related to Karabo and data analysis deployment: first about deployment and second about bridge.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- XPD is involved on MID tests this week;
- Shimadzu camera tests/development is cancelled due to AGIPD intervention at SPB.
DET:
- Nothing to report.
ITDM:
- Planning for next week maintenance and working with AGIPD/SPB intervention.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- A problem with SA2/cell37 coupler is found and will be fixed during shutdown.
SPB:
- Work on AGIPD is going OK. It has been found the expected minor issues. SPB expects AGIPD experts on site.
FXE:
- Nothing to report.
MID:
- Fast-ADC issue is under investigations together with EEE/Bruno F.
HED:
- Nothing to report.
SCS:
- Not present.
SQS:
- Nothing to report. |
180
|
23 Nov 2020, 11:33 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 23.11.2020 |
General Announcement:
- An e-mail to the readiness and stgl mailing lists was sent regarding the calendar share and updated task list. Please contact xo-planning@xfel.eu in case of any issue.
- Desy has made mandatory for the tunnel works the use a community mask all the time and a FFP2 mask if distance between two persons is less than 1.5m.
CTRL:
- CTRL sent requests to the DA colleagues. S. Hauf asks the colleagues to take a look and have a discussion afterwards.
- Karabo deployment on FXE will take place on Wednesday.
M. Turcato says one DET person must be back to work on that day to switch Detector off and on by again: Wed afternoon or Thursday morning it shall be OK.
DA:
- Nothing to report.
EEE:
- PLCs will start shutdown with planned activities on today and tomorrow in SA3 instruments.
VAC:
- Not present.
XPD:
- Plan for XGM/SCS is to keep it running (under vacuum). XPD reminds SCS people to keep XGM on vacuum.
DET:
- AGIPD intervention: is progressing quite well and advancing.
A. Violante says it is progressing faster than expected and many small issues were solved during the weekend; first quadrant was already re-mounted and tested.
By the end of this week an update on time planning will be made.
ITDM:
- Today ITDM starts the update of offline storage system (unavailable all day).
- The update on the online cluster system will start today in SA1 and then SA2 and SA3. It will not be available until Wednesday.
XRO:
- Working on the winter shutdown.
LAS:
- Carrying on planned maintenance work.
UNSYS:
- Not present.
SPB:
- AGIPD intervention is going well and is under control.
FXE:
- Cabling works starting today in the hutch.
- A. Galler asks if calibration of LPD was done. D. Khakhulin mentions that the raw data was used and calibration constants were not used.
M. Turcato says one has to discuss together with calibration team.
MID:
- All fine.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
181
|
25 Nov 2020, 10:14 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 25.11.2020 |
General Announcement:
- There is no UOM or XO Int. Meeting next Friday 27.11. We will continue with XO Int. Meeting next Monday 30.11.
CTRL:
- Karabo Deployment in FXE takes place today. Schedule for full Karabo deployment was sent to the Readiness mailing list.
- Two SA2/SA3 Beckhoff devices topics will be updated today (5-10 minutes downtime).
DA:
- No issues or announcement to report.
EEE:
- PLC side: tunnel updates to be started with CTRL.
- Change requests also to start today in a couple of instruments.
VAC:
- Not present.
XPD:
- PES/MEA: there are individual tasks ongoing.
DET:
- Nothing major to report. AGIPD intervention is goes faster than planned.
ITDM:
- Offline storage update was done and is available since Tuesday.
- Online cluster update has started: SA1 already available since Tuesday. Today ITDM will finish SA2/SA3. Crosscheck and start DAQ will follow.
XRO:
- Not present.
LAS:
- Nothing special to report.
UNSYS:
- Not present.
SPB:
- AGIPD work is ongoing and fine.
FXE:
- Wille company is installing cables in the hutch.
- Issue with craning on SA3 / SXP activities above FXE area: FXE asks for coordination to avoid interrupting activities.
Gerd W. says it shall not be a problem as crane will be used for few period. Martin K. will inform his phone number for arranging craning times.
MID:
- All fine at MID.
HED:
- All fine at HED.
SCS:
- Shutdown activities ongoing.
- Situation of JJ X-ray will be clarified by SCS staff.
SQS:
- Nothing to report. |
183
|
02 Dec 2020, 10:21 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 02.12.2020 |
General Announcement:
- Laser interlocks activities in SA2 are delayed by two days. Therefore, activity in SA1 will start later.
- Gerd W. reports the cooling water in SA3 was refilled; air conditionings are running and maintenance on them are ongoing;
next week temperature and humidity stability in other SASEs will be working; water leakage in SCS (yesterday) has been taken care by TS (crates are not damaged).
CTRL:
- Currently working in SASE2 deployment. SASE1 deployment happened yesterday and went relatively smooth.
- Fast ADC cards in lasers stop working. CTRL will need to look with EEE.
DA:
- Works on updates are progressing/done.
EEE:
- MicroTCA SA1 update is done. Next week continues with SASE3 update.
- Cabling installation ongoing in FXE. It will be followed by SQS, SCS and MID.
- SA2 and SA1 motion loops updates to be done this week.
- Fast ADC issue with Lasers: Bruno F. says the installed update is the last version and Karabo has also to be updated to its last version also. CTRL and EEE will discuss
together.
VAC:
- Maintenance activities are going as expected, some tasks are ahead of schedule.
XPD:
- SA3 PES is in the lab and being replaced parts. Plan is to test it today. XPD will discuss with VAC to install back this week or next week.
- XPD received a beamtime shift in PetraIII for the PES. This influences XPD schedule as many people are involved. Beamtime starts this Friday until next Wednesday.
- DAQ tests on Tuesday morning are scheduled. XPD will talk to ITDM to postpone to after the mentioned beamtime shift.
DET:
- Nothing major to report. AGIPD Intervention: so far, so good.
ITDM:
- Updates ongoing.
XRO:
- Not present.
LAS:
- ADC cards problems are communicated to CTRL.
- Observed issues with trigger delay signal and it points to issues with recent update. Currently the lasers use local timing. CTRL will investigate if this can be the
cause.
UNSYS:
- Not present.
SPB:
- Doing OK with AGIPD intervention: expected to be ready with final tests at start of the next week.
FXE:
- Cabling installation ongoing; Today there is the crane activity for SXP.
MID:
- All fine at MID.
HED:
- Everything is fine.
SCS:
- Not present.
SQS:
- Nothing to report. |