ID |
Date |
Author |
Location |
Severity |
Subject |
273
|
01 Nov 2021, 10:10 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 01.11.2021 |
General Announcement:
- XO is currently setting scheduling meetings with the groups for planning and cross-checking the preparation for the Winter maintenance 2021. Meetings will take place between this and next week.
CTRL:
- Nothing to report.
DA:
- Announcement on the online calibration: for SPB it will be upgraded to Karabo 2.11. Today is preparation and later this week it will be deployed for the user beamtime.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Together with CTRL and DA and HED/MID instrument: solving hardware problem of online cluster - one has to swap locations of SA3 and SA1. This is ongoing right now.
XRO:
- Nothing to report.
LAS:
- Water leak in SA1. TS is aware and fixing it. The PP-laser was not switched OFF.
UNSYS:
- All is good; The seeding crystal in SA2 is on the right position for the self-seeding. Support will be available for helping the expert.
SPB:
- Nothing to report; Interesting data was acquired along the week and analysis will take few weeks.
FXE:
- Nothing to add.
MID:
- All fine.
HED:
- User experiment;
- Yesterday after midnight beam intensity dropped down from 1mJ down to 300uJ and BKR did not managed to recovered it.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
274
|
03 Nov 2021, 10:12 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 03.11.2021 |
General Announcement:
-
CTRL:
- Nothing major to report.
DA:
- Nothing to report.
EEE:
- Nothing to report.
VAC:
- Not present.
XPD:
- Nothing to report.
DET:
- Issue concerning AGIPD cooling system at MID being not very effective: there is ice formation which stopped the flow of water. This already happened once but it is not so often.
Fix consists in flushing with dry air. Later today Detector will be running again.
ITDM:
- Fire incident yesterday on the labs (ITDM/EEE) was caused by a broken cooling system; the lab cooling water did not work for hours and let to problems with GPFS system in the lab as well as with servers
(same type of racks as the ones in the rack room).
- ITDM suggests the handling of such alarms have to be improved.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Issues with SA3/cell17: Phase shifter can not move. After rebooting PLC system one had to power cycle leading to lose half the SA3 undulator system.
A ZZ was done to recover downstream cells but finally the encoder of the PS motor is found damaged and need replacement. A request for access is created and is needed 4 persons for 2-3 hours.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine apart of the mentioned Detector issue.
HED:
- HED asks if the last Sunday's problem with machine (drop in intensity) is understood or if it may happen again. XO says until yesterday this was not fully understood and will enquire again with BKR
colleagues.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
275
|
08 Nov 2021, 10:07 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 08.11.2021 |
General Announcement:
- During the weekend there was a cryo plant failure which caused more than 30 hours downtime. This was due to the cold compressor failure on Friday evening.
Machine work hard to get back in operation and Sunday by 06:00AM beam was available to the experiments.
- Chris Y. asks if FXE had used the PAM. It seems not to be the case due to the beam downtime. C. Youngman will contact Yifeng J.
CTRL:
- Nothing to report.
DA:
- DA is interested on the MID migration problem (discussion is following below).
EEE:
- Problem with SQS digitizer regarding a drift when looking into pulse laser pattern. Problem seems to be coming from the source; to be further investigated;
- Beckhoff PLC at SCS had a problem and will be investigated today.
VAC:
- No issues to report.
XPD:
- This week XPD will measure at HED with the calorimeter.
- XPD needs confirmation from MID if Diamond detector was used at MID.
DET:
- Few OCD calls related to the AGIPD operation support.
ITDM:
- The MID issue seems not be linked to migration in a first analysis. It may relate with calibration. Queue was busy as many experiments taking data simultaneously.
By the other side, there were no indications of calibration being used. Today the migration was running normally.
XRO:
- Nothing to report.
LAS:
- Not present.
UNSYS:
- Another Phase shifter (SA3/cell20) presented a problem like the cell17. Waiting for getting access to fix and investigate the encoders. ZZ is scheduled for this morning.
SPB:
- Problem with AC unit at the laser hutch. TS was called onsite during the weekend and it was solved.
FXE:
- Nothing to report.
MID:
- Issue with data processing last night: if it looks good for IT and DA, then MID is relieved.
- Diamond detector use: looks reasonable and data was taken. The final analysis has still to be seen.
- MID refers that it was positively impressed how fast the seeding was brought back after the cryo failure.
HED:
- Nothing to report.
SCS:
- EEE got a call about scanner-y motor. A problem with cable was found and it was exchanged on Sunday morning. Since then it appears to be solved.
SQS:
- Nothing to report. |
276
|
10 Nov 2021, 10:16 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 10.11.2021 |
General Announcement:
-
CTRL:
- Issue with mc2 motor (pop-up issue for step-up command): it has been identified and fixed. New version will be deployed in the shutdown. Workaround until then is to stop and restart device.
DA:
- On the slow migration/calibration at MID: it is understood as overload of cluster nodes responsible for migration; mitigated by adding two more nodes. Calibration services shall run better from now.
EEE:
- Drifts with SQS digitizers: yesterday Bruno F. was working on it and it shall be fixed now.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing else to add apart of DA report - still more investigation is ongoing.
XRO:
- Nothing to report.
LAS:
- All fine with lasers.
UNSYS:
- In SA3: two encoders that were exchanged (cell#17 and cell#20) on Monday's ZZ.
- Another PS (SA3/cell21) stopped working. Motor has to be replaced in a next ZZ (request placed).
Suren K. also says that the whole intersection is not working (not able to recover after restart the system - Quad stays in position for operation).
- SA1/cell14 a coupler of the undulator terminal was replaced.
SPB:
- AGIPD issue also observed last week and is under investigation. DET is aware and under investigation.
FXE:
- Nothing to report.
MID:
- Issue with calibration: MID would like to test it in order to close the review.
HED:
- Running user experiment this week.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
278
|
15 Nov 2021, 09:57 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 15.06.2021 |
General Announcement:
-
CTRL:
- Nothing to report.
DA:
- Interruption during the weekend on the automatic data processing: was disrupted on Saturday night (proposal that was finished and treated as same priority). A change of priorities had to take place for
HED having the data analyzed.
EEE:
- Faulty sensor at ATD broker - EEE will take a look.
VAC:
- No issues to report.
XPD:
- Nothing to report for XPD.
- Jan G. reports the machine overall performance was quite good without major incidents. Several issues were reported on the Operation e-log concerning the calibration pipeline and DAQ.
DET:
- Nothing to report.
ITDM:
- On the prioritization (refers to DA): major reorganization of priorities took place. This results in the fact jobs that are submitted to the partition, (many jobs for a current beamtime) are killed.
If instrument has beamtime please request reservations for the proper amount of jobs.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Planned access to exchange motor of PS21/SA3 this morning.
SPB:
- AGIPD data issues were seen and is in contact to DET and DOC. Issue relates to memory modules not sending data from time to time. Jolanta S.-D. says it can happen for 1-2 modules a train may miss and the
DAQ will discard the data.
This is not unusual and one has to look at the frequency which happens - there is nothing wrong with Detector; data looks as usual. Further discussions will take place.
FXE:
- Nothing to report. User beamtime seems to have run well.
MID:
- Restriction of priorities: MID jobs went through. Experiment jobs have priority and for this week one has to put back in priorities.
- K. Wrona says a system will prioritize the user beamtimes for each week, getting out of the system automatically.
HED:
- Two user experiments using AGIPD went nicely but a bit stuck due to the priorities list (slow data processing).
- Beam instabilities occasionally: appearing and disappearing from time to time (maybe related to mirrors). It was reported to the stability task force.
Harald Sinn mentions about a crosstalk between SA2 with SA3. This HED case will be further studied.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
279
|
17 Nov 2021, 11:20 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 17.11.2021 |
General Announcement:
- XO will publish on Friday the full maintenance task list. Everyone is invited to also look at the Infrastructure calendar which contains main dates on this topic.
Please contact XO in case you have questions. The task list will be published via e-mail containing also the file location in an Alfresco folder.
- A power glitch was observed today at around 07:40AM. Machine was not affected. Please check your systems.
- Jan G. asks on Zoom operation issues seen yesterday (link of PBS meeting was shared with non-XFEL staff).
K. Wrona says that this particular case the link was posted at the website including the password and everyone from the world could access. ITDM will prepare an e-mail describing good working practices.
CTRL:
- No major updates.
DA:
- No major updates.
EEE:
- Nothing to report.
VAC:
- Nothing to report.
XPD:
- Systems were checked after this morning's power glitch and they seem fine.
- The transmissive imager (Diamond screens) in SA1 and SA2 are still in. To remove them the instruments shall contact Andreas Koch.
DET:
- Nothing to report.
ITDM:
- ITDM expects this week a lot of data for the user experiments that may overflow systems. ITDM is preparing contingency space for it.
XRO:
- Nothing to report.
LAS:
- No effects were seen from the power glitch; lasers are running.
UNSYS:
- Last Monday the ZZ access in SA3 (cell#22) was successful and the Phase Shifter motor was exchanged. Currently discussions are taking place to evaluate counter-measures for the case it is related to
radiation damage.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
280
|
22 Nov 2021, 10:13 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 22.11.2021 |
General Announcement:
- On Accessing EuXFEL premises at Schenefeld: it is no longer necessary to present the test result at the Gate; it is necessary to present the QR code for vaccination/recovery as well as the DACHS card at
the new terminal. For non-vaccinated people an official test certificate is mandatory.
CTRL:
- No major updates this time.
DA:
- Nothing special to report.
EEE:
- On Saturday EEE had to exchange a fuse at a MID motor (motor burned).
VAC:
- Nothing to report.
XPD:
- Nothing to report.
- A. Galler reported that for SA2 the interlock conditions for the upstream filters in SA2 are temporarily lifted allowing to go to 350 bunches (limit is 300 bunches). This is an exception that is valid
until tomorrow.
DET:
- Nothing to report; IT group is working to increase space for data.
ITDM:
- It collected 2.2 Petabytes of raw data for one user (MID) plus the same amount of calibrated data and there is still one day of beam delivery until tomorrow. Also SPB produced about another 1 Petabyte.
- If such requirements are needed in future please communicate to ITDM well in advance, largely more than one week before.
- A. Madsen says that the reason such big amount of data relates to an exceptionally well run, so it is difficult to predict how many data will be collected. K. Wrona says the communication went very well
this week. M. Turcato also says that when many detectors are running in parallel.
- Further discussions on this topic will take place.
XRO:
- Nothing to report; next Monday XRO plan to have the ramp down activities to have the current positions for the recovery.
LAS:
- From tomorrow on only SA1 will use the PP-laser.
UNSYS:
- Nothing to report.
SPB:
- Issue with calibration pipeline with AGIPD is under investigation (performance of the pipeline is not the expected one). P. Schmidt says that used calibration can not be fully supported on the weekend so
a fix can't be delivered in short time. In such cases it is better to use the old version.
- K. Wrona say one can plan for the maximum expected of acquired data but needs to have full resources for it. The current week was an exception. If one needs similar conditions like the current week it is
a big challenge.
- M. Sikorski completes that about 2-3 weeks before they can have an idea of how much data will be generated.
FXE:
- Nothing to report. M. Biednov says that they work at 9.3 keV that machine performance was poor and that would need to retune on Tuesday (about 1.5mJ).
MID:
- Nothing to report; all fine at MID.
HED:
- Nothing to report.
SCS:
- Experiment running so far good with minor issues.
SQS:
- Nothing to report. |
281
|
24 Nov 2021, 09:56 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 24.11.2021 |
General Announcement:
- Discussions on 3G rules access controlling are ongoing by the Covid task force. However, there is nothing else apart of what was recently announced.
- Next Monday and Tuesday (29-30.11) is dedicated to the ramp-down of the accelerator and photon systems and from Wednesday 01.12 it is the start of the Maintenance period.
CTRL:
- Nothing to report.
DA:
- P. Schmidt informs that instruments noticed early in the morning some issues with calibration.
DA is catching up to make it stable again (it relates to the new measure/state for the expected high data on this week). Please contact DOC if issues can't be solved by staff instrument.
EEE:
- Nothing to report.
VAC:
- Not present.
XPD:
- SA2 Filter interlock is put back (EPS limit of 300 bunches).
DET:
- DSSC cooling system issue (reported at SCS operation e-log): chiller had an alarm due to low level of coolant. The detector was stopped; chiller was refilled and brought back to operation; it seems to be
fine.
DET is investigating why the level had dropped. SCS is also investigating if it relates to a leak or low temperatures. Monica T. is in contact with Jan-Torben D.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Nothing to report.
SPB:
- Confirm that data will be taken also during the night.
FXE:
- Nothing to report; happy with beam performance.
MID:
- Organizing to get a training for the compressor to be able to switch it off when this is not needed.
HED:
- User onsite at the end of the week.
SCS:
- PLC-1 crashed on Monday taking time away for the experiments. A fan went off. Patrick G. says this is recognized that the fan went off and some cooling had to be switched down.
The maintenance of cooling systems is done by TS and it relates to a closed/sealed rack that can't exchange heat.
A solution permitting to spot/alarm when issues with cooling system appears in the racks must be put in place.
SQS:
- Nothing to report. Users are expected this week. |
282
|
29 Nov 2021, 11:50 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 29.11.2021 |
General Announcement:
- Today and tomorrow are the two last days of operation with ramp down of tunnels components. On Wednesday many activities are planned for the start of the maintenance period.
- Sandra Plett comments there are 1000 packages o Self-tests at EuXFEL with 5 tests each. Sandra P. also mentions that FFP2 masks can be taken from reception (few amounts) or warehouse (larger amounts).
CTRL:
- Karabo deployment will start accordingly to the maintenance planning.
DA:
- Nothing to report.
EEE:
- Preparing for maintenance tasks.
VAC:
- Nothing to report.
XPD:
- Naresh K. mentions the issue with emergency light test carried by TS. HED planned to make measurements with a dark room (all lights OFF).
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Nothing to report.
LAS:
- Nothing to report.
UNSYS:
- Not present.
SPB:
- Nothing to report.
FXE:
- Cooling issue in cw49 (opening rack doors): A. Galler says that for the experiments does not need to open racks. On 02.12 XO will inform if the cooling backup system for 20C water works.
MID:
- Training to operate air compression in the Hall was done. Normal state for MID compressor will be OFF and only switched ON when this is needed.
- Jan-Torben D. also asks for the training. This has to be scheduled with TS and XO can assist with it.
HED:
- Emergency light problem occurring on Friday: A. Galler says that this test is being done remotely every week on Mondays; The referred test was a maintenance task and TS was not aware of the presence of
users.
SCS:
- Nothing to report.
SQS:
- Re-arrangement of the beamline starting today.
- SQS is also interested in the training for compressors also. |
283
|
01 Dec 2021, 09:49 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 01.12.2021 |
General Announcement:
- Info that today the 20C cooling water will be switched to the towers was sent. Please also open rack doors to help until further notice. Tomorrow we will understand if cooling water tower is working and
if not, then A. Block and XO will communicate.
- Fire alarm happened this morning but not affecting people working at XHQ.
- Steffen H. says that DOC is not operational until middle January.
- Next Friday there won't be an UOM.
CTRL:
- Start with deployment today.
DA:
- Nothing to report.
EEE:
- Fully busy with preparation works and starting updates today.
VAC:
- Maintenance period activities been prepared. Tunnels are opened - if staff access rights are expired, please contact M. Dommach.
XPD:
- Not present.
DET:
- Nothing to report.
ITDM:
- Starting the maintenance activities today; On-line processes are unavailable until end of the week.
XRO:
- Last Monday: ramp down activities (stored motor positions for instruments with proper beam trajectories; XPD also worked with MCP detectors for all 3 SASEs (not done in SCS);
- XRO activities will be reported during this meeting series.
- Calibration SASE2 mirrors FD proposal was submitted for starting 2022.
- One PC at BKR (middle one) is extremely slow and shall be investigated during the maintenance time.
LAS:
- Maintenance on the PP-lasers are started.
UNSYS:
- Not present.
SPB:
- Not present.
FXE:
- Nothing to report.
MID:
- All fine at MID. It will investigate the water leak that happened this morning with chiller.
HED:
- Preparing XTD6 activity (exchange screen in monochromator) next week.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
284
|
06 Dec 2021, 11:54 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 06.12.2021 |
General Announcement:
- Cooling water situation at XHQ: initially there is no cooling water until possibly 13.12. To the best of XO knowledge all groups are informed. If this is not the case please contact XO.
As soon as water comes back to normal an e-mail will be sent.
- Jorg H. asks on the safety trainings for employees (relates to the sexual harassment traning from an e-mail of Sven M. / Desy this morning).
Sandra Plett comments that this is not mandatory to XFEL colleagues but it will be clarified today via an e-mail. The Basic training and hygiene trainings are the ones mandatory to all the staff.
CTRL:
- Last week Karabo deployment done except of the Lasers (to be done this week). If any issue is found please contact CTRL.
- Control network maintenance: control server maintenance takes place on Wednesday between 07:00 and 09:00.
DA:
- Online data analysis service is also down due to maintenance tasks.
EEE:
- Start this week with tunnel loop updates.
- Ongoing HED installation cables; also cabling work at SPB ongoing.
VAC:
- Continuing with shutdown activities (ongoing accordingly to the schedule).
- Power cut in XTD2 last Friday is not yet understood.
- Broken power supply that affected fire extinguisher system was resolved.
XPD:
- Continuing with maintenance tasks: XTD2 differential pumping systems; transport of hard X-ray PES taking place tomorrow at XTD2.
DET:
- Nothing to report.
ITDM:
- Wednesday early 7:00 to 9:00 no control network due to switches upgrade.
- Today and tomorrow online storage system will be upgraded in offline mode by being disconnected from the clients (therefore it takes shorter time).
- Planned for today and tomorrow the exchange of switches to Desy.
XRO:
- Continuing the list of activities: due to the stop of cooling water the preparation of the re-installation of the bender in XTD2 is delayed.
Lab is currently at 28C and when temperature is stable activities will resume. Only then will evaluate if intervention will be feasible or not.
- Tasks with EEE and motors will start next days.
LAS:
- Confusion with cooling water on lab area that was lost around 15:00 on Friday. Apart of that all is fine. XO will look into it when cooling water is back.
UNSYS:
- Two Apple-X undulators are in the shaft building for connecting vacuum pipes.
- Undulator SA#1-cell#33 to be moved to cell#2. Collimator/Pb shield was moved to downstream of Diagnostic Undulator.
SPB:
- Installations of hardware planned for this week.
- Working with EEE and others to get hardware moved to the new rack room.
- Needs to negotiate to keep vacuum running.
FXE:
- Not present.
MID:
- All fine at MID with maintenance works.
HED:
- Laser interlock update at HED: laser interlock not functional this week;
- Tunnel activities by HED is done.
SCS:
- Activities are ongoing accordingly to the plan.
- Experimental chamber is moving to SCS hutch today.
- Contacted TS in order to run the air flow in the laser run (due to SXP installations): discussed and solved with Alexander K. (TS). About the rack room the 20C is running but rack rooms must be kept
open.
- Alarm of the Draeger cabinets appeared this morning but not yet understood.
SQS:
- Nothing to report. |
285
|
08 Dec 2021, 11:02 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 08.12.2021 |
General Announcement:
- The temperatures in the Hall are warming and accordingly to Tobias C. this is related to cooling water as ventilation is also affected.
- A. Galler reminds on an e-mail sent yesterday related to check-in at XFEL today: one has to show vaccination status at the gate. For next week likely one can register the Covid status at the badge reader
and further communication will be sent.
- On next Saturday 11.Dec tunnels between Osdorf-Born and XHEXP1 will also be closed in the context of the fire extinguisher maintenance.
It is also reminded that all Schenefeld campus buildings are inaccessible and only SRP, TS and external company are allowed in the site.
CTRL:
- After the control server network update, Karabo is up again in all the topics.
DA:
- Nothing to report also including calibration team.
EEE:
- All maintenance tasks progressing in tunnels and instruments.
VAC:
- Not present due to tunnel works.
XPD:
- Progress yesterday was done by bringing hard Xray PES in XTD2 which was installed in the beamline with help from VAC and MEA groups
- Loop update with EEE. Bernard B. says SA3 motion2 PLC will be updated/finished today.
DET:
- Nothing to report.
ITDM:
- Commissioning proposals: interested people got an e-mail on the closing/reviewing of the current proposals. ITDM will eventually close it this week.
- Request for changes on new proposals have to be sent today to ITDM, before its creation. After being created, remains valid for the next half year. New proposals will be available for use from Monday.
- Maintenance tasks: online storage was finished yesterday evening. Currently everything online is available again. DAQ will be brought up today and cross-checked specially for SA1 where servers were
exchanged (new clean installation).
- Exchange of switches for transferring data between Desy and Schenefeld was done.
- Early morning network maintenance: just two routers need further investigation by NOC (using old firmware). Action shall be unnoticed as there is a redundant system.
XRO:
- Proceeding with maintenance planned tasks.
- Reminder on the delayed installation of the XTD2 new bender for SA1 because of problems with cooling/ air conditioning at the labs.
- XTD9 distribution mirror M3 for SA1: XRO noticed the soft limits were changed and position of one mirror was different.
Changes in soft limits may be related to power cut; XRO has changed back to the proper values but this has to be checked again during re-start of operation.
LAS:
- Nothing to report.
UNSYS:
- SA1 cell#33 undulator and rack have been shifted to cell#2.
- Apple-X small hiccups but optimistic that will be good for next week.
SPB:
- Nothing different from Monday.
FXE:
- Usual activities ongoing; tests with robot software library and middle layer devices for movements to be done today.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Nothing to report. All fine with transportation activities.
SQS:
- Nothing to report. |
286
|
13 Dec 2021, 10:51 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 13.12.2021 |
General Announcement:
- The cooling water various types shall be working/restored and it seems that temperatures are back to normal. In case of finding deviations in temperatures let TS or XO know. Racks and racks rooms can in
principle be closed.
- Tobias H. comments on the open rack doors: in general if racks have to be opened, please keep in mind the corridors in between the labs are fire safety doors and this has to be discussed with SRP or
Michael Prollius prior taking place.
The comment serves as a precaution for the future.
- An e-mail from SRP will be sent concerning disclosing the vaccination status and registration at the entrance gate.
CTRL:
- Upgrade/migration of all tunnel cameras to Aravis was concluded in tunnels.
DA:
- Nothing to report.
EEE:
- All runs smoothly at the moment.
VAC:
- Maintenance activities going accordingly with the plan.
XPD:
- Proceeding with shutdown tasks. Today alignment activities and vacuum activities at XTD2.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- This week resume the work in lab after restoring cooling water.
- XRO reminds the need to resume discussion with the ramp-up activities to divide commissioning days - a plan has to be done due to the high number of updated devices.
A date for recommissioning of these devices shall be fixed. XO will discuss with XRO.
LAS:
- Restarting the lasers and proceeding maintenance.
UNSYS:
- Plan to install Apple-X from today on.
SPB:
- Installing granites in experimental hutch.
- Intervention in optics hutch today: please refrain to access it for staff other than the ones performing task.
FXE:
- Access XTD2 to check CRL; commissioning motors in hutch.
MID:
- All fine at MID.
HED:
- Laser interlock upgrade last week was successful with Klaus Witt.
SCS:
- Activities are going accordingly to the plan.
SQS:
- Nothing to report. |
287
|
15 Dec 2021, 10:26 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 15.12.2021 |
General Announcement:
- Severe power cut affected Schenefeld campus yesterday: it seems to be associated to human error but investigations are still ongoing. As soon as information will be available, it will be distributed via
e-mail either from TS or from XO.
- Regarding access regulations coming into effect Thursday evening (16.12): those who shared vaccination status with SRP will be able to enter via side gates or without showing vaccination certificates to
the gate keepers.
Without showing to SRP one can enter only via the main gate.
CTRL:
- Due to power cut some devices went to error state; a bunch reset helped to bring them up. If errors appear again please write tickets to CTRL.
DA:
- Nothing to report.
EEE:
- Issues with power cut: lost electronics in instruments; lost parts in tunnels (but recovered). Please check systems if they need to be reset.
- XGM in SA1 and SA3 were also affected.
- uTCAs are back and running after power cut.
- Work with cabling is ongoing accordingly to the maintenance planning.
VAC:
- PLC in rack room were affected and recovered with help from CTRL and EEE. So far nothing severe seems to be appeared but investigations still ongoing.
XPD:
- XGM is under investigation by Theo M.
- HIREX in SA1 has a motor not moving. Today on-site investigation will be done together with Bernard B,
DET:
- Nothing to report.
ITDM:
- Power cut: problems with office switches had to be rebooted/recovered. It got stuck during the reboot process causing problems that switches had to be manually power cycled (maybe due to a second short
power failure).
XRO:
- Nothing to report.
LAS:
- Affected by power cut; PLCs were off but running again after help from EEE.
- Seeder seems to do not suffer major issues.
UNSYS:
- Problems with electricity at XHU due to power outage were recovered later in the afternoon.
- Two Apple-X undulators were installed in SA3 yesterday.
SPB:
- Could recover the essential systems quickly enough; no big damages so far.
FXE:
- Working in tunnel this morning. Investigations related to the power outage ongoing.
MID:
- Issues were recovered with EEE.
HED:
- Could recover vacuum systems; Laser systems were turned OFF.
- SA2 gas alarm controller was damaged due to the power outage. So there is no operation with liquid nitrogen and other gases from now. TS need to discuss with HED as tasks must be performed by latest mid-
January (lasers and/or cryo activities).
SCS:
- Not present.
SQS:
- Damage in the air conditioning (electronic and physical) in room 509 due to the power outage. To be discussed between SQS and TS. |
290
|
10 Jan 2022, 09:22 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 10.01.2022 |
General Announcement:
- The "Readiness" mailing list will be discontinued and substituted by the "xo-info@xfel.eu" list. This new list is owned by XO and please use it in future for sharing information to a broad audience.
- Next News on the Covid regulations will be communicated on Wednesday 12.01 or for the next Thursday's staff meeting.
- Please take attention to Zunaira A. e-mail on the radiation safety training. It shall be done until the end of January to avoid disruption in access rights.
CTRL:
- No major updates.
DA:
- Nothing to report.
EEE:
- SASE1 VAC loop will be update tomorrow including the PES system.
VAC:
- No issues to report. Continuing with small tasks being done at XTD2 and XTD6.
XPD:
- All continued tasks are fine, mainly XTD2, XTD6 and XTD9.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Carry on with maintenance works - SASE1 and SASE2 are in vacuum;
- XTD6 PBLM diamond to be exchanged this week as well as screens.
LAS:
- Not present.
UNSYS:
- Alignment for the Apple-X to be done this week and vacuum will be closed until Thursday.
SPB:
- Small issues with things moved into new racks; being followed with EEE.
FXE:
- Continuing with tasks.
MID:
- MID looks for a CL flange with SmartAct connection. Please contact MID or Joerg H. if you have one to help MID.
HED:
- All fine at HED.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
291
|
12 Jan 2022, 10:06 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 12.01.2022 |
General Announcement:
- SEPS interlock tests are scheduled for January 21st: starting early morning from 08:00 to 16:00 across all the SASEs;
In parallel D-3 tests the interlocks in all hutches including SXP. An e-mail to instruments was sent and a reply has to be directed to Zunaira A.
- Disabling SEPS interlock yesterday night was not necessary. Further discussion with A. Galler will follow.
CTRL:
- No planned updates.
DA:
- No updates to report.
EEE:
- Yesterday the SA1 vacuum update took place.
- MPS contacted Nerea J. on the SEPS interlocks issue: Nerea J. to be included in further discussions.
VAC:
- Nothing to report.
XPD:
- Preparing for tunnel closure: finishing Hard X-ray PES tests; works in XTD9 are completed.
- Works in XTD6 finishing EPS switches and installing upgrade M-ports in MCP (crate coming on Thursday and update interlock connection).
DET:
- Nothing to report.
ITDM:
- Remind on the Wednesday maintenance on the dCache and GPFS; Access to XFEL related data from Maxwell compute cluster is not guaranteed during the intervention. Please contact ITDM in case of conflicting
with it.
XRO:
- Not present.
LAS:
- Failed to upgrade the SASE1 front ends during Winter Maintenance (company did not come due to Corona regulations).
- 8D report for the Seeder shutdown issue at the start of the maintenance period: A. Galler informs that by the end of this week it shall be ready and distributed.
- Power supply connection: Max L. asks for XO support to follow up with TS.
UNSYS:
- Apple-X undulators are aligned and currently the intersection is being aligned together with air coils. Vacuum shall also be connected permitting to close tunnel tomorrow.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- Thanks to SPB group for helping with providing a flange.
HED:
- Apart from the SEPS all is fine.
SCS:
- Not present.
SQS:
- All fine. |
292
|
17 Jan 2022, 10:41 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 17.01.2022 |
General Announcement:
- SEPS interlock tests are scheduled for this Friday January 21st; Zunaira A. needs XTD6 closed by the SEPS interlock at 09:00.
Martin D. will evaluate if it is possible to close XTD6 before Friday pending to finish the work (without closed vacuum we won't have beam in future).
- Machine is tuning SA1 and SA3 this week and XTD2/4 tunnels are opening today and tomorrow for finishing Apple-X works.
CTRL:
- Nothing to report.
DA:
- Preparing for start of operation; nothing out of ordinary.
EEE:
- Nothing to report.
VAC:
- Vacuum all is fine.
- Tunnels closures: XTD9 tomorrow afternoon; XTD6 on Friday or Thursday afternoon (see general announcement).
XPD:
- Camera library tests/migration together with CTRL (Aravis library) from today afternoon. Some cameras won't work for the update (5mins or bit more).
Plan is to do all cameras - please keep in touch with A. Koch if cameras are needed.
DET:
- Nothing major to report: preparing detectors for start of the run.
ITDM:
- Reminder on the storage system (offline side) update on 19.01: dcache won't be available for the whole day; connection between online and offline won't work also. An e-mail will be sent when activity I
done.
XRO:
- Next week it starts the photon beamline alignment. Next Friday one needs the SCS or SQS hutch interlocked over the weekend.
LAS:
- Nothing to report.
UNSYS:
- Issue with phase shifter in SA2: BKR forgot to use the right operation mode.
- Apple-X: waiting to enter tunnel today at 12:00 to install air coils and finalize hardware commissioning.
SPB:
- Controller was affected by power glitch; fix ongoing.
FXE:
- Interlock system problem was detected and under fix.
MID:
- All fine at MID.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
293
|
19 Jan 2022, 11:00 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 19.01.2022 |
General Announcement:
- Machine status: SA2 is currently lasing @ 9keV and 2.5mJ. North Branch was closed yesterday and after a quick ZZ in XTD10 this morning the BBA and tuning will resume today and tomorrow.
- X-ray delivery tables for cw05 have been sent to the instruments and we thank to all that have already sent us a feedback.
- DOC will start next Monday. From MON-FRI from 9Am to 5PM. For the next weekend the photon staff in shift work shall send a ticket or only in case of urgent/critical matters call the DRC phone. The PLCs
are providing OCDs.
CTRL:
- Bassler cameras were migrated to Aravis library.
DA:
- Nothing to report.
EEE:
- ZZ is ongoing to check a faulty switch signal in XTD10.
VAC:
- Benoit R. is currently in the tunnel on the ZZ;
- VAC systems are all fine and ready to take beam.
XPD:
- Aravis library installed and tested; all went fine.
- XTD6 ongoing tasks are under progress and may be finished already at the end of the day. If progress is made accordingly to the plan, XTD6 can be closed tomorrow.
The new M-port crate was connected yesterday (upgrade of MCP). Almost all is finished; Wolfgang F. finishes small things.
DET:
- Nothing to report.
ITDM:
- Offline storage system update scheduled for today (Dcache affected). E-mail will be send when finished by end of the day.
XRO:
- Nothing to report; all maintenance activities completed.
LAS:
- As aftermath of cooling water and air conditioning issues during maintenance: problems with misalignment of all lasers; LAS has troubles to align them.
- For SA2, LAS is confident will be fine with alignment. The SA3 is a bit behind of SA2;
- SA1 presents the biggest misalignment after T and H shock - long pump pulse. The 280kHz test is now in secondary priority. LAS will report day-by-day and is not so confident will be ready for instruments
in February.
UNSYS:
- Finished to install air coils in Apple-X.
- SA2 phase shifter solved.
SPB:
- Nothing to report.
FXE:
- Interlock issues were fixed.
MID:
- All fine at MID.
HED:
- All fine at HED.
SCS:
- Nothing to report.
SQS:
- All good at SQS. |
294
|
24 Jan 2022, 11:18 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 24.01.2022 |
General Announcement:
- Steffen H. states DOC starts in special operation mode 09:00-17:00 for the weekdays. Jour-fixe also starts today at 13:00. Instruments are welcome to join.
- Regarding DATA OCD numbers, DRC started last Saturday: it did not work in some cases: please do not call directly any other OCD number, just the DOC or DRC number;
- Over the weekend an issue the bunch number pattern happened. It is ongoing and has to be tested.
CTRL:
- Nothing to report.
DA:
- Tomorrow DA members are in satellite workshop of their group.
EEE:
- Two OCD calls for ELMO drivers which lost configurations: solved.
VAC:
- No issues.
XPD:
- Commissioning work during the weekend on SA3 tunnel (mainly): bunch pattern could not be changed in any SASE.
- Tickets were sent and people responded promptly.
- One MPOD crate was in error in SA3: a power cycle in XTD10 was done (helped by XRO) yesterday night and fixed.
- Commissioning of Imagers, PES in SASE3; hard X-ray PES in SA1 (missing some DAQ) started; MCP-detector; and Shimadzu in SA2.
DET:
- Nothing to report.
ITDM:
- Relating to last Wednesday maintenance: a re-cabling at Desy side occurs and after that it observed unstable behaviors on the network.
Maybe today or tomorrow one needs to make some tests leading to a break of GPFS operation (short interruption of migrating data from online to offline).
XRO:
- SA3 proceeding well and for SQS several mirror positions are saved.
- On Sunday there was a problem preventing to complete the SCS (control issues of translation stages of pre-mirrors). It was solved but happened a couple of times.
Please experiments in SASE3 be aware that current translation stages positions may differ by 1mm.
- XRO implemented two-steps backlash corrections in SA3 (as done before in SA1): to solve the different positions after moving mirror.
Maurizio V. informs that instruments shall not be surprised if the motor overshoot and then return to correct position.
- SA2 beam transport: SRA calibrated; beam arriving to HED Pop-in instrument; PBLM was commissioned.
- SA1: starts today. This week focus goes to SA1 and SA2 realignment after the SESP interlock tests.
LAS:
- SA2 PP-laser: issues with ADQ card and smarAct: with help from EEE and CTRL, is fine now.
- SA3 PP-laser: AMPHOS maintenance this week; it shall be ready for user run.
- SA1 PP-laser: updating this one; LAS is working hard.
UNSYS:
- Beam trough in SA3; Working to correct some kicks in Apple-X.
- BKR requested to give a feedback that system is giving signals to BKR. Working on it.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- Not present.
HED:
- All fine.
SCS:
- Not present.
SQS:
- Nothing to report. |
295
|
26 Jan 2022, 10:46 |
Frederik Wolff-Fabris | General | Information | Notes XO Integration Meeting 26.01.2021 |
General Announcement:
- The cold compressor failed this morning and beam operation is currently stopped. We expect the beam operation for the alignment of beam systems will resume tomorrow morning (Thursday).
CTRL:
- Nothing to report.
DA:
- Nothing to report.
EEE:
- Closing last tasks for the maintenance.
VAC:
- All fine.
XPD:
- Issues with PES were solved by CTRL.
- Jan G. asks when beam would be aligned to M1 and M2 in SA1 and SA2? (see XRO report).
DET:
- Nothing to report.
ITDM:
- Work ongoing with company regarding Monday's report on offline storage system.
- Jan asks for the DAQ Karabo test with XPD. Janusz S. suggested to contact Partha who is on charge for this.
XRO:
- Ahead of schedule with alignment until this morning: SA1 and SA2 beam established in all experiments. SA1 beam on mirror but it needs to finish Bender calibration (nice beam at the end of the SA1
tunnel).
- Currently cryo is down and pushes back next days plans: Today planned schedule of tuning 17.2 keV in SA1 as well as Apple-X in SA3 is shifted to tomorrow.
If beam comes tomorrow morning SA2 is available for alignment. By end of the week one starts with SA3.
LAS:
- SA2 almost ready: it shall be finished by end of the day.
- SA3 Amphos maintenance is finished: maybe ready end of the week or start next week.
- SA1 is making progress: 1.1MHz operation shall be OK but 280kHz can be have some difficulties.
UNSYS:
- Power failure in SA1 central control node prevented communication to undulators during the evening. It was solved after power cycle.
SPB:
- Busy with aligning beamline.
FXE:
- Same as SPB; nothing else to report.
MID:
- All fine.
HED:
- Not present.
SCS:
- All fine.
SQS:
- Nothing to report. |