Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 FXE SPB MID HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 23 of 31  Not logged in ELOG logo
ID Date Authordown Location Severity Subject
  187   16 Dec 2020, 09:11 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 16.12.2020

General Announcement:

- Today is the last xo-integration meeting for this year. Xo wants to thank everyone for the participating in this meeting series while sharing information and giving updates.

- The next XO Int. Meeting will happen on 04 January.

CTRL:

- Beckhoff MC2 motor upgrades at SA3 is ongoing.

DA:

- no updates

EEE:

- all SA2 fast ADC tasks have been concluded.

- Cabling in HED is ongoing; On MID will happen next year.

- AIBS control at SA1 balcony room installation starts today.

- Testing Mc2 motor configurations in tunnel devices.

VAC:

- Trying to finish main shutdown tasks: GATT, new mirror chamber in SA3/SXP, SA2 M3 mirror chamber.

- M3 chamber InGa is clean and ready to be pumped down.

- XGM/SA1: broken gate will be replaced this week or next.

- Some members of VAC group are working on site next week.

XPD:

- XGM SA2/XTD6 was moved 4mm and is still in alignment (no damage found) -> Jan G and Harald S. will have a look today onsite. M. Dommach says the differential pump could not move that far away due to bellows. 

- Installing of switches for HRIX will be most likely finished this month and some maybe in January.

DET:

- nothing to report.

ITDM:

- nothing today.

XRO:

- Not present.

LAS:

- Laser Hutches are locked (no access allowed) by Safety and LAS members can not continue the work in laser rooms. LAS maintenance tasks can not be finished and may be delayed over the end of shutdown.

- Emission of SASE2 Seeder laser was put in OFF yesterday due to inspection by SLSO. This means laser was OFF and there is no guarantee the PP-laser will be available in February, as they will not start mode-locking again after an Emission OFF. 

- Typically the Seeder lifetime is 3-4 years. Each time it is switched OFF it shortens the lifetime by half. If Seeder is included in the Interlock system, and taking into account the interlock trips 1 time per week or month (case of SASE3), the availability of PP-lasers is in risk and consequently the whole operation for instruments.

- Alignment lasers will be no longer provided. This also relates to the safety decision - such lasers are not allowed to be operated.

UNSYS:

- not present.

SPB:

- nothing to report: quiet days.

FXE:

- Craning activities over the hutches due to SXP construction still ongoing: not always a notice from construction workers.

MID:

- all fine.

HED:

- nothing to report.

SCS:

- not present.

SQS:

- nothing to report

  188   06 Jan 2021, 09:05 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 06.01.2021

General Announcement:

  • Due to work restrictions, some of shutdown activities will not end as planned in time. This leads to a prolongation of the start-up phase starting next week. It means re-start of Accelerator and South branch will be delayed by 2-4 days while the North branch by about 1 week.
  • The first 2 weeks of X-Ray delivery (2.2-15.2) will most likely be affected and photon delivery cannot be guaranteed into all beam-lines at all days. Some flexibility in planning for the commissioning tasks is expected. A detailed schedule for an upcoming week will be available on the week before, latest on Friday.
  • Sara C. asks if tunnel access is allowed during this week. F. Wolff-Fabris understands tunnels still can be accessed this week apart of Covid regulations (in the context of tunnel maintenance tasks that have to be performed while tunnels are open). T. Tschentscher says the understanding is from Robert F. guidelines e-mail are still valid for this week and really important tasks may be conducted – an e-mail to Robert and Nicole pointing the task with the need of access could be sent. Steve A. reported from today’s Covid taskforce meeting that the regulations of last corona email are still valid - if there are any question send an email to health@xfel.eu.

CTRL:

- Due to incompatibility discovered after last update on Dec. 23rd, CTRL would like to update files on all microTCAs before user operation (Karabo devices) mostly likely next Friday.

- A network maintenance was performed by Desy. Please inform CTRL if issues are encountered.

DA:

- nothing to report

EEE:

- Preparing to finish the cabling in HED and MID.

- Nothing to report from PLC side.

VAC:

- nothing to report - tunnel update tomorrow

XPD:

- XPD OCD put in place during Christmas break had no emergencies calls.

- XPD expect earliest work to take place on next week.

DET:

- nothing to report

ITDM:

- Desy network team would like to do a update of the control network routers tomorrow at 1300 -> email comes to readiness

XRO:

- Proceeding with activities from last days of last year.

- No major updates. Testing the SA3 MC2 library update.

LAS:

- Lasers are down.

- SA2 seeder that took longer than expected time to be back in December after the OFF request by LSO. Probably on next OFF mode LAS has to exchange it.

UNSYS:

- Minimum gaps for SASE3 Undulator system is 10mm. Other SASEs shall be defined after the middle plan scans.

- Apple-X major work from Unsys is done.  

- Quad movers are working remotely.

- Remaining task: connecting fire extinguisher system in one rack.

SPB:

- nothing to report

FXE:

- nothing to add

MID:

- nothing to report

HED:

- nothing to report

SCS:

- not present

SQS:

- A High voltage power supply with double-polarity was installed. It is the first time this is installed at SQS.

  193   25 Jan 2021, 09:28 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 25.01.2021

General Announcement:

- Test of automatic switch of power bridge is shifted to Feb.2: this affects all devices connected to orange or green power sockets (no power for few seconds/minute). General e-mail will be sent. Instruments will need to care for equipment which is connected to these sockets. This also affects the E0 level as well as office space in XHQ. Pease plug devices out or re-plug in other sockets.

  • The Covid-19 regulations continue valid. Only critical work in the XHEXP1 or tunnel activities shall take place.

- Work on the Soft Mono in XTD10 (SA3) takes place this week. XRO and ME asks everybody to do not move any motors in this device during the whole week.

CTRL:

- New Script to fix label issues in Karabo is ready and will be run today. An e-mail will be sent.

- ITDM and CTRL managed to get updated the “Read the docs”. It runs properly.

DA:

- Not present due to Satellite Workshop.

EEE:

- nothing to report

VAC:

- continuing with tunnel works

XPD:

- Ongoning with few tasks in the tunnel

DET:

- Not present

ITDM:

- nothing to report. Offline GPFS under work today: no Maxwell access during this time.

XRO:

- Not present. Please see general announcements on the Soft Mono.

LAS:

- Error with remote operation on mirrors for SA3/SQS PP-laser is found. A discussion after the meeting will take place between LAS and XO.

UNSYS:

- Nothing to report

SPB:

- Nothing to report

FXE:

- Nothing to add

MID:

- all fine

HED:

- Nothing to report

SCS:

- Nothing to report

SQS:

- Nothing to report

  194   27 Jan 2021, 09:08 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 27.01.2021

General Announcement:

- FFP2-Masks needs to be used in the experiment control hutches from now on regardless the number of people working in this areas. Regulations concerning the maximum number of staff in this areas remain the same.

- SASE1 Laser interlock upgrade originally scheduled for this week was postponed to next week. As soon as we receive the information on the new date, this will be communicated.

CTRL:

-   Intervention for the new scenes for the size of the labels was originally scheduled for yesterday. This was postponed as additional tests will be conducted in CTRL lab. A new date will be announced. 

- Panel topics: some old types doocs undulator devices were replaced for a single device. CTRL would like to remove the old ones in 2-3 weeks and kindly ask groups to inform if they are still in use. CTRL will send an e-mail on this subject.

DA:

- Nothing special to report.

EEE:

- Nothing special to report.

VAC:

- Still working in XTD10 and XTD6.

XPD:

- Tunnel works ongoing in XTD6 and XTD9, e.g. Shimadzu cameras / Imagers.

- EEE will need to have access at the end of the Imager intervention of XTD6.

DET:

- SPB Agipd: issues with new data acquisition system. It has been under work.

- This week DET plans to test the new Control fix medium gain to be used in user experiments.

ITDM:

- Nothing special to report.

XRO:

- Not present.

LAS:

- After SASE2 AMPHOS update: there are issues with the interlock

UNSYS:

- Unfortunately neither Desy or EuXFEL could grant access to the external company for the XTD4 tunnel cleaning. UNSYS is trying to organize themselves. PKG shall not be switched on before the clean is done.

SPB:

- Tests with DAQ of the AGIPD: new version has issues with data assignment of aggregated version. Currently AGIPD is back in the old version.

- Work of EEE with some crates of rack room.

FXE:

- Nothing special to report.

MID:

- all fine

HED:

- Nothing special to report.

SCS:

- Nothing special to report.

SQS:

- nothing to say

  195   01 Feb 2021, 09:06 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 01.02.2021

General Announcement:

- The test of the emergency automatic switch of power bridge is scheduled for Tuesday, 2nd of February 2021, 09:00. This will affect orange outlets in XHQ and green and orange outlets in XHEXP1. A short interruption in the power supply lines will happen. We expect all groups have taken appropriate measures to place devices in safe mode or connected to white power outlets.

CTRL:

- not much to report, things are running

- Read the docs is operational again. A given feedback to CTRL is welcome.

DA:

- Nothing special to report.

EEE:

- Tomorrow update sase1 motion loop at 13:00: an email will be sent to readiness mailing list.

VAC:

- Nothing to report.

XPD:

-  Maintenance work ongoing: Imagers in XTD6; Work with EEE in XTD6 and XTD9.

DET:

- AGIPD: new control with new gain is tested. DAQ issue with data is fixed (K. Wrona informs it was related to a wrong hidden parameter setting).

ITDM:

- Downtime on Offline calibration system / GPFS: update to be made in about two weeks from now. K. Wrona says that ITDM would like to have one day of GPFS maintenance; it is important to switch to the new scratch files before we start operation. Data taking shall not be affected by this.

- Jan G. asks when would be required by the earliest. K. Wrona says they need to copy a lot of data before the update and it is estimated in two weeks (+-2 days), to take place in the second half of February. As photon activities take place in the last two weeks of February, one needs coordination to find a suitable date.

XRO:

- everything fine.

- Today closing of soft mono chamber in XTD10.

LAS:

- Upgrade in SA1 uTCA: information will be given to LAS this afternoon.

- Two Desy colleagues needs access in SA1 hutches. Use of FFP2 masks is mandatory.

UNSYS:

- ZZ in XTD4 is planned for Wednesday (for cleaning undulator area).

- Suren K. asks if the balcony rooms are effected by test emergency power switch. This will be discussed offline.

SPB:

- Nothing special to report.

FXE:

- Nothing special to report.

MID:

- all fine

HED:

- Nothing special to report.

SCS:

- Nothing special to report.

SQS:

- Nothing to report.

  196   03 Feb 2021, 09:26 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 03.02.2021

General Announcement:

- Automatic Power Switch Testing was done yesterday. The switches are working as expected and a protocol has been produced. Torsten Schoen informs this test is done every 4 years; from now on equipment can be connected again to the orange and green power outlets.

- Re-start of the accelerator takes place from last Monday. Currently the setup of the injector and LINAC is ongoing. There are no changes to the last plan for closing dates of the photon tunnels.

CTRL:

- CTRL works on the Doocs Undulator devices/panels that have to be changed.

DA:

- nothing to report.

EEE:

- EEE and LAS people are on-site checking the reported uTCA problems after the power switch (by LAS). Issue refers to no communication to the uTCA.

- SASE1 motion loop update yesterday was done.

VAC:

- Not present.

XPD:

- XTD6 and XTD9 Shimadzu cameras update continue.

- Next loop update for XTD6 will probably be next week together with Hirex changes.

DET:

- all fine

ITDM:

- everything is fine

XRO:

- VAC is busy in tunnel leaking check the XTD10 soft mono (pressure went down to 10-6mbar already) but it takes more time.

LAS:

- Yesterday the SA1 and SA3 PP-laser systems were lost. Tomasz J. and EEE are looking into it.

UNSYS:

- today ZZ in XTD4 for the cleaning at SASE3. PRC has informed the BKR.

SPB:

- Problems with trigger of Gotthard are reported by Marcin S. It may be also linked to a communication issue.

FXE:

- nothing to add

MID:

- all fine

HED:

- nothing to add

SCS:

- not present.

SQS:

- Wanted to run digitizer on uTCA but could not find/communicate to it. Under investigation this afternoon.

  197   08 Feb 2021, 08:30 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 08.02.2021

General Announcement:

- XTD10 tunnel was closed last Friday; XTD9 to be closed by end of this week; XTD6 is undefined – mostly likely by Monday. Any schedule deviation will be communicated.

CTRL:

- Deployment of offline services from 10.2 to 12.2. Services such as offline calibrations are not reliable in this time. An e-mail was sent and another reminder will be sent again.

DA:

- no news.

EEE:

- Update of PPL uTCA in SA1 today (laser);

- SASE2 PLC loop update to take place tomorrow at 13:00 (Hirex and Shimadzu camera).

VAC:

- Nothing to report

XPD:

- No issues to report.

- Andreas K. says in SA1 hardware activities are done; testing optical alignment.

- Hirex SA2 update is missing.

DET:

- nothing to report

ITDM:

- nothing to report

XRO:

- nothing to report

LAS:

- Switching on SA1 laser – after uTCA upgrade. Small issue relates to the missing trigger from Desy (dynamic triggers). Point of contact are Tim Wilksen and Lars Froelich.

UNSYS:

- Hick-ups with controls during the weekend. BKR opened gaps to 220mm and caused troubles (usually they open o 210mm).

- One ambient field coil needs ZZ for fixing (SA2/cell12); not urgent.

- Cleaning was done in XTD4.

SPB:

- setting up for the first set of commissioning

- Gotthard trigger seems to be solved.

FXE:

- nothing to add

MID:

- all fine

HED:

- nothing to add

SCS:

- not present.

SQS:

- uTCA digitizer issue: ongoing.

  198   10 Feb 2021, 09:20 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 10.02.2021

General Announcement:

- Tunnel closure: it has been decided both XTD6 and XTD9 will be searched and closed on Monday 15.02.

- Temporary tunnel access requests: for ZZ access please be aware of the new Redmine ticket for access requests. The link to it appears in the BKR E-log.

- Use of FFP2 masks in experiments control hutches:  Sandra Plett informs rules such as temporary stop of work after 75 minutes of mask use or a day off after two working days with masks are not mandatory - these are only a recommendation; If two people are <1.5m, one has to use masks. While Seated or alone you can remove the masks; However, if there are people running around, please use masks also if seated; The wearing time is just a recommendation - it is not mandatory to make the break.

CTRL:

- Graphical user interface update: there is a new version to install in the gateway and BKR.

- Update of data loggers and GUI servers to take place.

DA:

- all going well.

EEE:

- SA2 loop update (MOV) was done yesterday for Hirex and Shimadzu.

VAC:

- nothing to report

XPD:

- DACHS terminal in XTD9 did not allow access: database connection issue. Andreas K. has contacted M. Prollius.

DET:

- Deployment of offline calibration pipeline from today until Friday. It will be communicated when it’s finished.

ITDM:

- Two weeks ago, ITDM pre-announced an offline GPFS update. Preparations will be ready by Friday. Ideally ITDM will deploy on next Monday. Please let K. Wrona aware if this date is not suitable.

XRO:

- Not present.

LAS:

- Upgrade on uTCA crate in SA1 – ADC acquires data. A small problem with data acquisition was found and a Redmine ticket was created. In any case all is fine for switching PPL-SA1 on.

- Today between 2:00-08:00AM there was no trigger coming to PPL-SA1. It may be from Desy or from Balcony room. Bruno F. will investigate the balcony rooms.

UNSYS:

- Unsys will request access for next week to fix a SA2 ambient coil.

SPB:

- nothing

FXE:

- nothing

MID:

- All fine

HED:

- nothing

SCS:

- nothing

SQS:

- Issues with trigger are fixed.

- uTCA firmware updated; now fixed.

  200   17 Feb 2021, 09:27 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 17.02.2021

General Announcement:

CTRL:

- update for GUI server this morning is finished.

-  CTRL advices people starting systems such as cameras, in case of connecting issues please try first to reboot the devices first.

DA:

- nothing special to report.

EEE:

- will update PPL update mTCA SASE3 - affects only LAS group

VAC:

- nothing to report.

XPD:

- Recommissioning diagnostics in SA1.

- Pop-in YAG screen in FXE/SA1 was found broken. Any information on when was working last time please send to Andreas K. In this Pop-in is a second screen with diamond which is working. YAG exchange is for next maintenance period.

DET:

- The offline calibration service update is finished and is available again. Please contact DET in case of issues.

ITDM:

- nothing to report.

XRO:

- Working at BKR yesterday with commissioning SA1 alignment; SA1 is aligned until SPB and FXE.

- Today continuing with SA2 depending on beam stability.

- Idoia F. mentions that computers were not running at BKR. A check after shutdowns and prior operation is welcome.

LAS:

- The AMPHOS in SA3 needs a new power supply. Expected delivery and exchange on Friday.

- PLC update in SA3 and uTCA upgrade on Friday.

- Advancing with SA1-PPL commissioning.

- LAS will send e-mail to Zunaira A. for that MEA3 personal is contacted on the problems with sensor boxes.

UNSYS:

- Ambient field coil was fixed.

- New minimum gaps will be calculated and communicated to XO.

SPB:

- After interlock test last week there is an issue with the new DACHS terminal. The hutches can’t be accessed. Pouneh S. says DACHS is working but this particular new terminal might not be activated in the system. An special permission must be given. Tobias Haas will be informed to contact MEA3.

FXE:

- Martin K. will ask FXE colleagues on YAG screen information. Currently he is not aware when this happens (similar event back in 2019). 

MID:

- all fine.

HED:

- all fine.

SCS:

- in contact with Eric B. to get the beam shutter back on Friday for closing hutches during the weekend.

SQS:

- Early next week review with the digitizer engineers.

- Pouneh S. asks that laser interlock in laser hutch does not work. The new parts have been shipped and arrive soon. Together with TS electrician one has to exchange the relay. Daniel Rivas is informed.

  202   24 Feb 2021, 09:03 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 24.02.2021

General Announcement:

- Due to a failure alarm in the smoke extraction system in XTD6+8+9, Desy colleagues will need to reboot a crate and exchange one module. This will bring down the air conditioning for the tunnels during about two hours. The activity is scheduled for March 1st starting at 11:30AM and it appears in the EuXFEL infrastructure calendar.

- It has been found a situation with one of the elevators to the XHEXP1 Hall. Due to corona regulations a maximum of 3 people is allowed simultaneously in the elevator. This number is indicated by signs/papers fixed nearby. However, someone unauthorized has modified the number “3” to an “8”. We are investigating the situation. The shields have been corrected back to “3” but we would like to remind signs and warnings must not be modified by unauthorized persons.

- Gerd W. informs that SXP chicane modification foreseen to take place today and tomorrow is shifted to Monday or Tuesday due to delays in delivery of components. Affected people have been contacted by PMO via but in case of issues with the new plan, please contact Gerd W.

CTRL:

- Few requests for tickets on Karabo devices were received. CTRL reminds to please add device ID in the ticket.

- SA3 motors: a few changes were done due to migration of MC2 motors. If someone or a group needs more input, please directly contact Steffen Hauf.

- Regarding Data Operation Center (DOC): CTRL would like to have a functional account to the instrument e-logs to be able to access it and track problems when OCD gets called. If instruments have any concerns with this, please contact Steffen Hauf.

DA:

- Nothing today.

EEE:

- No main issue.

- EEE also asks that when submitting tickets, please include the instrument and component name, preferentiably in the header.

- PLC side: ZZ to check motors.

VAC:

- Nothing to report.

XPD:

- Continuing today with beam commissioning of XTD6 components (HIREX and MCP).

DET:

- Nothing special to report.

ITDM:

- Nothing special.

XRO:

- Issues will be addressed when tunnels are open.

LAS:

- SA1 and SA3 PP-lasers are making progress.

UNSYS:

- Nothing to report.

SPB:

- Issues with Interlock and DACHS are solved.

FXE:

- Interlock issues are solved.

MID:

- Nothing special.

HED:

- Thomas Preston asks if the Covid regulations in hutches can be sent by e-mail. Sandra Plett will send the link to the documents after the meeting.

- The mask occupancy for the hutch was changed (are the signs updated or not?). Thomas P. says it was 4-5 but someone wrote new numbers in the signs. Sandra P. says that if everybody is using FFP2 masks, then more people are allowed than the max. occupancy.

- Pouneh S. says if people use laser safety googles, one can use FFP2 with filters. Sandra P. says that rules also apply for this case.

- Issues with cameras: DAQ did not run (did not take data, probably due to lost configurations). HED will send ticket.

SCS:

- Nothing to report.

SQS:

- Yesterday a meeting to review digitizers took place.

  204   03 Mar 2021, 09:23 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 03.03.2021

General Announcement:

- Hick-ups in the beam operation are happening since last night. There is no obvious indications on the origin of the problem and MXL is investigating to restore a normal beam delivery condition.

- There is an issue with “Operation-2021” e-log where some SPB staff can’t login to create entries. This will be addressed by XO/PRC and ITDM (see ITDM entry).

CTRL:

- SA1 DAQ was dropping some trains (the late ones) in the XGM. CTRL and ITDM are following up and will test the XGM late this week or next week. CTRL will contact XPD for deployment first in SA2 and SA3.

DA:

- News from the DOC: deployment of major updates ongoing/to take place.

EEE:

- nothing to report

VAC:

- not present

XPD:

- Commissioning and setting up for SA3 HIREX

DET:

- preparing for commissioning Detectors

ITDM:

- Janusz S. says the configuration for the “Operation-2021” e-log uses the same configuration as previously set for 2020. F. Wolff-Fabris likens the issue with possible new users and asks SPB to provide the name of users who can’t login into the e-log.

Janusz S. says one can use the copy function from instruments e-log to place a copy into the Operations-2021 e-log. Jan G. says that login is necessary to avoid the entry being as “Anonymous”.

XO and ITDM will address it offline.

XRO:

- not present

LAS:

- In SA3 PP-laser one had exchange of power supply for AMPHOS due to communication errors.

- In SA1 PP-laser one observes a loss of 1/4 of power in AMPHOS. LAS is not sure if there is enough power for users.

- Two problems with triggers on Monday: 1. a faulty board in SA3; 2. MCS group update timing system caused trigger loss in all hutches (due to a bug in their server).

UNSYS:

- nothing to report

SPB:

- Couple of Helps asked to DOC and SPB/SFX are thankful for that.

- Login at Operation-2021 e-log does not work for 4 people.

FXE:

- Prepared for AIBS. Few PLC problems remain but not related to AIBS.

- Next week between Tuesday/Friday there are no access to the hutch due to floor work ongoing.

MID:

- all fine

HED:

- all fine

SCS:

- sliding door will probably be repaired today with an external company.

SQS:

- nothing to report

  206   10 Mar 2021, 09:20 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 10.03.2021

General Announcement:

- nothing for now

CTRL:

- A few issues for DOC: Cameras IP addresses need to be reconfigured

- Continuing investigation of train-IDs on Doocs/XGM (they oscillate). CTRL will follow and close contact with Theo M. and Desy colleagues.

DA:

- nothing to report

EEE:

- nothing to report

VAC:

- nothing to report

XPD:

- Not present.

DET:

- nothing special – commissioning this week

ITDM:

- nothing to report

XRO:

- Maurizio V. reminds the SEPS interlock check in SA3 still has to happen as it is a condition prior beam operation. Zunaira A. says she sent an e-mail to xo-planning.

F. Wolff-Fabris asks if the SCS door is fixed or not, a pre-condition for the test. Zunaira A. replies the problem was with the power supply to the door and is fixed.

A date for the SEPS test can be planned by XO.

LAS:

- SASE2 interlock: LAS got unexpected calls when disconnecting AMPHOS cables (thus triggering SA2 Interlocks).

- SA3 AMPHOS no news since Monday. The ADC cards do not receive train-IDs.

- LAS suffer of lack of laser lab, missing PLC installation. LAS asks XO for a follow-up on prioritizations.

- Zunaira says that lasers are coupled to the interlocks (yesterday event happens as all lasers in room A23 and A07 are connected to the HED and MID interlocks). Thomas Preston says it triggers the warning lights for the laser lab. Zunaira A. says if shutters are closed it shall not be a problem if lasers reach this shutter signals. Tomasz J. says that ideally they must be decoupled.

UNSYS:

- nothing to report

SPB:

- going on with the plan; Detector commissioning from tomorrow.

FXE:

- the floor works are ongoing - looking good.

MID:

- not present

HED:

- nothing to report

SCS:

- A new experimental chamber (chemistry chamber) will be installed for beam commissioning on next month. A CPU in rack room is needed for recommissioning the motor an Martin T. waits for feedback.

SQS:

- Staff is getting the search training which was expiring.

- Noticed issues notice with DACHS system which show the access properties with delay. 

  208   17 Mar 2021, 09:15 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 17.03.2021

General Announcement:

- nothing.

CTRL:

- LAS reported the pulse statistics did not work properly: re-started systems due to lost connections.

- OCD call related to HED AGIPD test system: fix ongoing.

- Update on train-ID slow data on XGM: CTRL is in touch with Desy and Theo. Question to SPB and XPD: how much effort is needed for that CTRL and ITDM to place the missing information in the run. Involved parts will discuss an agreement by the end of the week. Steffen H. will prepare a summary to inform XPD.

DA:

- no important news.

EEE:

- no important news.

VAC:

- nothing to report.

XPD:

- nothing to report.

DET:

- nothing special.

ITDM:

- all fine.

XRO:

- Not present.

LAS:

Going fine: Laser pattern issue from yesterday was solved by CTRL.

UNSYS:

- Problem on Monday with quad mover cell18/SA1 took 6 hours on ZZ. It is now fixed.

- A problem with undulator segment on cell36/SA1 needs diagnostic and eventual ZZ access. RCs are contacted.

SPB:

- The samples to be used next week are challenging to prepare. 

FXE:

- Used beam yesterday. No problems to report.

MID:

- all fine.

HED:

- nothing to report.

SCS:

- nothing to report.

SQS:

- all good.

  224   12 May 2021, 09:21 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 12.05.2021

General Announcement:

- User Operations Meeting takes place on next Friday 09:00.

CTRL:

- CTRL would like to excuse itself to people receiving an automatic SMS tonight for notification. This was a test message and now it’s turned OFF (Good side: test for lasers went well).

- ADQ digitizers: two fixes are driven and based for SQS commissioning. CTRL encourage instruments to get in touch with contacts for making the update for the last version.

DA:

- Follow up from Monday / calibration update (on behalf of cal team): it worked as planned. AGIPD seems to run faster now.

- For SQS Remi detectors/digitizers can also be used.

EEE:

- nothing to report

VAC:

- nothing to report

XPD:

- are preparing for FD studies next week

DET:

- Follow-up on LPD online calibration pipeline pointed last Monday’s meeting: work is on progress and is expected to be ready for next week.

- DAQ processing for AGIPD: a minor bug in validation is reported and under investigation. But it is currently working.

- Issue reported for SPB for data aggregators: follow up with CTRL.

ITDM:

- nothing to report

XRO:

- nothing to report

LAS:

- nothing to report

UNSYS:

- nothing to report

SPB:

- not present

FXE:

- Thank to DET and DA for pipeline calibration. Initially it looks promising.

- Last week: digitizer went to wrong configuration. To be fixed with new updated.

- FXE will use the beam during the nights. D. Khakhulin will inform PRC and XO via e-mail.

MID:

- all fine

HED:

- not present

SCS:

- nothing to report

SQS:

- nothing to add

  234   16 Jun 2021, 08:37 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 16.06.2021

General Announcement:

- Interlock test next 2 weeks starting on monday - from next wednesday the photon tunnels will be closed; Tests are the XHEXP1 take place on 30.06, 01.07 and 02.07 with the last date as reserve; An email is going to be send by FWF to the xo-info@xfel.eu list.

CTRL:

- List of items to be sent to readiness: 1. problems with lima based cameras (Lima library does not work – needs to be updated: instruments contacts shall get in touch); 2. In SA1 there is legacy Undulator Karabo scenes – vector properties have to be adapted; 3. Counter Clocking motor devices upgraded next week.

- A. Koch asks if SA1 was working this morning. Steffen H. says there were updates but must be running now.

DA:

- Next wednesday will be the Offline calibration update deployment

EEE:

- Shutdown activities going well; "Wille" is short in people but scheduling adjustments with FXE and MID is organized.

- If there are short circuits or mal functions in the Beckhoff crates and current is not high enough to trigger the fuse it may be a fire risk. Solution is developed without impact in firmware, just a replacement of fuses. EEE starts today at SPB the changes; Based on this experience a plan will be made for the other systems and EEE will get in touch with groups.

VAC:

- try to get the works finished in the tunnels

- waiting for the Beckhoff crate works

XPD:

- Most of the tunnels works have been done; others are ongoing; No surprises are expected.

- MKK logbook from Desy reporting about electrical issue. XO will check. (XO update at 13:30: automatic mailing system is being commissioned and under testig; no power failure happened at the time).

DET:

- Work with DSSC in SQS going as planned.

ITDM:

- Work currently being done at FXE.

- Reminder of Friday afternoon gateway update.

XRO:

- Not present.

LAS:

- Nothing special to report.

UNSYS:

- Greasing the linear stages for undulators but not in SA2 (missing grease).

- SA1/SA2 after middle plane alignment UNSYS redefined the minimum gap and will be communicated through XO.

- Cell#36/SA2 there is a cable disconnected that need to be fixed.

- SA1/SA2 working with PS tables for extending operating tables for higher photon energies.

SPB:

- Work with EEE changing all the crates (Beckhoff)

FXE:

- Nothing to add

 

MID:

- Several ongoing infrastructure works are going fine.

- Worried with Beckhoff fuses (reported two years ago by Konstantin). It will be good to hear experience with SPB.

HED:

- Nothing to add

SCS:

- Nothing to add

SQS:

- Nothing to add

  245   26 Jul 2021, 09:20 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 26.07.2021

General Announcement:

- Power glitch 3 times at night, what brought down the beam for 1,5 hour each time. Please report to XO if you find devices affected.

CTRL:

- Datalogger update to take place today between 16:00-17:00. Please inform CTRL in case of problems with this schedule.

- Aravis camera and bunch pattern decoder to be updated at around 14:30. Please inform CTRL in case of problems with this schedule.

DA:

- OCD side: no new problems.

- Follow up last Wednesday: better is to use the new Karabo with the fix and not to downgrade the Karabo version.

EEE:

- Ticket from SCS was solved.

VAC:

- Some trouble with the pumps in the gas attenuator SA3. It needs to be fixed today in the tunnel with a ZZ (30 mins).

ME.

- Not present.

XPD:

- Are investigating, if something is affected by the power glitch. So far nothing was found.

DET:

- Nothing to report.

ITDM:

- Two issues to follow with MID: 1. Bassler camera seems to be the data wrongly recorded at DQA. Tests need to be done; it’s a camera used for feedback. Ulrike B. will talk offline with K. Wrona.

- Second issue: on Friday there was one ADC from tunnels to be used from MID was requested a bit late. Data comes from the uTCA from Desy network. If this device is going to be used in full, it has to be set inside Desy network.

XRO:

- Not present.

LAS:

- Two incidents: the power glitch at 19:23 yesterday took Amphos in SA1 down. It is back and operational; In SA3 LAS is further investigating the Amphos.

UNSYS:

- No problem with the power glitch.

- The temperature in SA3 Undulators is stable.

SPB:

- SPB imager at XTD9: is not working and has to be changed the outcome needs to be fixed of XPD. Naresh K. will contact SPB.

FXE:

- Not present.

MID:

- All good at MID – not heard on affected systems due to the power glitch.

- Friday afternoon confusion with the fix of Extra Foam bridge. Probably DA is aware.

HED:

- Vacuum system ok after power glitches.

- Nothing to report.

SCS:

- Not present.

SQS:

- Not affected by the power glitch.

  246   28 Jul 2021, 09:29 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 28.07.2021

General Announcement:

- No general announcement.

CTRL:

- Problem with cameras / DAQ: fixed yesterday and tested at SA2/MID.

- Issues with SA3 vacuum loop: work ongoing and further investigations are needed.

DA:

- Nothing to report.

EEE:

- Cables available for HED to be installed next week – date to be discussed offline.

- Fast electronics side: working on FXE and HED issues;

- Nerea J. reports on the VAC/SA3 issues: instability with devices going into an unknown state. Alessandro S. works at the night shift in the DOC and will try to fix a permanent fix during the night.

VAC:

- SA3 issues are affecting also the gas attenuator (not only general SA3 vacuum loop but also GATT is affected, which has its own loop). There are no physical events - seems to be electronic. Stability issues to connect devices; Seems to be more related to infrastructure (PLCs). Further investigations are needed.

ME:

- Not present.

XPD:

- SPB imager: change setting of motors (it shall work now) and next week when A. Koch is back further investigations will be done.

DET:

- During the night after 02:00AM: SPB AGIPD K-bridge went OFF. Under investigation. Detector recovered this morning.

ITDM:

- Nothing to report.

XRO:

- Not present.

LAS:

- Nothing special to report.

UNSYS:

- Nothing to report.

SPB:

- Issues with wings of Agipd, causing also interlock trigger. Under investigation.

- 11PM last night: laser interlock triggered. It happened also in the past and SPB will get in touch with Safety today.

- XTD9 IMG: currently working but a permanent fix is welcome in the future.

FXE:

- Not present.

MID:

- All fine.

HED:

- Major issues with Agipd detector yesterday (thanks to DOC for help): 1. Could not kill the service and CTRL investigated; 2. Clock and control did not work (it has worked at CFEL last week). The issue was solved by changing the clock controller back to old configuration (EEE helped). Bruno F. comments that the clock control had to be downgraded in this detector (it worked in other detectors, only this one did not work). Further investigations will follow.

SCS:

- Change of speed from the monochromator was not possible. EEE and CTRL are working on it.

- Next week will run with a special train picker / bunch sequence pattern as the one used last week.

SQS:

- Vacuum problem as mentioned.

- Commissioning DSSC for future beamtimes.

  87   27 Nov 2019, 09:25 Andreas GallerGeneralInformationdispatch meeting Mon Nov 25 2019

FXE: no issues

SPB: no issues

MID: Split & Delay line arriving on Friday, MID aims for a quick integration into instrument / beamline
         EEE informed to switch Hexapod motors from loop3 to loop 7

HED: changing loop1 (JJ X-ray motors) from open to closed loop operation

 

SQS: KB works, see sometimes spontaneous switching of end switch logics

EEE: MID, SPB, VAC loop updates running this week;
         no triggers for experiments available until end of power off work
         Jörn: cabling for MID is in full swing, for future installations first check phase 1 cabling, for non blocking
                                   HED: cabling
                                   SCS company Witte 30 ccables
                                   FXE: two CRDs not approved yet, no cabling during this shutdown

Controls:
 

  88   28 Nov 2019, 15:42 Andreas GallerGeneralProblemaffected ITDM Systems after power cut tonight

XHQ.E1.092

SW-HH-AL-271 was available
SW-HH-AL-272 came back after the DL switch was back (see below, ca 09:33)
SW-HH-AL-273 had to be power cycled and rebooted by NOC through console to come back fully (ca. 10:52).

 

SASE1 balcony room 38

restarted at 4am:
DELL servers:
exflprx03n0
exflprx04n0
exflprx112
exflpcl01n0 to 08n0
exflpcl18n0

 

SuperMicro servers:
exflonc09 to 12 (4-in-1 chassis)
exflong06

 

IBM ESS systems stayed off, as well as storage jbods
exfl-ons-​​​​​gs111 and 112
exfl-ons-gl111 and 112
exfl-ons-gl113 and 114

 

IBM server (hosting ems servers for OS installations) was off as well
exflkvm01

 

SuperMicro servers stayed off:
exflong03 to 05

 

juniper (management network) switch stuck at booting, power cycled, after long time finally came back, but we were lacking management network over whole site

 

SW-HH-AL-DL-07 (office network distribution layer) switch was stuck at booting, came back after power cycling. Some (not all) office switches were not available because of this, seemingly the failover between AL and DL didn't work. Both this switch and the (permanently available) ...DL-08 were correctly cabled to separate PDUs.
Second fuse in BR38 rack 7 was out and had to be pushed back in manually (made power redundant again).

 

SA1 tunnel:

exflcon08n0 restarted at 4am, but apparently switch in the tunnel was off till 10:30 and since that time the server was accessible

 

E0.604

All systems were affected. We need the clear statement from YS why the second line did not work.

 

DELL servers rebooted at 4am:
exflprx01n0
exflprx02n0
exflpcl51n0
exflpcl52n0
exflpcl53n0
exflcon03n0
exflcon04n0
exflcon05n0
exflcon150
exflcon153
exflcon156
exflcon157
exflons01 and 02
exfldaq01 and 02 (gitlab CIs)
exflst105 (karabo project db - development and labs)

 

IBM servers stayed off:
exflpcl54n0
exflpcl55n0
exflpcl56n0
exflpcl57n0
exflonc41
exflonc41

 

IBM ESS system stayed off:
exfl-ons-gs901 and 902
There are also several servers (proxmox, serving VM OnlyOffice) all rebooted at 4am

 

List of h/w failures:

exflpcl05 PSU dead

exflong04 and 05 PSU dead (in exflong04 was actually since yesterday after reboot for OS updates)

 

may not be related to power glitch

exflprx106 RAM DIMM error

Intel server:
exflonc21 to 24 (4-in-1 chassis) PSU dead

 

  95   15 Jan 2020, 09:09 Andreas GallerGeneralInformationnotes Dispatch meeting Jan 13 2020

SPB:

  • Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
  • optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished

FXE:

  • XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE

MID:

  • SDL should enter the hutch by the end of January

SCS:

  • pulse (train) picker to be installed in the tunnel by the end of this week

 EEE:

  • FXE had been migrated to MC2 lib
    • motors need to be recalibrated
  • XTD6 motion part is to be migrated to MC2 lib
  • SQS gets interlock updates

CTRL:

  • busy with tunnel, additional support for XPD (imagers motion components)

ELOG V3.1.4-7c3fd00