Group Selection Page HELP Controls Group Data Analysis Group ITDM WP76 DET Vacuum EEE WP78 FXE SPB HED SQS SXP Sample Environment Operation PSPO XO TS Migrated2Zulip
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Technical integration and coordination logbook, Contact: A. Violante, G. Wellenreuther, Page 23 of 32  Not logged in ELOG logo
New entries since:Thu Jan 1 01:00:00 1970
ID Date Author Location Severity Subject
  188   06 Jan 2021, 09:05 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 06.01.2021

General Announcement:

  • Due to work restrictions, some of shutdown activities will not end as planned in time. This leads to a prolongation of the start-up phase starting next week. It means re-start of Accelerator and South branch will be delayed by 2-4 days while the North branch by about 1 week.
  • The first 2 weeks of X-Ray delivery (2.2-15.2) will most likely be affected and photon delivery cannot be guaranteed into all beam-lines at all days. Some flexibility in planning for the commissioning tasks is expected. A detailed schedule for an upcoming week will be available on the week before, latest on Friday.
  • Sara C. asks if tunnel access is allowed during this week. F. Wolff-Fabris understands tunnels still can be accessed this week apart of Covid regulations (in the context of tunnel maintenance tasks that have to be performed while tunnels are open). T. Tschentscher says the understanding is from Robert F. guidelines e-mail are still valid for this week and really important tasks may be conducted – an e-mail to Robert and Nicole pointing the task with the need of access could be sent. Steve A. reported from today’s Covid taskforce meeting that the regulations of last corona email are still valid - if there are any question send an email to health@xfel.eu.

CTRL:

- Due to incompatibility discovered after last update on Dec. 23rd, CTRL would like to update files on all microTCAs before user operation (Karabo devices) mostly likely next Friday.

- A network maintenance was performed by Desy. Please inform CTRL if issues are encountered.

DA:

- nothing to report

EEE:

- Preparing to finish the cabling in HED and MID.

- Nothing to report from PLC side.

VAC:

- nothing to report - tunnel update tomorrow

XPD:

- XPD OCD put in place during Christmas break had no emergencies calls.

- XPD expect earliest work to take place on next week.

DET:

- nothing to report

ITDM:

- Desy network team would like to do a update of the control network routers tomorrow at 1300 -> email comes to readiness

XRO:

- Proceeding with activities from last days of last year.

- No major updates. Testing the SA3 MC2 library update.

LAS:

- Lasers are down.

- SA2 seeder that took longer than expected time to be back in December after the OFF request by LSO. Probably on next OFF mode LAS has to exchange it.

UNSYS:

- Minimum gaps for SASE3 Undulator system is 10mm. Other SASEs shall be defined after the middle plan scans.

- Apple-X major work from Unsys is done.  

- Quad movers are working remotely.

- Remaining task: connecting fire extinguisher system in one rack.

SPB:

- nothing to report

FXE:

- nothing to add

MID:

- nothing to report

HED:

- nothing to report

SCS:

- not present

SQS:

- A High voltage power supply with double-polarity was installed. It is the first time this is installed at SQS.

  187   16 Dec 2020, 09:11 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 16.12.2020

General Announcement:

- Today is the last xo-integration meeting for this year. Xo wants to thank everyone for the participating in this meeting series while sharing information and giving updates.

- The next XO Int. Meeting will happen on 04 January.

CTRL:

- Beckhoff MC2 motor upgrades at SA3 is ongoing.

DA:

- no updates

EEE:

- all SA2 fast ADC tasks have been concluded.

- Cabling in HED is ongoing; On MID will happen next year.

- AIBS control at SA1 balcony room installation starts today.

- Testing Mc2 motor configurations in tunnel devices.

VAC:

- Trying to finish main shutdown tasks: GATT, new mirror chamber in SA3/SXP, SA2 M3 mirror chamber.

- M3 chamber InGa is clean and ready to be pumped down.

- XGM/SA1: broken gate will be replaced this week or next.

- Some members of VAC group are working on site next week.

XPD:

- XGM SA2/XTD6 was moved 4mm and is still in alignment (no damage found) -> Jan G and Harald S. will have a look today onsite. M. Dommach says the differential pump could not move that far away due to bellows. 

- Installing of switches for HRIX will be most likely finished this month and some maybe in January.

DET:

- nothing to report.

ITDM:

- nothing today.

XRO:

- Not present.

LAS:

- Laser Hutches are locked (no access allowed) by Safety and LAS members can not continue the work in laser rooms. LAS maintenance tasks can not be finished and may be delayed over the end of shutdown.

- Emission of SASE2 Seeder laser was put in OFF yesterday due to inspection by SLSO. This means laser was OFF and there is no guarantee the PP-laser will be available in February, as they will not start mode-locking again after an Emission OFF. 

- Typically the Seeder lifetime is 3-4 years. Each time it is switched OFF it shortens the lifetime by half. If Seeder is included in the Interlock system, and taking into account the interlock trips 1 time per week or month (case of SASE3), the availability of PP-lasers is in risk and consequently the whole operation for instruments.

- Alignment lasers will be no longer provided. This also relates to the safety decision - such lasers are not allowed to be operated.

UNSYS:

- not present.

SPB:

- nothing to report: quiet days.

FXE:

- Craning activities over the hutches due to SXP construction still ongoing: not always a notice from construction workers.

MID:

- all fine.

HED:

- nothing to report.

SCS:

- not present.

SQS:

- nothing to report

  186   14 Dec 2020, 09:00 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 14.12.2020

General Announcement:

- Please write to xo.planning, if something comes up with intteruption of the maintenance due to new COVID-19 regulations.

CTRL:

- Migration of Beckhoff MC2 motors at SA3.

- Checking operation (e.g. soft mono at SA3).

- Final checks together with EEE.

DA:

- nothing to report

EEE:

- Migration at SA3 MC2 PLC´s.

- Fast ADC: SA2 maintenance starts.

- Cabling to be done in SQS today and tomorrow at HED.

- Crate was sent to Poland and shall arrive this week.

VAC:

- not present.

XPD:

- problem / alarm XGM at SASE1 / XTD2  - Needs to re-cable and install new gauges; requires VAC group.

- PES was re-installed in XTD10 and has been pumped down.

- DAQ test on friday - things are fine.

- XTD6 XGM alignment starts today - used to be confirmed with VAC.

DET:

- AGIPD intervention: need some internal coordination also with DASY groups - it will let XO know, if support needed

ITDM:

- XFEL gateway next Monday

- pre-anouncement: 90% of maxwell cluster update to be done on January 4/5. Communication to be send to Readiness mailing list.

XRO:

- Testing Controls in tunnel devices - needs to fix SA3 expected bugs.

- XTD10 close the vaccuum for M6 mirror chamber this week.

- XTD6 some problems with M3 mirror: to be pumped down this week and checked start of January.

LAS:

- Small delay on works at SA3, but no risk for users.

- Question from LAS: is the SA1 laser interlock update takes place?

- SA2 AMPHOS: external company is coming today

UNSYS:

- not present

SPB:

- nothing to report

FXE:

- nothing to report

MID:

- all fine

HED:

- don´t have anything

SCS:

- nothing to report

SQS:

- nothing to report

  185   09 Dec 2020, 10:24 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 09.12.2020
General Announcement:
- Regarding the e-mail related to the general radiation training and procedure to deliver a signed certificate, one can also send it by conventional post to Zunaira A.

CTRL:
- BKR machines has been updated to the new Karabo release.
- The Notification service update will follow form today.
- CTRL reminds on getting configuration from the timing feature on the GUI: right click on the device and then "get configuration". 
An e-mail with detailed instructions will be sent today.

DA:
- Nothing to report.

EEE:
- Fast electronics: SASE3 maintenance is finished; SA2 will follow.
- PLC side: SQS has some loops running but not all of them due to the cooling issue. SCS will get their loops soon running.
- Cabling works at FXE is completed; at HED is ongoing.

VAC:
- Not present due to tunnel activities.

XPD:
- XPD has completed PES beamtime at Petra. Would like to thank ITDM for the smooth work. XPD has collected data for over 300 runs.
- Today XPD brings back other PES to SA3 tunnel. There is an appointment with MEA and the plan is to connect to the beamline tomorrow together with VAC.
- A. Koch asks EEE at CTRL if DAQ tests on XPD tunnel devices can happen on Friday. 
Nerea J. points SA3 is not yet commissioned (has to check motors) but SA1 and SA2 can be done on Friday as it has only pending vacuum update. 
Steffen H. says that cameras can be tested in all 3 SASEs; connections shall be fine; requests a list of all cameras. 
Jan G. adds the idea is to verify the components can be operated and acquire data, following a test with XPD/ITDM. 
Steffen H. reminds that for testing the digitizers one has to re-start (faster ADC) the device server. 

DET:
- Not present.

ITDM: 
- Nothing to report.

XRO:
- Nothing present.

LAS:
- Lasers are on good track.
- Interlock tests: there was miscommunication. The SA2 AMPHOS is disassembled for maintenance and therefore the PP-laser test in SA2 is postponed to next week. 

UNSYS:
- Not present.

SPB:
- Nearly done with putting together the AGIPD. The Final test for DQA begin next week.
- Problem with timing but currently it has been running on local mode.

FXE:
- Not present.

MID:
- All fine at MID.

HED:
- Nothing to report.

SCS:
- Not present.

SQS:
- Nothing to report.
- SQS has ongoing work on Beckhoff crates.
  184   07 Dec 2020, 11:08 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 07.12.2020
General Announcement:
-	 

CTRL:
- Deployment updates: control network topics are through - SPB to be done in January.
- BKR and notification service updates to be deployed tomorrow or Wednesday.

DA:
- Nothing to report.

EEE:
- Cabling: finishing FXE and starting HED in the course of this week.
- Fast ADC: uTCA in SCS will be restarted for updated. SCS will contact Bruno F.
- SCS racks: cooling water is disconnected. Thomas Baumann informs TS needs to order cooling water lines and only in January will be connected. 
It is suggested to run with open doors in the meantime; Nerea J. says if one needs 1-2 loops then this is fine; Thomas B., Ulf B. and Nerea J. will discuss the details. 

VAC:
- Nothing to report.

XPD:
- PES installation will happen next Wednesday/Thursday. J. Laksman will contact VAC.
- Alarms in various XGM seem last week were due to updates. It is not a real problem.

DET:
- Nothing to report.

ITDM: 
- Lennart B. coordinates with instruments.
- Due to tomorrow timing update, the upgrade of DAQ on tunnels devices to start on Friday. Others will be contacted individually.
- Jan G. asks on issues of recording DAQ of PES during Petra ongoing beamtime. K. Wrona says it would be better to be informed in advance on the beamtime. 
The issue happens due to overload with a buggy script. If this script is not running, then the issue should not repeat again. 
At Petra it is used a standard control network which is different from the one at EuXFEL.

XRO:
- Not present.

LAS:
- All lasers are in local timing mode.
- LAS requests Bruno F. to have a crate update today as they have ongoing work in SA3. Bruno F. says the updates are agreed to be done today.
- Laser interlock upgrades: there is no news or known progress.

UNSYS:
- Not present.

SPB:
- Detector intervention ongoing and nearly finished. Test with DAQ and final test by end of this week. It is expected to be ready for Monday.

FXE:
- Elektro Wille firma working in the hutch.

MID:
- All fine at MID.

HED:
- Nothing to report.

SCS:
- Water leak last week was the only major event.

SQS:
- Nothing to add.
  183   02 Dec 2020, 10:21 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 02.12.2020
General Announcement:
- Laser interlocks activities in SA2 are delayed by two days. Therefore, activity in SA1 will start later.
- Gerd W. reports the cooling water in SA3 was refilled; air conditionings are running and maintenance on them are ongoing; 
next week temperature and humidity stability in other SASEs will be working; water leakage in SCS (yesterday) has been taken care by TS (crates are not damaged).

CTRL:
- Currently working in SASE2 deployment. SASE1 deployment happened yesterday and went relatively smooth. 
- Fast ADC cards in lasers stop working. CTRL will need to look with EEE.

DA:
- Works on updates are progressing/done.

EEE:
- MicroTCA SA1 update is done. Next week continues with SASE3 update.
- Cabling installation ongoing in FXE. It will be followed by SQS, SCS and MID.
- SA2 and SA1 motion loops updates to be done this week.
- Fast ADC issue with Lasers: Bruno F. says the installed update is the last version and Karabo has also to be updated to its last version also. CTRL and EEE will discuss 
together.

VAC:
- Maintenance activities are going as expected, some tasks are ahead of schedule.

XPD:
- SA3 PES is in the lab and being replaced parts. Plan is to test it today. XPD will discuss with VAC to install back this week or next week.
- XPD received a beamtime shift in PetraIII for the PES. This influences XPD schedule as many people are involved. Beamtime starts this Friday until next Wednesday.
- DAQ tests on Tuesday morning are scheduled. XPD will talk to ITDM to postpone to after the mentioned beamtime shift. 

DET:
- Nothing major to report. AGIPD Intervention: so far, so good.

ITDM: 
- Updates ongoing.

XRO:
- Not present.

LAS:
- ADC cards problems are communicated to CTRL. 
- Observed issues with trigger delay signal and it points to issues with recent update. Currently the lasers use local timing. CTRL will investigate if this can be the 
cause.

UNSYS:
- Not present.

SPB:
- Doing OK with AGIPD intervention: expected to be ready with final tests at start of the next week.

FXE:
- Cabling installation ongoing; Today there is the crane activity for SXP.

MID:
- All fine at MID.

HED:
- Everything is fine.

SCS:
- Not present.

SQS:
- Nothing to report.
  182   30 Nov 2020, 09:15 Arvid GebhardtGeneralInformationNotes XO Integration Meeting 30.11.2020

General Announcement:

- /

CTRL:

- Karabo 2.10 FXE was sucessful.

- go on with SPB and go on with other instruments. On Thurthday 3rd all SASE3 topics.

- Connenting Gui clients, if gateway there is no need to write a tiket this week. If local installation, please make an update.

- network upgrade takes place at SASE1 tomorrow. Karabo will be updated later.

DA:

- /

EEE:

- MircoTCA updates today and on going. Timing won´t be avaliable for short period.

VAC:

- /

XPD:

- /

DET:

- /

ITDM:

- Tomorrow network update SASE1, but there no Karbo update.

- at the beginning of January DESY-IT will do some work on data center / maxwell clust. interuptions are expected. Schedule is still be to expacted.

XRO:

- Not present.

LAS:

- Tests on the mircoTCA failt. More tests needs to be done.

- LAS request comunication from instruments go thru XO.

UNSYS:

- Not present.

SPB:

- Aigipd intervention is running well. To be tested by the end of this week.

FXE:

- cable installation on going.

MID:

- all fine

HED:

- /

SCS:

- Not present.

SQS:

- /

  181   25 Nov 2020, 10:14 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 25.11.2020
General Announcement:
- There is no UOM or XO Int. Meeting next Friday 27.11. We will continue with XO Int. Meeting next Monday 30.11.

CTRL:
- Karabo Deployment in FXE takes place today. Schedule for full Karabo deployment was sent to the Readiness mailing list.
- Two SA2/SA3 Beckhoff devices topics will be updated today (5-10 minutes downtime).

DA:
- No issues or announcement to report.

EEE:
- PLC side: tunnel updates to be started with CTRL.
- Change requests also to start today in a couple of instruments.

VAC:
- Not present.

XPD:
- PES/MEA: there are individual tasks ongoing.

DET:
- Nothing major to report. AGIPD intervention is goes faster than planned.

ITDM: 
- Offline storage update was done and is available since Tuesday.
- Online cluster update has started: SA1 already available since Tuesday. Today ITDM will finish SA2/SA3. Crosscheck and start DAQ will follow.

XRO:
- Not present.

LAS:
- Nothing special to report.

UNSYS:
- Not present.

SPB:
- AGIPD work is ongoing and fine.

FXE:
- Wille company is installing cables in the hutch.
- Issue with craning on SA3 / SXP activities above FXE area: FXE asks for coordination to avoid interrupting activities. 
Gerd W. says it shall not be a problem as crane will be used for few period. Martin K. will inform his phone number for arranging craning times.

MID:
- All fine at MID.

HED:
- All fine at HED.

SCS:
- Shutdown activities ongoing.
- Situation of JJ X-ray will be clarified by SCS staff. 

SQS:
- Nothing to report.
  180   23 Nov 2020, 11:33 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 23.11.2020
General Announcement:
- An e-mail to the readiness and stgl mailing lists was sent regarding the calendar share and updated task list. Please contact xo-planning@xfel.eu in case of any issue.
- Desy has made mandatory for the tunnel works the use a community mask all the time and a FFP2 mask if distance between two persons is less than 1.5m.

CTRL:
- CTRL sent requests to the DA colleagues. S. Hauf asks the colleagues to take a look and have a discussion afterwards. 
- Karabo deployment on FXE will take place on Wednesday. 
M. Turcato says one DET person must be back to work on that day to switch Detector off and on by again: Wed afternoon or Thursday morning it shall be OK.

DA:
- Nothing to report.

EEE:
- PLCs will start shutdown with planned activities on today and tomorrow in SA3 instruments.

VAC:
- Not present.

XPD:
- Plan for XGM/SCS is to keep it running (under vacuum). XPD reminds SCS people to keep XGM on vacuum. 

DET:
- AGIPD intervention: is progressing quite well and advancing. 
A. Violante says it is progressing faster than expected and many small issues were solved during the weekend; first quadrant was already re-mounted and tested. 
By the end of this week an update on time planning will be made. 

ITDM: 
- Today ITDM starts the update of offline storage system (unavailable all day).
- The update on the online cluster system will start today in SA1 and then SA2 and SA3. It will not be available until Wednesday.

XRO:
- Working on the winter shutdown.

LAS:
- Carrying on planned maintenance work.

UNSYS:
- Not present.

SPB:
- AGIPD intervention is going well and is under control.

FXE:
- Cabling works starting today in the hutch.
- A. Galler asks if calibration of LPD was done. D. Khakhulin mentions that the raw data was used and calibration constants were not used. 
M. Turcato says one has to discuss together with calibration team.

MID:
- All fine.

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  179   18 Nov 2020, 11:23 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 18.11.2020
General Announcement:
- Updates on the maintenance schedule will be made available in Alfresco. Guidelines in how to integrate calendars will be sent to readiness mailing list, for example.
- Please feel free to contact XO by to share any thoughts related to the Maintenance.

CTRL:
- Successful tests of HED pulser were done on Monday;
- CTRL is preparing Karabo 2.10 deployment: a schedule will be sent around per e-mail. FXE starts on 25/11. Remaining areas take place on Tue/Wed/Thu of week 49. 
SPB will be discussed later due to the AGIPD work.

DA:
- DA has received two messages from CTRL related to Karabo and data analysis deployment: first about deployment and second about bridge.

EEE:
- Nothing to report.

VAC:
- Nothing to report.

XPD:
- XPD is involved on MID tests this week;
- Shimadzu camera tests/development is cancelled due to AGIPD intervention at SPB.

DET:
- Nothing to report.

ITDM: 
- Planning for next week maintenance and working with AGIPD/SPB intervention.

XRO:
- Nothing to report.

LAS:
- Nothing to report.

UNSYS:
- A problem with SA2/cell37 coupler is found and will be fixed during shutdown.

SPB:
- Work on AGIPD is going OK. It has been found the expected minor issues. SPB expects AGIPD experts on site.

FXE:
- Nothing to report.

MID:
- Fast-ADC issue is under investigations together with EEE/Bruno F.

HED:
- Nothing to report.

SCS:
- Not present.

SQS:
- Nothing to report.
  178   16 Nov 2020, 13:14 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 16.11.2020
General Announcement:
-	 

CTRL:
- Further improvements on the SA3 vacuum loop will continue during the shutdown.

DA:
- No issues to report.

EEE:
- 1 OCD call from SCS call related to PLC: hardware issue.
- Fast electronics side: MID timing system problem is still present. Frequency that this issue happens is reduced.
- For the Winter maintenance, EEE will ship some crates to Poland. Due to Covid-19 situation, only crates with non-critical changes for next run will be sent. EEE is in 
closed contacts with instruments.

VAC:
- 1 OCD during the weekend was related to GATT and promptly solved.

XPD:
- Nothing to report.

DET:
- Last Friday problem with offline jobs was solved (it had to be followed by IT).
- SPB activity for AGIPD baseline intervention/correction to start today.

ITDM: 
- Maintenance on the offline storage will happen next Monday 23.11. An e-mail was sent this morning. The system will be OFF and all data will not be accessible that day.
- ITDM will also start the maintenance of online cluster on 23.11. They will not be available until the end of maintenance (about 3 days).

XRO:
- Nothing to report.

LAS:
- OCD calls from SCS: 1. observed crystal degradation in SCS side; 2. laser pattern server could not be changed (LAS forward the message to CTRL).
- All other PP-lasers are fine.

UNSYS:
- BKR tested air coils in the last SA2 undulators cells for performing work on Wednesday for photon beam alignment (there was a problem with Cell37/SA2 not responding).

SPB:
- Instrument and beam delivery were running successfully last week;
- This week SPB starts working with AGIPD: one aim to work as fast as possible to finish before Christmas break.

FXE:
- D3 is currently preparing the tests.

MID:
- Continues to use beam this week for detector tests. 

HED:
- Tested AGIPD during last weekend. Currently HED is changing setup to use High Intensity laser this week.

SCS:
- Not present.

SQS:
- Nothing to report.
  177   11 Nov 2020, 10:49 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 11.11.2020
General Announcement:
- XO is in contact with some groups regarding the winter shutdown, in order to get additional information or solve potential conflicts. 
If there are any question, please contact us in this meeting or via e-mail.
- Steffen adds on the idea of using FXE for initial deployment of updated Karabo version. It will take place after 23/Nov.
- Jan asks how XO needs to be informed for tasks that groups deal with each other. A. Galler says XPD is on the list for setting a date to discuss this. 
Groups are encouraged to communicate with other support groups beforehand.

CTRL:
- Misbehaving timing server at MID yesterday was initially solved by EEE. EEE replaced the hardware of the timing system itself but problem continued. 
System seems now to be healthy but from time to time there are small timing interruptions. Investigation is ongoing and will be discussed in the digitizer meeting of Data 
department.
- One CTRL ticket on Doocs device at SCS: it was a problem with custom installation. CTRL asks to be informed in case groups have custom installations. This may pose a 
risk when deploying new updates.

DA:
- AGIPD train-ID issue: a small workaround until shutdown is in place. One has to set the individual channel and seems to be now working. Next intervention to take place 
during Winter maintenance time.

EEE:
- Nothing else to report aside of the MID timing system.

VAC:
- Nothing to report.

XPD:
- SA3 XGM was switched to Ne as requested by instruments.

DET:
- Preparation for activities on AGIPD at SPB/SFX during the winter time is ongoing.

ITDM: 
- Pre-announcement: at the start of the maintenance, ITDM has activity on the computer cluster. A separate announcement will be sent.

XRO:
- Nothing to report.

LAS:
- Nothing to report.

UNSYS:
- Nothing to report.

SPB:
- No issues to report.
- Intervention date of AGIPD: SPB requires due to highest priority work that between 16/11-18/12 no interruptions of electrical power or the need of EQP may happen. 

FXE:
- Crosstalk SA1/SA3 was not seen yesterday as SQS used only 1 bunch.

MID:
- Nothing to add besides what CTRL and EEE have mentioned.

HED:
- Next week HED does not use X-ray beam: commissioning of the high power laser and test of pulse on demand are planned. A. Galler says it pulse on demand was fixed by 
CTRLs and can be used. 
- Question on use of ADCs: Bruno says some trains will not be recorder or be repeated. This may be related to the timing issue like in MID. 
But data can be acquired. It is under investigation accordingly to Bruno, with CTRL, DA, ITDM.

SCS:
- Not present.

SQS:
- Nothing to report.
  176   10 Nov 2020, 10:53 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 09.11.2020
General Announcement:
- Maintenance summary task list has been made available at the S:\ folder as well as in the XO Alfresco page.

CTRL:
- On Thursday laser1 went down: error on deployment related to fix zero train ID.
- Lasers safety deployment is fixed.
- SA3 vac loop: progress was made and new version of Karabo needs re-start this morning.
- SA3 fixed MC2 motor issues. Coordinated with instruments to do tests today.

DA:
- 1 OCD was received and solved during the weekend.

EEE:
- Announces that on tomorrow's PBS there will be a EEE presentation on the Digitizer instruments.

VAC:
- Nothing to report.

XPD:
- No issues. The switch of XGM settings at different energies went well.

DET:
- Nothing to report.

ITDM: 
- Meeting with MID on train-ID is set. There is a mitigation error plan which will be implemented.

XRO:
- Nothing to report. Minor issues on CTRLs but are not stoppers.

LAS:
- PP-laser for SPB run was interrupted for 45 minutes.
- Issue with synchronization / motor becomes unstable, relating to motor slippage. Hardware will be replaced during the shutdown.

UNSYS:
- Nothing to report.

SPB:
- Finished user experiment: went OK but machine not very stable. Intensity and pointing very unstable. After beam losses timing came quite different. 
Data for pointing quantification to be analyzed.

FXE:
- Unstable beam the whole week. 
- Crosstalk between SA3 and SA1: sometimes BKR could fix/work on it, sometimes not. This happens when FXE runs at 150 bunches and SA3 varying 1-100. 
SA1 intensity goes down by factor 2.

MID:
- DET and ITDM contacted during the weekend.

HED:
- Not present.

SCS:
- Not present.

SQS:
- Crosstalk SA1/SA3: night shift crew said it is not a problem for next week if run without Interleaved.
  175   04 Nov 2020, 10:55 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 04.11.2020
General Announcement:
- XO reminds on the 4.5MHz operation: MXL tested a 12hs switching mode and this will be used along the week. 
The night shift operates at 4.5 MHz and day shift at 2.25MHz to avoid sending unnecessary bunches to the dumps.

CTRL:
- Issues with MC2 motors (bug in moving, mainly in SA3). CTRL has a fix but it is hard to find a proper time slot this week to apply it. The idea is to deploy next Monday 
only in SA3 first. 
If issues are found beforehand, please contact CTRL.
- SQS mentions a bug in the mirror observed yesterday which relates to the above described issue. SQS has to observe it to verify it moves all the way and then one can 
wait until Monday for the fix.

DA:
- Nothing to report.

EEE:
- Crate issue in SA2: it was not needed to exchange the crate. A reboot on Monday afternoon helped to bring system back. 
Crate works as normal. During winter shutdown EEE will replace crate and power modules.
- MID and LAS reported issues with train-ID / timing servers. Triggers have not been delivered in 10Hz. This is in discussion with DESY. 
One solution is an update of the DOOCS timing serves that do not show errors which does not interrupt timing servers. In MID this may be related to data recording. 
For the PP Laser further discuss with Desy colleagues will be made. This affects only MID because the use of old version of Doocs server. 
- MCP problems reported by XPD: mismatch on the firmware version and EEE plans to upgrade this week. 
- Preparation for the winter maintenance is ongoing and installations can start on 23/11.

VAC:
- VAC would like to thank EEE and CTRL for the SA3 loop vacuum fix.

XPD:
- XPD found another issue with MCP: wrong assigned triggers. A ticket will be created. 

DET:
- Update of MID AGIPD is still an issue but this is not critical at the moment.
- CTRL has accessed the LPD detector issue.

ITDM: 
- K. Wrona informs ITDM has the reservation of Maxwell for data analysis during next two weeks. 
The user groups have to be informed that one has to use Maxwell reservation to quick analyze data.

XRO:
- Nothing to report.

LAS:
- SA1 SPB PP-laser had to increase energy and is on work.
- SA2 PP-laser missing trigger from around 7PM yesterday. Laser went OFF due to this and luckily LAS was watching it and was able to switch ON quickly. LAS is in touch 
with EEE and Desy. 
A fallback solution could be to generate triggers by hardware but there is no progress from Desy side. LAS will prepare specifications for this module.

UNSYS:
- Small hiccup in one SA2 undulator that was remotely corrected.

SPB:
- Trying to align beam: SPB sees elliptical rings. Andreas G. suggests if this continues please communicate to the PRC.

FXE:
- No further information on issues with the LPD.

MID:
- Tomorrow it takes place the AGIPD train-ID meeting.

HED:
- Vacuum valves in differential pumping system were strangely closed on Friday 2AM. HED discovered this only yesterday. Pressure slowly went up. HED will further 
investigate.

SCS:
- Not present.

SQS:
- 4.5MHz operation had gun trips due to the overheating of the dump. 
F. Wolff-Fabris comments that was due to the radiation alarms on the dumps that removed beam permission by about one hour around 4AM.
  174   03 Nov 2020, 08:39 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 02.11.2020
General Announcement:
- A. Galler reminds the attendance that this meeting is also an opportunity for making comments/questions on the new Covid-19 rules that start to take place.

CTRL:
- Main issue last days was related to the SA3 VAC PLC. The applied fix was left running during the weekend. So far it works faster and reacts faster for diagnostics. 
CTRL is getting some statistics in order to get the final solution. If something goes wrong, please call OCD.

- No OCD calls during the weekend.

DA:
- No issues or news.

EEE:
- uTCA crate of SA2 balcony room is not reachable. A quick fix on Friday was not successful and EEE needs to power cycle or replace the whole crate. 
This will interrupt timing for PP-laser and the signals sent to MPS. Meeting between EEE, lasers, instruments and XO will take place. 
- (Note after meeting: the power cycle of the actual crate will be done today after lunch. If this does not helps then immediately exchange of crate will be done 
afterwards. 
PP-laser will be brought to a safe state and takes about 4-5 hours to restart it).

VAC:
- Nothing to report.

XPD:
- Worked at BKR with MCP detectors. XPD reported to CTRL an issue with ADC - some channels see no signal or not standard ones. 
EEE knows and runs some scripts - this happens from time to time and may affect recording data.
- In general, from XGM side, Theo M. has prepared a way to switch settings to extreme changes of photon energies.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Nothing to report.

UNSYS:
- Motor problem with SA1/cell31 last week was repaired and undulator should be working fine.

SPB:
- STAB campaign last week went fine and collected data that has to be analyzed.

FXE:
- Nothing to add.

MID:
- Waiting for invitation to a meeting on the AGIPD train-ID.

HED:
- Requests the beam delivery schedule in order to know if has to schedule people for Tuesday night.

SCS:
- Not present.

SQS:
- Nothing to report.
  173   29 Oct 2020, 08:11 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 28.10.2020
General Announcement:
-	 

CTRL:
- SA3 VAC loop - 2 approaches to find out the root cause (discussed also with EEE):
1. Thursday 8-11:00AM to test the whole loop by partitioning batches of devices. This may pinpoint the single device that cause the problem.
2. Reproduce the problem at lab level. Based on the reproducibility this approach will be continued in parallel. 

DA:
- Nothing to report.

EEE:
- PLC colleagues are troubleshooting SA3 VAC loop.
- DET reported that measurements are carrying on wrong train-IDs. Tests are being made in the lab (not reproducible) but EEE is working on it.

VAC:
- SA3 VAC loop is main activity - interaction with Michaela P.

XPD:
- Group mainly busy with Photon Diag Conference this week.
- Supporting STAB and not heard any major issues in this effort.

DET:
- Report on the train-ID. This affects all the AGIPDs including the mini-half.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Preparing for next week operation.

UNSYS:
- Nothing to report.

SPB:
- Collected lot of data last few days: SPB/SFX thanks LAS and accelerator teams for support.

FXE:
- Nothing to report.

MID:
- All fine.

HED:
- All fine.

SCS:
- Issue with door interlock: Eric B. has investigated this morning. Problem seems to be with the maglocks and has been solved for now but a long term has to be worked on.

SQS:
- Nothing to report.
  172   26 Oct 2020, 11:18 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 26.10.2020
General Announcement:
-	 

CTRL:
- SA3 vacuum loops problems: seems to compromise some experiments. Beckhoff loops are changing states very quickly and compromises the whole system. 
CTRL ask for the instruments to give info on the type of command given that makes motors change states. CTRL investigates together with EEE as more data has been produced 
as compared to the past. 
This problem does not happen in SA1/SA2 and may be related to the PES which will be checked this afternoon.
- Please call the new OCD numbers: 6541 and 6542. Specially on weekends it is the fast way to contact. This number is valid since last spring.

DA:
- Nothing to report.

EEE:
- Nothing to report.

VAC:
- Nothing to report except of SA3 vacuum loops.

XPD:
- Not present.

DET:
- XO asks on the train-ID progress of Agipd. Monica T. says that the plan is to correct in the DAQ level during the next shutdown. 
K. Wrona adds that the fixing takes long and the current investigation focus on doing in the DAQ level during January.

ITDM: 
- Maxwell nodes used to login in the Maxwell will be rebooted today, causing access limitations.

XRO:
- Nothing to report.

LAS:
- Bunch pattern in demand: LAS assume XO will take care of this topic. Andreas G. says XO will take care but need to be informed on what is needed. 
Tomasz J. mentions it needs some coordination between instruments and machine.

UNSYS:
- Couple of issues in SA3 undulator cell: it got stuck, mostly during machine development times by running out of range. This was corrected during the weekend.

SPB:
- Moving on with WP8. Continuing today.

FXE:
- Nothing to add.

MID:
- MID makes a general reminder on the train-ID development with AGIPD: MID expects to have a meeting to discuss a long term solution plus the test procedure for 
implementation after the quick fix. 
This has to be planned in advance.

HED:
- Nothing to report.

SCS:
- Nothing to report.

SQS:
- SQS has suffered during the weekend with the VAC issue.
- CTRL side can look at the PES this afternoon.
  171   21 Oct 2020, 11:11 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 21.10.2020
General Announcement:
- XO thank the effort made by preparing the activity lists for the maintenance period. In few cases XO is contacting groups back with questions and/or comments.
- The Winter maintenance period was adjusted yesterday. The ending period is now on January 13th, 2021, instead of January 4th. 

CTRL:
- CTRL is reaching conclusions on recent Beckhoff problems: 
1. CTRL would like to upgrade most of installations in instruments next week (upgrade of motors software). This upgrade is in general a step forward for the PLC 
connections when it is dropped (re-starting). 
2. For SASE installations, CTRL will make next week during machine days. 
3. The GATT will not be upgraded now – this will start during Winter maintenance period. Upgrade of GATT is a priority for CTRL (has been added to the maintenance list 
submitted to XO). 

DA:
- Nothing to report.

EEE:
- Intervention in SA3 PP-laser was done.

VAC:
- No new issues to report. 
- Maintenance on SA3 vacuum loop: in case of problems, one has to restart in PLC level with EEE staff and may result a bit longer recovery. 
Steffen H. comments it has been worked on and may be solved in the next day or so. The restart procedure is easy but lengthy. CTRL/OCD knows what to do.

XPD:
- One of vacuum pumps lost Beckhoff connection.

DET:
- Nothing to report.

ITDM: 
- Nothing to report.

XRO:
- Nothing to report.

LAS:
- Nothing to report.

UNSYS:
- Nothing to report.

SPB:
- Not present.

FXE:
- Nothing to report.

MID:
- All is super at MID.

HED:
- All sounds great at the moment.

SCS:
- Nothing to report.

SQS:
- Nothing to report.
  170   19 Oct 2020, 15:29 Frederik Wolff-FabrisGeneralInformationNotes XO Integration Meeting 19.10.2020
General Announcement:
- A Vacuum issue in one of the dumps brought machine down at around 1:30AM. Only around 7AM machine could re-start running and tuning again.
- XO thanks the groups who have already sent the maintenance lists. For the remaining groups, please contact at the xo-planning@xfel.eu.

CTRL:
- Few minor OCD calls during the weekend: One relates to a camera triggering.
- Issue with SA3 vacuum loop seems to be currently fixed but needs further investigations.
- Update of Beckhoff motors is planned for this week. CTRL will get in touch with Instruments.

DA:
- Nothing special to report.

EEE:
- Nothing to report.
- Question to XO on Maintenance: Could we have a draft of tasks list before Nov. 6th? 
Andreas G. answers that XO is working on it and as soon as lists are finished this will be published. In case info on partial parts is already needed, may be provided 
depending on the progress.

VAC:
- SA3 vacuum loop issue was mentioned by CTRL (see above).

XPD:
- Nothing to report.

DET:
- Nothing special to report.

ITDM: 
- Update will be done on DA acquisition of HED. This has been communicated with instrument personnel.

XRO:
- No bad news to report.

LAS:
- SA3 PP-laser presents small power drop in the laser beam: LAS will take care this week.
- SA1 PP-laser has a problem with ADC and stop acquiring data.
- Need to update firmware of ADC cards to get train ID's in SA3 PP-laser. This was requested to EEE. This may bring laser down again and may not be ready by Thursday. XO, 
LAS and EEE will discuss offline. 

UNSYS:
- Nothing to report.

SPB:
- Last week was smooth.

FXE:
- Problems with tunnels last night at around 1:00AM. More than half the shift was lost with no beam delivery: main user data was therefore not acquired. 

MID:
- Nothing to report.

HED:
- Beam was lost between 1AM to 7AM. Last data has been taken between 8:00 and 9:00AM. 
- During the weekend, beam drift was observed at Pop-in at the end of the tunnel. There are data and has to be analyzed.
- Discussion on implementation of Dynamic trigger will take place.

SCS:
- Not present.

SQS:
- Minor issue with Data: one data aggregator was not working properly. OCD and SQS personnel have solved it. 
K. Wrona comments that too many properties overloaded the aggregator. A long term solution will be worked on.
  169   14 Oct 2020, 09:22 Andreas GallerGeneralInformationNotes XO Integration Meeting 14.10.2020

General Announcement:

  • .

CTRL:

  • problems in SA3
    • VAC loop unstable, devices cannot be brought into a stable state --> what used to be a restart every  two weeks, cannot be recovered these days
    • issues with motors developing a short time out
      • somewhere there is a beavy load on Beckhoff loops
      • TM comments: at times the movements happen very late after issuing the command
    • influxDB sees high load
      • any modification of or running a new macro?
      • Please let CTRL know
    • the issue started with VAC --> devugging shoud start there

DA:

  • extrafoam OCD call

EEE:

  • working together with CTRL on SA3 issue
  • OCD call from SCS but call was not correctly frowarded

VAC:

  • not present

XPD:

  • no issues

DET:

  • OCD call from SCS --> was fixed

ITDM:

  • nothing to report

XRO:

  • nothing to report

LAS:

  • no issues

UNSYS: 

  • nthing to report

SPB/SFX:

  • all fine

FXE:

  • people on shift could not initiate a hutch search
  • GABS access right missing (Gebietsabsuche)
    • introduced without informing FXE

MID:

  • waiting for invitation AGIPD trainID solution

HED:

  • JUNGFRAU issue presists
  • HexaPOD is working again
    • work around

SCS:

  •  

SQS:

  • DAQ issue
    • no data was recorded without
  • vac is an issue but not endangering
  • motor issue is vital to the instrument

ELOG V3.1.4-7c3fd00