Dear all,

 the emergency power off test has been successfully performed in SPB.

Nobody can instantly cut the power to non hazardous racks (see racklist.xlsx managed by Ulf Brueggmann) by pushing the emergency off button anymore. The outlets in the experiment hutch do not go off. This is due to some missing cables. These will be installed in the next weeks and a new test will be planned for next shutdown.

Furthermore three racks for D.02 (SPB optics) and two racks for D.13 (SPB laser) cannot be powered off anymore by pushing the emergency off in D.09. The missing two emergency power off switching devices will be mounted in UV-XHEXP1-01-13-AV in the next shutdown. Until then the power to those five racks (D03-UH1 - D03-UH3, D13-UH1, D13-UH2) cannot be cut by emergency off buttons in D.02 or D.13.

Summary of Karabo 2.2.3 deployment dates:

• Karabo 2.2.3 deployed in SASE3 tunnels last friday ==> no major issues. DAQ can be deployed
• Karabo in SASE2 Tunnels to be deployed today at 14:00
• Karabo in SASE1 Tunnels to be deployed tomorrow at 08:00
• Karabo in SPB/SFX experiment to be deployed today and tomorrow
• Karabo in FXE experiment to be deployed on Wednesday

SPB

• By tomorrow the mirror installation will be done.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.

ITDM

• Infiniband port fixing ongoing
• GPFS bug to be fixed with a patch that should be issued on 24.04. To be checked whether it is still OK to install the patch or it is too late.

General

• PSPO will ask HSinn for a detailed schedule concerning closing of the tunnels to fine tune the activities going on this week in the SASE2 and 3 tunnels

Dear all,

I was just informed by Ulf Brueggmann that the emergency power off test was successfully performed in FXE.

The  reality is now reflecting the planned logic, exeption being the power plugs in the experiment hutch. They do not go off. This is due to some missing cables. These will be installed in the next weeks and a new test will be planned for next shutdown.

The tests will continue tomorrow in SPB/SFX.

This entry is on behalve of Frank Babies, if you have any doubt about it, contact him:

On 16.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from FXE
"fxe-rr-sys-utc-1"
"fxe-rr-sys-utc-2"
because of the Power Down issue on Monday.

On 17.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from SPB
"spb-rr-sys-utc-1"
"spb-rr-sys-utc-2"
because of the Power Down issue on Tuesday.

This will affect timing system and MPS communication.

Interlock systems SASE3 - TÜV test

The pre test by D3 will be held on 20.06.

On 05.07.The TÜV Test for SASE3 will be done together with the SASE2 test and the repetition of the SASE1 test

Emergency power test will be performed at the following dates:

FXE - Mon. 16.04.

SPB - Tue. 17.04. 

Schedule

• 10:00 Power will go off in SASE1 - Only redundant power line will stay ON
• Reprogramming of the PLC of the Emergency power off system - Estimated 1.5 Hrs.
• Power ON again ==> Emergency OFF test will be carried out. (info from AE: AE PLCs will not go off. uTCA crates will be shut down before)
• About at 15:00 everything should be back to normal.

For technical questions contact T. Schoen and U. Brueggmann.

changes according to the racklist on the electrical power distribution cabinets for rackrooms E.21 and E.22 are done. Power supply should be stable now. Please note that there will be emergency power off tests prior to the final acceptance test of SASE3 electrical installations around mid of May. You will be notified about the exact time and date.

Hi all, for your information below a message from Chris Youngman concerning Karabo 2.2.3 installations in SASE3.

"preparation work has concluded and we will definitely try to deploy Karabo 2.2.3 on SA3 machines TODAY 13.04.. SA3 s/w will not work from 8:00 on. Hopefully we will finish during the morning.

SA1 and SA2 installations should continue as normal."

Die Arbeiten an der Kühlwassersteuerung konnten innerhalb des genannten

Zeitraums abgeschlossen werden,

so dass die Kühlwasserversorgung nun schon einige Zeit wieder geregelt

läuft.

Großer Dank an alle Beteiligten, die den Ablauf der Abschaltung meines

Wissens auswirkungsfrei gestalten konnten.

Wir bitten über kurze Rückmeldungen zur Optimierung, sollten

Schwierigkeiten im Zusammenhang mit der Abschaltung entstanden sein.

Mit freundlichen Grüßen

Jörg Hanik

SPB

1. Detector z motion would be needed. AE checks the status of the crate. still pending
2. MIrrors installation ongoing. NO ACCESS to D.02!!!!
3. Laser Interlock change of logic by K.Witt will be done on monday at 9:00
4. Karabo update installation dates being agreed with CAS. Probably early next week.

FYI

Below the E-Mail from Hugo Santos with the Karabo deployment dates.

Dear All,

I would like to call your attention to the fact that we made Karabo 2.2.3 available last Monday (04/04, and the changelog 06/04). From the last Readiness Meeting, I got informed that we agreed and are allowed to proceed with the Karabo rollout planning on SASE3 and SASE2. (SASE1 still needs agreement):

1. SASE3
   1. PRE-CONDITION: (AE) PLC version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed. (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 13/04/2018 (Friday)
   3. Should be the first one because of the completeness of this system: VAC, PES, XGM and GAT loops, for instance, that have to be checked and run properly afterward.
   4. Downtime expected: 9:00 to 14:00. 
   5. (AE) PLCs version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed.
   6. No major change regarding DET systems
   7. CAS Members for this specific task: Chris and Wajid.
2. SASE2
   1. PRE-CONDITION: SASE3 Deployment task must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 16/04/2018 (Monday)
   3. Downtime expected: 11:00 to 13:00.
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET system.
   6. CAS Members for this specific task: Chris and Wajid.
3. SASE1
   1. PRE-CONDITION: SASE2 and SASE3 Deployment tasks must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues).
   2. Date: (17/04/2018 - Tuesday - to be agreed)
   3. Downtime expected: to be defined. 
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET systems.
   6. CAS Members for this specific task: Alessandro and Wajid

Finally, please, also see that as soon as we have the Karabo 2.2.3 running during the shutdown, we will have the most comfortable time to perform the setup, configurations and stabilize it, still, during the shutdown. This way we can handover the control system properly to the operators.

Please, if you do not agree or are not comfortable with something in this message, let me know as soon as possible. Otherwise, I will assume that it does not contain any showstopper and we (CAS) can proceed.

H.S.

General

Emergency off test in SPB is shifted. Date to be agreed on Friday. FXE test stays on 16.04. TS will confirm that.

MKK will shut down the cooling water for Tunnels and XHEXP1 on Thursday startin from 11:00 for maintenance purposes.

1. Based on the information we have the intervention could last up to 2 hrs. UBrueggmann is trying to check whether is possible to limit the duration of the intervention to bunches of 45 min. This could avoid the shut down of the Turbo pumps.
2. UBrueggmann is taking care of opening the rack doors in the tunnels (all of them!) and close them again when the water is back. 
3. The experiments should take care of opening the doors of the racks in the rackrooms. Also the main door.
4. AV will speak with the important groups involved and then send a summary e-mail to staff today afternoon.

SPB

1. Detector z motion would be needed. AE checks the status of the crate
2. Quadrant motion - Suggestion from AE is to perform tests and implement it in the next shutdown.
3. MIrrors installation ongoing. NO ACCESS to D.02!!!!
4. Laser Interlock/Rad. Saf. interlock issue still being investigated.  No news from SRP.

SASE2 -CYoungman

Goal is to stop the HW changes by the end of Wed. Thursday final loops and firmware will be updated. Both in SASE2 and 3.

Due to some delay in the installation of some necessary cables, the emergency power off test foreseen for today 06.04. in the SPB experiment will not take place.

The details will be reported by Ulf Brueggmann today in the dispatch meeting at 13:00. The needed cabling should be done this week, so that the test in FXE planned for Monday, 16.04 CAN take place.

TS will confirm the FXE date and propose the nex date for SPB this Friday in the Readiness meeting.

On Friday, 6.04, there will be a maintenance of the online storage system. This will tackle the issues we had just before the last user beamtimes, when we decided to apply workaround to keep the system working. The availability of the storage system is not guaranteed, especially when faulty h/w parts needs to be exchanged.

Best, 

Janusz

Tomorrow, 5.04, between 9:30 and 12:00 there will be urgent maintenance of one of the two redundant main switches for the control network. The network connectivity during that time is not guaranteed, but also any major interruptions are not expected. For the two switches in sase1 tunnels with only one uplink, there could be short (~1min) interruption.

Best, 

Janusz

SPB/SFX:

• Mirror installation next week

FXE:

• Replaced broken diamond-window

SCS:

• Installation of alignment laser
• Mirror tank installation next week

SQS:

• Not present / no issues.

SASE tunnels:

• C. Youngman is coordinating the works in the SASE1, 2 and 3 tunnels, daily meetings are at 9:30 every morning.
• Works have been scheduled and planned, most work to happen in SASE3 and SASE2.
• Staff should be aware that works e.g. regarding the vacuum loops have been scheduled and are ongoing, in doubt contact C. Youngman, N. Jardon or V. Bondar

CAS:

• Release test with Karabo 2.2.3 are concluding probably today, including test with PLC firmwares 1.26 and 1.27.
• This would lead to a release for ITDM & Detectors by this evening. A timeslot is required to test the new Karabo with ITDM / DAQ.
• Deployment at the beamline could potentially be in two weeks, if the test with ITDM and Detectors will conclude fine.

XPD:

• Email to xfel-op-allhands was received, indicating that there could have been a shortage of cooling water for the magnets and racks supplied from XS4 and also XS2 (should be SASE2-branch only). If you have electronic racks in this area, check the temperature, email will be forwarded to readiness.

Detectors:

• A single supermodule of the LPD-detector is flickering, is being investigated.

ITDM:

• A rise of the temperature in the balcony room from 27° to 39° was observed in the morning from ca. 6:00 until 6:30.
• This lead to damage at two power supplies.
• There was no indication by MKK or TS about any kind of shortage or shutdown of the cooling water supply.

SPB

• Sample delivery installed. Tonight will be commissioned. Experiment set-up starts tomorrow night

FXE

• Digitizers were already implemented in the DAQ but there is an issue related to the Karabo device. CAS already looking into that.

AE

• Follow up to the issue of re-setting pumps after interlock is triggered: loops updated. 
• SASE2 MOV loop patched
• Imagers in XTD1 have to be tested with Karabo

ITDM

• On monday afternoon problems with the storage servers were reported. The issue is a defective board, whoch is meanwhile been deactivated. IBM is working on that. The system seems to be working in this state.

CAS

• The next Karabo release will be able to handle better big loops. ASilenzi points out that the next update is really important since it adresses problems like the restart of time servers issue.

Readiness Meeting Issue list

• Item 2 (XGM SASE3): After restarting of the CPP servers the XGM turbo pumps stopped, valves closed and it required a manual intervention to restore them. ==> Firmware updated. CLOSED

Operation Meeting Issuel List

• Item 1 (Karabo rollout process): Testing steps of a new Karabo release have to be discussed with AE-ITDM-DET ==> Meeting took place. Closed

Update regarding Digitizer:

----- Forwarded Message -----
From: "hamed sotoudi namin" <hamed.sotoudi.namin@xfel.eu>
To: "Sergey Esenov" <serguei.essenov@xfel.eu>, "andre" <andreas.galler@xfel.eu>
Cc: "Djelloul Boukhelef" <djelloul.boukhelef@xfel.eu>, "Dennis Goeries" <dennis.goeries@xfel.eu>, "Leonce Mekinda" <leonce.mekinda@xfel.eu>, "Patrick Gessler" <patrick.gessler@xfel.eu>, "Hugo Santos" <hugo.santos@xfel.eu>, "Krzysztof Wrona" <krzysztof.wrona@xfel.eu>, fxe-instrument@xfel.eu
Sent: Wednesday, 28 March, 2018 17:37:28
Subject: RE: adqDigitzer

Hi everybody,

Today morning me, Andreas and Leonce have tested digitizer with software in 
FXE hutch. We tried the same configuration as it was and checked the log 
file, we observed that software was missing synchronization, that means 
either packets from digitizer have some problems ( missed or corrupted) , or 
buffers are too small in software. We reinitialized everything and problem 
disappeared ( at least for synchronization).
This afternoon I tested digitizer in our lab in APD mode with Raw data mode 
and also checked values; everything was fine. Because till today we have 
tested digitizer in fully load and enabled APD and raw data for all channels 
we observed no issues. I tried to set the same configuration which Andreas 
applied in their system ( APD for one channel and raw data for all channels) 
I encountered the same problems that he had mentioned in Email. Conclusion 
is that Digitizer has no problem when Algorithms are enabled for all 
channels. But when one algorithm is enabled for only one channel there is 
problem in packets, or even there is no packet from digitizer sometimes. 
Which gives timeout error.
Current solution is that to enable required algorithm for all channels and 
ignore result from unwanted ones.
I do really apologize for this inconvenience caused and will do my best to 
solve the problem in Firmware as soon as possible.

Best,
Hamed

Info from BKR:  the southern tunnels were opened, thus XTD1 can be accessed now.

General

Due to the triggering of the emergency power off in FXE caused by electrical works being done on Sunday, FXE and SPB lost beam and electronics was switched off. In general the efficiency of the communication between groups has to be impoved. An important first step is the publication to a easy accessible site of the OCDs phone numbers list. PSPO will organize that.

Comment by PGessler

"It was clear to all, that currently the impact on operation due to emergency power off is severe, since the Racks with the PLC CPUs as well as the MicroTCA systems will be also switched off, what is not what was defined/specified. Therefore it was agreed, that no work on the power distribution/emergency off will be done until the defined date (for FXE 16th April). That work was done there nevertheless is extremely careless. The fact, that this was done on a Sunday and also during user time makes it even much worse then that.  It has to be completely followed up and made clear, that this is not acceptable and will not be repeated again!"

SPB

A run of 14 hrs acquiring data was a sort of stress test for the DAQ. It worked. ITDM would like to remove the data. Should be ok.

Due to the triggering of the emergency power off also SPB lost the beam. This was sort of strange since the power off signal was triggered in FXE. AE reported that this behavior is to be expected with the current configuration. This has to be discussed.

FXE

DAQ is recovering from the power off. 

Tripping of the interlock caused again by control unit of ion pumps in the week end. During the shutdown the responsible units will be replaced.

In general is is agreed to prepare an overview interlcok panel in the BKR. AE is on boat, CAS to be checked.

500 kHz operation is problematic for the machine due to heavy losses. They are only able to deliver 1 bunch at this rate. FXE will discuss this directly with the PRC (this week is Jan Gruenert)

AE related

Follow up to the issue of re-setting pumps after interlock is triggered: VAC and EPS loops need to be updated. HSinn was pushing to have it done as soon as possible without waiting for the shutdown. This is not riskless for the experiments. This will be discussed in a small round (AE, PRC, SPB, FXE)

ITDM

Due to the power outage many devices do not have proper train id. CAS should check it. Time servers also need to be checked.

FXE needs to use digitizers for the upcoming measurements session. These were not implemented in the DAQ. ITDM/FXE/CAS meeting needed to clarify and solve the issue .

 DET

LPD calibration had a problem but should be ok now. 

More supermodules coming.

Communication with upcoming userve concerning data format and other issue has to be improved according to the DET group. SPB suggests to tackle the issue offline.

Dear all,
apparently the Emergency Power-Off at FXE was triggered at around 10:00 - 10:30 (my estimate). As far as I know power has been switched on again. BKR / PRC have been informed. More information will be provided by TS / Torsten Schön soon.

Edit: T. Schoen is on site and can be contacted under 0160 977 287 97

Regard, Gerd