No new issues to be discussed where reported by any groups.

The status and issues reported on friday are still valid. 

SPB

On monday there was an issue with a detector related PLC due to misconnection of one EtherCAT cable. The issue has been solved and it "just" resulted in 3 of hours delay in  the AGIPD calibration activity.

FXE, SCS, SQS, XPD, XRO,ITDM no issues reported

AE related

uTCA in XTD 6 are installed 

VAC-Related

Marc Messerchmidt reported  abouit an entry in the DESY Logbook about a pump that triggered the interlock in the SASE1 tunnels. According to Nerea this this seems to be related to the old issue already known. VAC group is working on that. 

 CAS

CYoungman is working on the priority list for the components in SASE2 tunnels. Double checking with HSinn happening soon.

Access for Tue/Wed next week to XTD1 will be requested by CY through HS

Beam is in SPB/FXE this week due to the HERCULES project.

General

Due to the triggering of the emergency power off in FXE caused by electrical works being done on Sunday, FXE and SPB lost beam and electronics was switched off. In general the efficiency of the communication between groups has to be impoved. An important first step is the publication to a easy accessible site of the OCDs phone numbers list. PSPO will organize that.

Comment by PGessler

"It was clear to all, that currently the impact on operation due to emergency power off is severe, since the Racks with the PLC CPUs as well as the MicroTCA systems will be also switched off, what is not what was defined/specified. Therefore it was agreed, that no work on the power distribution/emergency off will be done until the defined date (for FXE 16th April). That work was done there nevertheless is extremely careless. The fact, that this was done on a Sunday and also during user time makes it even much worse then that.  It has to be completely followed up and made clear, that this is not acceptable and will not be repeated again!"

SPB

A run of 14 hrs acquiring data was a sort of stress test for the DAQ. It worked. ITDM would like to remove the data. Should be ok.

Due to the triggering of the emergency power off also SPB lost the beam. This was sort of strange since the power off signal was triggered in FXE. AE reported that this behavior is to be expected with the current configuration. This has to be discussed.

FXE

DAQ is recovering from the power off. 

Tripping of the interlock caused again by control unit of ion pumps in the week end. During the shutdown the responsible units will be replaced.

In general is is agreed to prepare an overview interlcok panel in the BKR. AE is on boat, CAS to be checked.

500 kHz operation is problematic for the machine due to heavy losses. They are only able to deliver 1 bunch at this rate. FXE will discuss this directly with the PRC (this week is Jan Gruenert)

AE related

Follow up to the issue of re-setting pumps after interlock is triggered: VAC and EPS loops need to be updated. HSinn was pushing to have it done as soon as possible without waiting for the shutdown. This is not riskless for the experiments. This will be discussed in a small round (AE, PRC, SPB, FXE)

ITDM

Due to the power outage many devices do not have proper train id. CAS should check it. Time servers also need to be checked.

FXE needs to use digitizers for the upcoming measurements session. These were not implemented in the DAQ. ITDM/FXE/CAS meeting needed to clarify and solve the issue .

 DET

LPD calibration had a problem but should be ok now. 

More supermodules coming.

Communication with upcoming userve concerning data format and other issue has to be improved according to the DET group. SPB suggests to tackle the issue offline.

Info from BKR:  the southern tunnels were opened, thus XTD1 can be accessed now.

SPB

• Sample delivery installed. Tonight will be commissioned. Experiment set-up starts tomorrow night

FXE

• Digitizers were already implemented in the DAQ but there is an issue related to the Karabo device. CAS already looking into that.

AE

• Follow up to the issue of re-setting pumps after interlock is triggered: loops updated. 
• SASE2 MOV loop patched
• Imagers in XTD1 have to be tested with Karabo

ITDM

• On monday afternoon problems with the storage servers were reported. The issue is a defective board, whoch is meanwhile been deactivated. IBM is working on that. The system seems to be working in this state.

CAS

• The next Karabo release will be able to handle better big loops. ASilenzi points out that the next update is really important since it adresses problems like the restart of time servers issue.

Readiness Meeting Issue list

• Item 2 (XGM SASE3): After restarting of the CPP servers the XGM turbo pumps stopped, valves closed and it required a manual intervention to restore them. ==> Firmware updated. CLOSED

Operation Meeting Issuel List

• Item 1 (Karabo rollout process): Testing steps of a new Karabo release have to be discussed with AE-ITDM-DET ==> Meeting took place. Closed

Update regarding Digitizer:

----- Forwarded Message -----
From: "hamed sotoudi namin" <hamed.sotoudi.namin@xfel.eu>
To: "Sergey Esenov" <serguei.essenov@xfel.eu>, "andre" <andreas.galler@xfel.eu>
Cc: "Djelloul Boukhelef" <djelloul.boukhelef@xfel.eu>, "Dennis Goeries" <dennis.goeries@xfel.eu>, "Leonce Mekinda" <leonce.mekinda@xfel.eu>, "Patrick Gessler" <patrick.gessler@xfel.eu>, "Hugo Santos" <hugo.santos@xfel.eu>, "Krzysztof Wrona" <krzysztof.wrona@xfel.eu>, fxe-instrument@xfel.eu
Sent: Wednesday, 28 March, 2018 17:37:28
Subject: RE: adqDigitzer

Hi everybody,

Today morning me, Andreas and Leonce have tested digitizer with software in 
FXE hutch. We tried the same configuration as it was and checked the log 
file, we observed that software was missing synchronization, that means 
either packets from digitizer have some problems ( missed or corrupted) , or 
buffers are too small in software. We reinitialized everything and problem 
disappeared ( at least for synchronization).
This afternoon I tested digitizer in our lab in APD mode with Raw data mode 
and also checked values; everything was fine. Because till today we have 
tested digitizer in fully load and enabled APD and raw data for all channels 
we observed no issues. I tried to set the same configuration which Andreas 
applied in their system ( APD for one channel and raw data for all channels) 
I encountered the same problems that he had mentioned in Email. Conclusion 
is that Digitizer has no problem when Algorithms are enabled for all 
channels. But when one algorithm is enabled for only one channel there is 
problem in packets, or even there is no packet from digitizer sometimes. 
Which gives timeout error.
Current solution is that to enable required algorithm for all channels and 
ignore result from unwanted ones.
I do really apologize for this inconvenience caused and will do my best to 
solve the problem in Firmware as soon as possible.

Best,
Hamed

SPB/SFX:

• Mirror installation next week

FXE:

• Replaced broken diamond-window

SCS:

• Installation of alignment laser
• Mirror tank installation next week

SQS:

• Not present / no issues.

SASE tunnels:

• C. Youngman is coordinating the works in the SASE1, 2 and 3 tunnels, daily meetings are at 9:30 every morning.
• Works have been scheduled and planned, most work to happen in SASE3 and SASE2.
• Staff should be aware that works e.g. regarding the vacuum loops have been scheduled and are ongoing, in doubt contact C. Youngman, N. Jardon or V. Bondar

CAS:

• Release test with Karabo 2.2.3 are concluding probably today, including test with PLC firmwares 1.26 and 1.27.
• This would lead to a release for ITDM & Detectors by this evening. A timeslot is required to test the new Karabo with ITDM / DAQ.
• Deployment at the beamline could potentially be in two weeks, if the test with ITDM and Detectors will conclude fine.

XPD:

• Email to xfel-op-allhands was received, indicating that there could have been a shortage of cooling water for the magnets and racks supplied from XS4 and also XS2 (should be SASE2-branch only). If you have electronic racks in this area, check the temperature, email will be forwarded to readiness.

Detectors:

• A single supermodule of the LPD-detector is flickering, is being investigated.

ITDM:

• A rise of the temperature in the balcony room from 27° to 39° was observed in the morning from ca. 6:00 until 6:30.
• This lead to damage at two power supplies.
• There was no indication by MKK or TS about any kind of shortage or shutdown of the cooling water supply.

Tomorrow, 5.04, between 9:30 and 12:00 there will be urgent maintenance of one of the two redundant main switches for the control network. The network connectivity during that time is not guaranteed, but also any major interruptions are not expected. For the two switches in sase1 tunnels with only one uplink, there could be short (~1min) interruption.

Best, 

Janusz

On Friday, 6.04, there will be a maintenance of the online storage system. This will tackle the issues we had just before the last user beamtimes, when we decided to apply workaround to keep the system working. The availability of the storage system is not guaranteed, especially when faulty h/w parts needs to be exchanged.

Best, 

Janusz

General

Emergency off test in SPB is shifted. Date to be agreed on Friday. FXE test stays on 16.04. TS will confirm that.

MKK will shut down the cooling water for Tunnels and XHEXP1 on Thursday startin from 11:00 for maintenance purposes.

1. Based on the information we have the intervention could last up to 2 hrs. UBrueggmann is trying to check whether is possible to limit the duration of the intervention to bunches of 45 min. This could avoid the shut down of the Turbo pumps.
2. UBrueggmann is taking care of opening the rack doors in the tunnels (all of them!) and close them again when the water is back. 
3. The experiments should take care of opening the doors of the racks in the rackrooms. Also the main door.
4. AV will speak with the important groups involved and then send a summary e-mail to staff today afternoon.

SPB

1. Detector z motion would be needed. AE checks the status of the crate
2. Quadrant motion - Suggestion from AE is to perform tests and implement it in the next shutdown.
3. MIrrors installation ongoing. NO ACCESS to D.02!!!!
4. Laser Interlock/Rad. Saf. interlock issue still being investigated.  No news from SRP.

SASE2 -CYoungman

Goal is to stop the HW changes by the end of Wed. Thursday final loops and firmware will be updated. Both in SASE2 and 3.

FYI

Below the E-Mail from Hugo Santos with the Karabo deployment dates.

Dear All,

I would like to call your attention to the fact that we made Karabo 2.2.3 available last Monday (04/04, and the changelog 06/04). From the last Readiness Meeting, I got informed that we agreed and are allowed to proceed with the Karabo rollout planning on SASE3 and SASE2. (SASE1 still needs agreement):

1. SASE3
   1. PRE-CONDITION: (AE) PLC version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed. (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 13/04/2018 (Friday)
   3. Should be the first one because of the completeness of this system: VAC, PES, XGM and GAT loops, for instance, that have to be checked and run properly afterward.
   4. Downtime expected: 9:00 to 14:00. 
   5. (AE) PLCs version and (ITDM) DAQ should be interface-compatible, otherwise, the deployment will not be able to be performed.
   6. No major change regarding DET systems
   7. CAS Members for this specific task: Chris and Wajid.
2. SASE2
   1. PRE-CONDITION: SASE3 Deployment task must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues)
   2. Date: 16/04/2018 (Monday)
   3. Downtime expected: 11:00 to 13:00.
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET system.
   6. CAS Members for this specific task: Chris and Wajid.
3. SASE1
   1. PRE-CONDITION: SASE2 and SASE3 Deployment tasks must be done and Karabo 2.2.3 running with PLC and DAQ (No Karabo-DAQ and Karabo-PLC interface issues).
   2. Date: (17/04/2018 - Tuesday - to be agreed)
   3. Downtime expected: to be defined. 
   4. (AE) PLCs version and (ITDM) DAQ should be compatible, otherwise, the deployment will not be able to be performed.
   5. No major change regarding DET systems.
   6. CAS Members for this specific task: Alessandro and Wajid

Finally, please, also see that as soon as we have the Karabo 2.2.3 running during the shutdown, we will have the most comfortable time to perform the setup, configurations and stabilize it, still, during the shutdown. This way we can handover the control system properly to the operators.

Please, if you do not agree or are not comfortable with something in this message, let me know as soon as possible. Otherwise, I will assume that it does not contain any showstopper and we (CAS) can proceed.

H.S.

SPB

1. Detector z motion would be needed. AE checks the status of the crate. still pending
2. MIrrors installation ongoing. NO ACCESS to D.02!!!!
3. Laser Interlock change of logic by K.Witt will be done on monday at 9:00
4. Karabo update installation dates being agreed with CAS. Probably early next week.

Die Arbeiten an der Kühlwassersteuerung konnten innerhalb des genannten

Zeitraums abgeschlossen werden,

so dass die Kühlwasserversorgung nun schon einige Zeit wieder geregelt

läuft.

Großer Dank an alle Beteiligten, die den Ablauf der Abschaltung meines

Wissens auswirkungsfrei gestalten konnten.

Wir bitten über kurze Rückmeldungen zur Optimierung, sollten

Schwierigkeiten im Zusammenhang mit der Abschaltung entstanden sein.

Mit freundlichen Grüßen

Jörg Hanik

Hi all, for your information below a message from Chris Youngman concerning Karabo 2.2.3 installations in SASE3.

"preparation work has concluded and we will definitely try to deploy Karabo 2.2.3 on SA3 machines TODAY 13.04.. SA3 s/w will not work from 8:00 on. Hopefully we will finish during the morning.

SA1 and SA2 installations should continue as normal."

This entry is on behalve of Frank Babies, if you have any doubt about it, contact him:

On 16.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from FXE
"fxe-rr-sys-utc-1"
"fxe-rr-sys-utc-2"
because of the Power Down issue on Monday.

On 17.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from SPB
"spb-rr-sys-utc-1"
"spb-rr-sys-utc-2"
because of the Power Down issue on Tuesday.

This will affect timing system and MPS communication.

Summary of Karabo 2.2.3 deployment dates:

• Karabo 2.2.3 deployed in SASE3 tunnels last friday ==> no major issues. DAQ can be deployed
• Karabo in SASE2 Tunnels to be deployed today at 14:00
• Karabo in SASE1 Tunnels to be deployed tomorrow at 08:00
• Karabo in SPB/SFX experiment to be deployed today and tomorrow
• Karabo in FXE experiment to be deployed on Wednesday

SPB

• By tomorrow the mirror installation will be done.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.

ITDM

• Infiniband port fixing ongoing
• GPFS bug to be fixed with a patch that should be issued on 24.04. To be checked whether it is still OK to install the patch or it is too late.

General

• PSPO will ask HSinn for a detailed schedule concerning closing of the tunnels to fine tune the activities going on this week in the SASE2 and 3 tunnels

SPB

• Karabo has been deployed  without major  problems! 
• Now PLCs update follows. Sample chamber's PLC will be updated next week due to ongoing tests until monday.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE

AE

• Timing servers update - this Thurdsay, starting at 14:00 for 1 hour the trigger and train id signals will be not available.
• Updating PLCs for MPS in the tunnels 

ITDM

• Network stability problems in SASE3 are understood and a workaroung is being implemented.

General

Closing time of the tunnels on monday still not clear. To be clarified by PSPO.

Dear all,

below please find the hottest and most up-to-date times for the closing of the tunnels after the shutdown:

Cheers, Gerd

SPB

• Timing tool for optical laser installed
• Sample chamber's PLC will be updated next week due to ongoing tests until monday. Pending.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Pending. 
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Pending.

AE

• SASE2 EPS Loop running
• Updating PLCs for MPS in the tunnels. The update was stopped last friday due to a problem with the on/off funcionality of some ion pumps. The problem was found and the fixing requires an update of the VAC loops whoch will be done in the next hours/days. In case the loop update is not possible due to specific requirements the problem can be bypassed in Karabo. 

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

SPB

• Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
• Smaract motors for PAM are needed and Nerea is on it.
• It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
• During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions  have to be checked next week when HSinn is back. 
• During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it. 
• Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
• Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

• Access  of XTD1 today @14:00 to fix some EPS switches
• SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

AE

• SCS exp: first loop running. 2 creates missing but loop is up.
• MPS connection tunnels SASE2 done

ITDM

• Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
• MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30