General Announcement:
- Two severe power glitches happened on the weekend (SAT and SUN around 7AM). We thank all colleagues that were in shift, OCD or volunteered to help assessing the situation and restoring their operation
ability.
- The presentation of the Winter Maintenance planning period is postponed due to time issues.
- Janusz S. reminds on cyber security: tomorrow the next step to secure Zimbra mail client takes place. A detailed e-mail will be sent.
- XO will create a report on the power issues.
CTRL:
- Issues with power glitch: restoring parameters, Karabo, SEPS, SA2 started from scratch.
DA:
- Nothing to report; end of chain regarding the power issues.
EEE:
- Same issues with power.
- Will follow up the SCS train ID.
VAC:
- No effect on VAC; no access to SA2 then it was blind / no Karabo, but at the end was fine.
- Issue with HED and MID Monochromator - to be investigated. Access is needed today.
XPD:
- As PRC Jan thanks all people who helped to restore SA and SUN (two power glitches around 7AM); Also Desy/MKK confirmed that machines were briefly affected. EuXFEL accelerator recovered quickly.
- Consequences on SAT: 1. Power went away of whole SA3 area; 2. Tunnels mostly not affected but water cooling pumps were switched OFF (affecting also balcony rooms).
One auxiliary pump did not turned ON and several hours later (10-11AM this was too hot) and brought down services. SA2 tunnel cooling water temperature raised but on SUN it was faster to act/recover (MKK;
30mins).
- XGM at XTD6 went down.
- MKK does not see the temperature but only humidity. To be implemented in the coming days.
- SA2 Karabo totally down.
- Interlocks went down (EEE, D3, SRP) and had to unblock radiation alarms. A Desy mTCA crate went down and had to be manually power cycled (due to cooling water lack at balcony room).
- PP-laser in SA3 went down. SA3 experiment at SCS was cancelled at the point.
DET:
- No specific feedback for DET; shall be OK.
- Broker pile up was not fully understood. SA2 was restarted and back to normal; MID restarted yesterday and will get in touch with them.
ITDM:
- Cooling issue: discovered by alarm (Kimon who informed MKK); it needs a better procedure to inform MKK.
- After 30 mins called from MKK saying T was going down, but due to open doors.
- Steffen H. says a report has to be done with goal oriented to avoid this in future.
XRO:
- Looking into systems: in tunnels, if power is OFF then mean cooling shall be OFF. Cameras do not work remotely (connection to servers; only one is accessible).
LAS:
- SA3 is down: problems with main amplifier;
- Checking for damage after second power glitch.
UNSYS:
- All fine.
SPB:
- Nothing to report. User got data.
FXE:
- SA1 was also affected by power glitch (interlock crashed): had to be reset by D3; AIBS went to error and be reset by DOC.m
- Prevented continued with user (lost 6+4 hours of beamtime). User went well and collected data.
- On Saturday the AMPHOS laser broke down in FXE. SRP and Management took measure to bypass. Not sure if caused by electricity. To be followed up this week.
MID:
- Nothing to report. Request: in such severe power glitches an e-mail to MID has also to be sent.
HED:
- Could recover after SA2 Karabo broke down. Recovery faster than expected.
- Continued with user program. Data was successful.
SCS:
- Stopped running experiment.
- Interlock problem with laser helped by Pouneh (thanks!).
SQS:
- Nothing to report.
SXP:
- Challenging communication. Manuel gave phone number to PRC and will give to XO.
- Power glitch: Desy colleagues do not have access to SXP hutches. |