This day we had lots of issues.
- Nightshift noticed that the mirror 2 feedback was not running anymore since about 22hrs; PBLM2 cam was unknown; powercycling recovered the camera, feedback operational again.
- Continued shooting until ~9:00
- At ~9:30 the whole Karabo, online cluster were dead. No control possible anymore. Whole SASE2 was affected.
- In the meantime, BKR called us, EPS is triggered and bunch number went to zero. However, we didn't (couldn't) change anything due to the Karabo issue. It may be the side-effect of Karabo-issue.
- Karabo slowly came back at ~15:15
- DOC asked us to take test DAQ. JF4 control is stuck at INIT state and cannot take DAQ. DA08 went to error. HED Playground server cannot be restarted.... Need to solve those issues one by one.
- The remaining shift we mostly spent in restarting servers, motors, reloading configurations
- We could not start Jungfrau 4 - at some point the controller was started and has hung in INIT state for the last few hours. It doesn't respond to shutdown commands to initialise it correctly in order. We called DOC, they could restart it.
|