ID |
Date |
Author |
Location |
Severity |
Subject |
102
|
16 Mar 2020, 09:36 |
Frederik Wolff-Fabris | General | Information | Notes Dipatch Meeting March 16th, 2020 |
FXE:
- Beamline is in safe state;
- Minor Jungfrau calibration issues over last week. In touch with DET and EEE;
SPB:
- Good exercise: experiment run with only 2 users. Successful beamtime with data taken: to be analysed by users;
MID:
- Successful beamtime with good beam conditions and data has been taken;
- Beamline is in safe state and most of staff is in telecommuting;
EEE:
- Start collecting a list of broad activities to take place in case of a long shutdown;
- No major issues to be fixed;
CTRL:
- Few cameras and detectors hiccups – e.g. DAQ re-start bring system back;
- Most of people will run from home office / telecommuting;
- Tests together with EEE/FXE still to be performed this week - to be scheduled;
DET:
- DET is placing a plan to warmed up detectors and be ready for shutdown;
- AGIPD, DSSC are already in safe mode; |
101
|
10 Mar 2020, 12:44 |
Frederik Wolff-Fabris | General | Information | notes Dipatch Meeting March 9th, 2020 |
* Users are scheduled to be here this week (4 beamlines in user runs). SPB, HED, MID report several users have cancelled the travel. Beamlines are making mitigation plans
to run experiments.
FXE:
- Setting up and working with detectors; Working with 4-bounds MONO; Von-Amos detector working;
- Jumps of beam position have been observed and have to be evaluated;
- Jungfrau detector crashed and DET was contacted;
SPB:
- Smooth run after new tuned up on Tuesday;
- Jump on beam position comes from mirrors;
- It was observed a change of number of bunches on Tuesday night: from 2 to 30 for example; XO comment: This time should be not considered as regular beam delivery and
only parasitic use. Nevertheless, better information on what kind of changes the machine may perform during the shift can help for the future use of parasitic beam.
- AGIPB was successful; also preparation for users; Mitigated troubles with missing user;
SCS:
- Not present
SQS:
- Not present
HED:
- Solving issues with motors together with EEE last week;
- Hexapod vacuum feedthrough: missing flange has to be borrowed from another group - still to be found;
MID:
- The Pump-probe laser test last week was very successful;
- Re-start of some devices was needed;
- For this week: reduction of user group from 11 to only 3 visitors. Beamtime is a continuation from last year's interrupted time due to the power outage. Local staff is
mitigating missing users;
EEE:
- Nothing to report
CTRL:
- Nothing to report
ITDM:
- Not present
VAC:
- Gas attenuator in SQS: not above 10.2 mbar - in investigation. A stress test has to be made and shall be agreed with BKR;
DET:
- Not present
LAS:
- Not present
XPD:
- Not present |
100
|
04 Mar 2020, 08:49 |
Andreas Galler | General | Information | notes Dispatch meeting Mar 2 2020 |
SPB:
- AGIPD improvement campaign great success
- EEE --> 2nd C&C worked just fine
FXE:
- BTM tests successful
- Johann MDL motors should be in closed loop
- JF CTRL device crashed
MID:
- AGIPD Flat Fields done
- Data Correction pipeline was down --> 1 day lost
- PPLaser comm this week
- extrafoam issues
SCS:
EEE:
- updates for SDL (MID)
- fastADC issue --> don't take more than 40k samples; bugfix in progress
CTRL:
- 8 OCD events
- mostly deployment isues
- FXE DA3 overload
- old version of scantool @ SCS
XPD:
- unusual behaviour of beam pos measured with XGM
- cross check done yesterday
ITDM:
- DA crashes --> being investigated
HED:
- tecnosoft terminal issue
- motor ran into Ty limit (CCW) and got stuck due to tecnosoft logic
|
99
|
26 Feb 2020, 09:06 |
Andreas Galler | General | Information | notes Dispatch meeting Feb 24 2020 |
SPB:
FXE:
- not present due to group retreat
MID:
- Mono done, no severe issues from commissioning weekend
SCS:
EEE:
- SQS pnCCD cables have been installed, waiting for crate to connect
CTRL:
- two minor OCD call and one regarding AGIPD
- SPB not aware that no support was scheduled for AGIPD
XRO:
- piezo karabo device and firmware in HED mono need to be checked for compatibility
ITDM:
- 2 calls
- derver down and restarted
- data aggregator down and restarted
- new RunConfig delpoyed at FXE
DA:
- jupyterhub @SCS showing performance issues --> investigated by DA
|
98
|
17 Feb 2020, 12:02 |
Frederik Wolff-Fabris | General | Information | notes Dipatch Meeting Feb. 17th, 2020 |
FXE:
Nothing to report: no issues;
MID:
Delay line pumping down; Connecting all devices in preparation to be ready for Friday;
EEE:
Finishing SQS interlock scenes and SPB loops;
ELMO parameters lost themselves (in connection to XRO/Maurizio request). Parameters to be checked this week and next shutdown;
XPD ticket to be checked;
Cabling shall be started for SQS/SCS: waiting for open hutch;
CTRL:
Ticket from XPD (shared EEE/CTRL): dry run did not work at BKR: missing pieces.
Please instruments check the systems during the day prior Friday when the Test Run starts;
Request from CTRL/Chris: Define guidelines for the users/support on the ticketing and e-log (who has responsibilities to check devices?).
ITDM:
Ticked from XPD/Liuba: it is a ticket with link (?) to E-log.
* Mc2 motors: XRO needs a list of all motors in order to check configurations. Example: SA3 MONO pre-mirrors with torque set to zero; SA3 exit
slits with wrong settings.
* CTRL reminds the instruments to check devices during working hours prior Friday when the Test Run starts with beam delivery. |
97
|
10 Feb 2020, 10:12 |
Frederik Wolff-Fabris | General | Information | Notes Dispatch meeting Feb 10th, 2020 |
SPB: Nothing to report;
FXE: Everything ongoing; nothing special to be reported;
MID: Finalize motors of split and delay line; MID has to test interlock for vacuum loop update.
HED: Commissioning of laser heat table. It may need assistance from EEE and CTRL during the week.
SCS: Nothing to report - Last week's mounted setup will be disassembled in order to mount the standard setup for users;
EEE: OCD was called from XRO during beamline alignment of SA2: 1 BMPI motor had a problem and is under investigation - EEE reminds on the new homing position for
negative direction motion.
CTRL: Following reset of train-ID and timing system for today.
XPD: Shifted Beam Pointing Studies from A. Koch to Tuesday.
DET: Nothing to report
*machine is currently not running due to filter clean in the cold compressor. Shall be re-started this afternoon and by tomorrow must be lasing.
*Master time reset and train-ID reset this morning. Then EEE (check systems), followed by CTRL (karabo servers re-start) and ITDM (check DAQ operation). LAS and DET are
informed.
*Wednesday: Network stress test with Cameras by CTRL on Wednesday 12:00-14:30. |
96
|
22 Jan 2020, 09:16 |
Andreas Galler | General | Information | notes Dispatch meeting Jan 20 2020 |
SPB:
FXE:
- MC2 library motor commissioning on-going
MID:
- SDL motor parameters defined, however one crate is blocked by Wille and 6 motor connectors (Harting material) are still missing
once fixed, motor paramaters can be uploaded
- WCB (wall connection boxes) in otics hutch are partially finished
HED:
- Wille is working on some crates and the respective Beckhoff loops are down for the moment
- loop updates from Tuesday on
EEE:
- additional limit switched for XTD6 CRLs (MID) are being installed
- HED is asking for limit switches on their XTD6 CRLs as well --> EEE and HED will investigate when and how
- cable change in FXE starts this week with exchaging wrongly poled SHV cables
CTRL:
VAC:
XRO:
|
95
|
15 Jan 2020, 09:09 |
Andreas Galler | General | Information | notes Dispatch meeting Jan 13 2020 |
SPB:
- Testing coordinated movement of Mirror motors with the MC2 lib on lemo drives
- optics hutch vented accidentally vented but root cause is understood and vacuum had been reestablished
FXE:
- XTD2 CRLs have encoder problems --> cables damaged --> repair strategy is being evaluated with EEE
MID:
- SDL should enter the hutch by the end of January
SCS:
- pulse (train) picker to be installed in the tunnel by the end of this week
EEE:
- FXE had been migrated to MC2 lib
- motors need to be recalibrated
- XTD6 motion part is to be migrated to MC2 lib
- SQS gets interlock updates
CTRL:
- busy with tunnel, additional support for XPD (imagers motion components)
|
94
|
08 Jan 2020, 12:36 |
Adriano Violante | General | Information | Report of the Technical integration meeting |
SPB:
- Coordinatd motion for MC2 motors is going to be rolled out by EEE soon.
FXE:
MID:
- SDL tests ongoing.
- All SDL cables have been installed and tested apart from 6 cables (still waiting for pins to be delivered).
- Various loops update and hexapod integration ongoing.
- CRLs - EPS loop will be update probably this week. (involves position sensors in the y direction for CRLs in SASE2 and 1. To be done in the framework of the full beam operation project)
EEE:
- SA1 VAC loop update. Issues with the eplan export. Discussion ongoing with VAC and XPD.
- SA2 MOV loop update for imagers MC2 motors. This week a meeting with all the stakeholder will be held (MID, HED, EEE, Controls) and a suitable date (during next week) for the update will be found. PMO will ask MDommach to provide some feedback on the Tunnel closing schedule. As soon as the feedback is available, it will be distrubuted through the readiness mailing list.
|
93
|
18 Dec 2019, 10:06 |
Adriano Violante | General | Information | Report after tech. Int. meeting 09.30 |
MID:
- SDL tests ongoing. Important motors have been tested. Alignment of the mechanics with the alignment laser ongoing. Will be pumped over Christmas.
EEE:
- uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
- SASE2 EPS loop will be updated probably today.
- SPB/SFX is getting a new CPU.
- Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.
ITDM
- Meeting this week ITDM/EEE/TS to discuss how to mitigate issues caused by accidents (e.g. cooling water interruption caused by power cut).
Controls:
- CY sent a mail several weeks ago to XO and the DH. The topic is emergency procedures during Christmas. No reply. CY announces that in the period 23.12-07.01 the Controls OCDs would not be available.
AoB
- No dispatch meeting on 06.01. Next meeting will be the Technical Integration one on 08.01.
|
92
|
16 Dec 2019, 09:28 |
Adriano Violante | General | Information | Report after dispatch meeting 09.00 |
FXE:
- uTCA update has been completed by EEE on Friday evening. Information will be passed to controls for the finalization.
SCS
- The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP.
- MT will talk to VAC to clarify how to go on with the train picker integration in the tunnel.
SQS
- uTCA has been updated.
- A channel in a digitizer is bisbehaving. The issue is being fixed.
- Steinmeyer stages will be commissioned today with MS.
MID:
- SDL tests ongoing
- Hexapod controllers will be moved to the rackroom. Cables have to be pulled and only after the position of the controllers can be chosen.
- Loop12 will be updated today by EEE
EEE:
- uTCA puppet migration: Instr. done. Tunnels and balconies probably will be updated at the beginning of next year.
- SASE1 and 3 EPS loop will be updated probably today.
- PLC in SPB with issues is being worked on.
- Migration to MC2 motors has been agreed with controls, XRO, XPD. in XTD 6 XPD the upgrade will be done in KW2. For the rest a plan has to be made.
XPD:
- Some issue with the font size in the new Karabo GUI. Controls will have a look.
Controls:
- Karabo updates ongoing. SASE3 today until 13:00. SA1, 2 on Wed.
- MC2 motors updates has been started also in FXE.
- SASE3: meeting with Lasermet for the integration of interlock signals in Karabo. Controls has handed over the scenes to the instruments that will optimize them.
- firmware of the power supply for the AGIPD has to be updated urgently (by Wed.). CY sent a mail to AM. If no reply the power supplies will be switched off for the update.
|
91
|
11 Dec 2019, 09:54 |
Adriano Violante | General | Information | Report after Tech. Int. meeting 09.30 |
FXE:
- New optical table will be installed today. Access to the hutch could be limited.
- The new Airbox installed by D3 is colliding with the sliding door. It might be problematic to bring the new laser table in. SRP informed. It is being followed up.
- uTCA update will be done on Thursday.
SCS
- The interlock system for the big sliding door has to be fixed. JTD is following up the issue with TS and SRP. It is not so clear who are the main players so far.
MID:
- SDL will be completed at the beginning of next week (inc. cables measurement).
- SDL tests ongoing
- Hexapod controllers will be moved to the rackroom. KS is waiting for the controller allocation in the racks in relation to the cables length from MFobian.
EEE:
- uTCA puppet migration: in MID it has been done yesterday. Today SQS and SCS will undergo the update. Tomorrow HED and FXE. Next week tunnels.
- 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. ITDM has to come back to BF to define a date.
- Update of loops in the tunnels are scheduled for next week
|
90
|
09 Dec 2019, 09:22 |
Adriano Violante | General | Information | Report after dispatch meeting 09.00 |
FXE:
- No major issues.
- Karabo will be updated to the version 2.7.0. on Friday.
- uTCA update will be done on Thursday.
SCS:
- it was forgotten to involve EEE in the PLC update of the train picker in the tunnel. Now the issue is solved.
MID:
- SDL cabling ongoing.
- SDL preliminary tests ongoing
- Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian.
HED:
- IC1: cheking of the encoder is going on with the help of EEE (Hazem Ali)
- New interlock definitions for Loop 6 are giong to be delivered with some delay. NJ is informed.
- Hexapod controllers will be moved to the rackroom. KS is waiting for the cables length from MFobian.
- XPD installed the diamond screens in the pop in monitors in the tunnel.
EEE:
- uTCA puippet migration will start today in the optical laser systems. Next weeks it is planned to do it in the tunnel (XPD is informed). Info mail will follow.
- 1 day has been requested to ITDM to clean up and re arrange uTCA related cablin in the balconies. Info mail will follow.
- SCS: new set of cables (microscope and others) will be ready for installation tomorrow.
Controls:
- 13.12. Karabo 2.7.0 deployment will begin.
- Normal motors will be upgraded to MC2. First tests are foreseen in MID tunnels. XPD and XRO are affected. XPD has been informed in this dispatch meeting. KS will inform MV (XRO). MS (EEE).
|
89
|
04 Dec 2019, 14:18 |
Adriano Violante | General | Information | Summary of the technical integration meeting |
FXE
- No major issues - PLCs update has been started by EEE (BB)
SCS
- Hrixs and related components will be delivered on 13 and 14.01.2020. SCS is coordinating the works with TS in order to crane the chambers into the hall (collision with the scaffolding required for HED dipole laser)
- Puls picker update in XTD10. Requires update of loops. EEE is not informed. JTD follows the issue up.
EEE
- General work is ongoing.
- Next week the EPS loops will be updated in the framework of the full beam operation project.
- Issue with a PLC in SPB crashing after update to Win10. Beckhoff is working to a solution. If this does no work the PLC will be replaced.
Controls
- A mail will be sent out early next week to announce th deployment schedule of the new Karabo release.
|
88
|
28 Nov 2019, 15:42 |
Andreas Galler | General | Problem | affected ITDM Systems after power cut tonight |
XHQ.E1.092
SW-HH-AL-271 was available
SW-HH-AL-272 came back after the DL switch was back (see below, ca 09:33)
SW-HH-AL-273 had to be power cycled and rebooted by NOC through console to come back fully (ca. 10:52).
SASE1 balcony room 38
restarted at 4am:
DELL servers:
exflprx03n0
exflprx04n0
exflprx112
exflpcl01n0 to 08n0
exflpcl18n0
SuperMicro servers:
exflonc09 to 12 (4-in-1 chassis)
exflong06
IBM ESS systems stayed off, as well as storage jbods
exfl-ons-gs111 and 112
exfl-ons-gl111 and 112
exfl-ons-gl113 and 114
IBM server (hosting ems servers for OS installations) was off as well
exflkvm01
SuperMicro servers stayed off:
exflong03 to 05
juniper (management network) switch stuck at booting, power cycled, after long time finally came back, but we were lacking management network over whole site
SW-HH-AL-DL-07 (office network distribution layer) switch was stuck at booting, came back after power cycling. Some (not all) office switches were not available because of this, seemingly the failover between AL and DL didn't work. Both this switch and the (permanently available) ...DL-08 were correctly cabled to separate PDUs.
Second fuse in BR38 rack 7 was out and had to be pushed back in manually (made power redundant again).
SA1 tunnel:
exflcon08n0 restarted at 4am, but apparently switch in the tunnel was off till 10:30 and since that time the server was accessible
E0.604
All systems were affected. We need the clear statement from YS why the second line did not work.
DELL servers rebooted at 4am:
exflprx01n0
exflprx02n0
exflpcl51n0
exflpcl52n0
exflpcl53n0
exflcon03n0
exflcon04n0
exflcon05n0
exflcon150
exflcon153
exflcon156
exflcon157
exflons01 and 02
exfldaq01 and 02 (gitlab CIs)
exflst105 (karabo project db - development and labs)
IBM servers stayed off:
exflpcl54n0
exflpcl55n0
exflpcl56n0
exflpcl57n0
exflonc41
exflonc41
IBM ESS system stayed off:
exfl-ons-gs901 and 902
There are also several servers (proxmox, serving VM OnlyOffice) all rebooted at 4am
List of h/w failures:
exflpcl05 PSU dead
exflong04 and 05 PSU dead (in exflong04 was actually since yesterday after reboot for OS updates)
may not be related to power glitch
exflprx106 RAM DIMM error
Intel server:
exflonc21 to 24 (4-in-1 chassis) PSU dead
|
87
|
27 Nov 2019, 09:25 |
Andreas Galler | General | Information | dispatch meeting Mon Nov 25 2019 |
FXE: no issues
SPB: no issues
MID: Split & Delay line arriving on Friday, MID aims for a quick integration into instrument / beamline
EEE informed to switch Hexapod motors from loop3 to loop 7
HED: changing loop1 (JJ X-ray motors) from open to closed loop operation
SQS: KB works, see sometimes spontaneous switching of end switch logics
EEE: MID, SPB, VAC loop updates running this week;
no triggers for experiments available until end of power off work
Jörn: cabling for MID is in full swing, for future installations first check phase 1 cabling, for non blocking
HED: cabling
SCS company Witte 30 ccables
FXE: two CRDs not approved yet, no cabling during this shutdown
Controls:
|
86
|
20 Nov 2019, 10:12 |
Adriano Violante | General | Information | Minutes of the Technical integration meeting 20.11.19 |
FXE
- Wille was asked by D3 to pull new cables for the Pandora in FXE on 19.11.. This was not known by FXE. Will be discussed with Safety/DNölle.
- Mono was fixed and is now ready for commissioning with beam
SPB/SFX
- uTCA update planned for tomorrow#
- Chris Blöm will leave the company next week. His tasks need to be re assigned.
- Order of loops upgrade will be discussed with M.Stupar as soon as he is back.
- The drag chain will be opened for cables re organization. HV cables will be also replaced by Wille . SPB/SFX will get the new HV cables from Sonay Sayar.
SCS
SQS
MID
- Loop11 (AGIPD) needed. The task was not envisaged in the shutdown task list and is required by the detector group. Will be discussed with MS.
- It has to be checked whether the delivery of the SDL is confirmed and the patch panels infrastructure is ready for cables installation.
HED
Optical lasers
VAC
XRO
XPD
DET
EEE
- Power supplies - firmware update will be performed in the next days. Mail will be sent around.
- Tming system update. No triggers will be available. Will be announced by mail.
- uTCA CPUs migration. The CPUs stored valued will be lost. Karabo devices will need to be re installed. Karabo stored values will not be lost. Will be announced by mail.
Controls
- The overnight and weekend OCD service will stop during the shutdown. OCD only available in the time 6:00 - 18:00
- Karabo 2.7.0 deployment date for the tunnels is the 13.12.- Instruments to follow.
ITDM
SRP
TS
- The power cut works will start at 6:00 in the morning. The electronics should ne put in a safe state the day before.
- TS can provide support for temporary light and powerr during the power cut periods.
AoB |
85
|
13 Nov 2019, 10:12 |
Adriano Violante | General | Information | Status report after tech. Int. Meeting 9.30 |
FXE
- Radiation tests ongoing.
- EEE informs that the PLC drawings will be exchanged in the WS
SPB/SFX
SCS
SQS
MID
HED
- During the first week of shutdown (KW47) the loop1 devices will be put into closed loop (e.g. for backlash compensation).
- An electrical shutdown seems to be necessary to upgrade the protection against the power glitches, especially to protect the optical lasers. This is now planned for the 17/18.19 for SASE2 and similar dates for SASE1 and 3. AV will try to collect more information and come back to the instruments ans technical groups.
Optical lasers
VAC
XRO
XPD
DET
EEE
- EEE needs the updated interlock definitions from VAC
- The list of requested services during the WS is very long. NJ is trying to plan the works inside EEE and will come back to XO/PMO in case priorities have to be assigned. Also, late requests and changes are making things more difficult.
Controls
- The overnight and weekend OCD service will stop during the shutdown. OCD only available until 18:00
- Karabo 2.7.0 deployment date (now forseen for KW51) is being verified. More info to come.
ITDM
SRP
TS
AoB |
84
|
12 Nov 2019, 09:49 |
Adriano Violante | General | Information | Status report after dispatch meeting 09.00 |
FXE
- FXE – radiation tests and LPD tests planned for this week. Small issues with the scan tools are being addressed.
SPB/SFX
SCS
SQS
MID
- First measurements with the pulsed magnet have been performed last week with the power supply installed temporarily in the experiment hutch. This week only the cryostat is going to be used. Next step will be to bring the magnet power supply in the rackroom to comply with the electrical safety requirements. MID will follow it up.
HED
Pptical lasers
VAC
XRO
XPD
DET
EEE
Controls
- Many IT/controls services will be down in KW48 (shutdown)
ITDM
- Server 13 for SCS will be down today.
- General remark from Controls and ITDM: devices and functionalities (e.g. new devices in the DAQ, data aggregators…) that have not been tested during the normal working ours should not be used during shifts in the weekend or during the night. This creates a huge and evitable overhead for the people in OCD.
SRP
TS
AoB |
83
|
28 Aug 2019, 10:33 |
Adriano Violante | General | Information | Minutes of the Technical integration meeting 28.08.2019 |
XO
FXE
- Leak in the LPD cooling system has been found. DET on it. Ongoing
- Laser went out of synch several time during the last weeks. Being addressed with the synch. group.
- Mono middle layer is not working properly. Calibration parameters are not reloaded correctly after restart. CAS on it. Ongoing.
- SA1 MOV loop has been updated. Minor issues with the CRL encoders are being addressed.
SPB/SFX
- CSLITS - discrepancies between the behavior of normal motors vs MC2 motors. CAS on it. ==> will be fixed once Karabo 2.6.1. has been deployed.
SCS
SQS
MID
- During last WE big issues with the AGIPD were experienced. Probably calibration pipeline related. DET on it.
- Laser hutch A.07 is not supposed to be accessible in the next weeks because of the laser operation. SRP points out that they are not sure whether the interlocks are properly working and active. To be clarified offline between MID, HED and SRP.
HED
- Access to the experiment hutch, Optics and Rackroom will be restricted this week due to laser shots tests.
Optical lasers
VAC
XRO
- Change of middle layer devices for the attenuator in SASE2 will be discussed offline with CAS
XPD
DET
EEE
- Loops are being updated in coordination with the instruments.
CAS
- 2.6.1. ready to be deployed. Deployment schedule has been agreed with individual groups.
ITDM
- Maxwell cluster maintenance on 2-3. 09.
- Control PCs wil be deployed during the shutdown week
- Claoud service (DESY platform) will be put into operation. An official announcement will details will follow.
SRP
TS
AoB |