Notes XO Integration Meeting 24.08.2020, posted by Frederik Wolff-Fabris on 24 Aug 2020, 11:39
|
General Announcement:
- The safety rules require the use of masks when moving around as well as the limitation on the number of allowed people per room.
XO asks people to stick to rules of 1 person per office as well as using masks when working closely.
CTRL:
- Few minor issues during the weekend. Major one related to SQS optical spectrometer which did not reach 10 Hz.
- Today an upgrade in the archiving system will take place. The update of timing servers for all SASEs will take this afternoon.
The property “bunch pattern table” will not be filled anymore.
DA:
- Few OCDs calls were all manageable. DA reminds instruments and groups to let system online to be monitored.
EEE:
- One OCD call during the weekend.
- Cabling with Detector group to happen.
- EEE asks if the SA3 interlock/pre-absorber will be tested today.
VAC:
- Nothing to report.
XPD:
- There was a problem with XTD2/SA1 graphite filter accordingly to Operation E-log entry. Maybe it got damaged between 20. and 21. This will be further investigated
between XPD and SPB/SFX.
DET:
- 1 OCD call which was easily solved.
ITDM:
- Friday the Scratch system (for temporary data) was out of space and data were removed (70Tb). It may be a problem again and ITDM would like to remind people to keep
data only for short time.
There is 1.5 Petabytes storage with currently only 60Terabytes free space.
- An e-mail about this issue was sent to a distribution list stating that from 01/09 data will be removed. ITDM asks people to clean their data in the proposals
directories.
XRO:
- Preparing for ZZ accesses in XTD2 and in XTD10.
LAS:
- SA1 laser is temporarily fixed with stable operation during the weekend. It required changes in software. This PP-laser is working and further exchanges are postponed.
- LAS reminds that OCD calls regarding timing synchronization between laser pattern and beam must be addressed with CTRL OCDs.
- Jan G. offers assistance with 5Hz operation on diagnostic cameras. A. Galler suggests to include also EEE and CTRL in discussions.
UNSYS:
- Nothing to report.
SPB:
- Graphite issue: inserted on Tuesday for 15 keV and was in the beam for 4 days until Saturday. J. Gruenert says this is not wrong. An e-log entry was done by Romain
Letrun.
- 1MHz images in camera worked during the weekend.
FXE:
- Issues: 1. Electron bunch arrival monitor recording did not work; 2. When SASE3 changes to many bunches, then FXE sees decreased intensity; 3. PP-laser seems not
suitable for our operation.
- For #1, ITDM will have a look in it; For #2, this will be discussed with machine; For #3, further discussions will happen in separate meeting.
MID:
- No issues: Split delay line is under commissioning. It was quite successful week.
HED:
- Successful week.
- The shot on demand feature was tested with EEE and CTRL.
SCS:
- Issue with saving sample position.
SQS:
- SA1 and SA3 interactions: FXE suffers when we have many pulses. M. Meyer proposes to have discussion with machine.
A. Galler proposes to discuss during the machine meeting at 10AM and asks for additional info per e-mail. |
Notes XO Integration Meeting 18.05.2020, posted by Frederik Wolff-Fabris on 19 Aug 2020, 12:08
|
General Announcement:
- A. Galler reminds the audience on the daily informal BKR chat taking place at 15:00: link was sent by e-mail.
CTRL:
- Tests on the HED bunch patter / 5Hz were done yesterday;
- Five devices related to the timing server will be re-started on Monday. This update of timing server shall be transparent.
- Other requests will be discussed separated in another rounds with respective involved groups.
- For Laser switch between SCS/SQS please call LAS OCD for troubleshooting (experts mean faster solutions).
DA:
- DA reminds the group on its Redmine ticketing system that was implemented in February and helps on the planning and scheduling of activities. Instruments are encouraged
to use the DA ticket system.
EEE:
- Initial test with HED 5 Hz pulse on demand went successfully yesterday.
- Missing crate in HED will be installed.
- SCS installation of cables will be done.
- The special trigger for overlapping bunch in SQS was fixed this morning. CTRL was asked to prepare another parameter in Karabo.
This shall be OK and tomorrow it shall be fine to use the adjusting the timing. S. Hauf points a device has to be adjusted.
- A. Galler informs the tentative date for the SEPS update will be next Monday.
VAC:
- ZZ done on Monday to check status of SA3 devices.
XPD:
- X-ray PES undergoing tests at Petra-III / Desy.
- A SQS pop-in could not be driven out of the limit switch this night. This was solved in the morning (back to work) and it was related to a problem with motor counts.
DET:
- Preparation for user run at SQS: still some data to be taken with DSSC but it has been under work.
ITDM:
- Together with CTRL and DA, it was set dedicated server for logging in to get more insight in the experiment.
- In the main page of user proposal there is a permanent Zoom link to be used for communication with users. It stays for 2 weeks: SQS is working and other two experiments
will be set today.
- In addition each experiment will have a dedicated zoom room.
XRO:
- SA1 M1 continues stay in place. By the other side, XRO would like to ask for a ZZ to complete the chiller level.
- Interlock tests in SA3 did not happened (changing of Interlock from GATT to pre-shutter) Z. Ansari informs that from her side it can take place on Monday 24th.
LAS:
- SA1 laser: the front-end is switching off in regular base (3x since Monday). There is not enough power for experiment.
Meeting with manufacture take place today at 10 to build strategy to investigate and decide next steps. LAS will inform next steps.
PP-laser is expected to be working for this and coming weeks.
- SA3 Amphos: it can be operated. A final fix will be done in the in next blue weeks or during winter shutdown.
- SA2 laser is fine.
UNSYS:
- Nothing to report.
SPB:
- Working on CRL alignment and also testing of camera.
- Some alignment activities were interrupted due to the Laser issue. At 20% lower energy, it is not enough. Impact on commissioning if laser is not back may affect user
experiments later.
FXE:
- Nothing to report.
MID:
- Commissioning of the split delay line: lower branch is successfully done.
HED:
- Setting up for experiment; few issues with diodes.
SCS:
- Nothing to report.
- XO reminds the protocol to increase bunch numbers out of what is agreed on the delivery tables: 1. try to discuss with other experiment first; 2. call the PRC.
SQS:
- The zoom account was tested last night.
- Meeting with DET today.
- All are going fine. |
Notes XO Integration Meeting 17.08.2020, posted by Andreas Galler on 14 Aug 2020, 10:40
|
General Announcement:
- An informal Run coordinators' BKR chat to exchange ideas happens every day at 15:00.
It is a forum in case technical and scientific people wish to make formal requests. XO will send an e-mail with Zoom link.
- XO would like to ask instruments to pay attention to the parameters of the Delivery and Operators table, especially regarding the bunch limits for the North branch.
Ideally we shall discuss one week before beam delivery.
- H. Sinn notices that if SPB move some motors, this is temporarily cutting SA3 beam due to MPS. SQS has not observed this effect. Harald will further investigate.
Adrian M. says SCS impacts SPB in different ways.
- A. Galler mentions that SRP/task force will distribute updates related to the currently raising Covid issues.
CTRL:
- One group of motors at the same SCS/PLC was not accepting changes. A re-start with servers in small loops must help.
- Timing servers (switching off archive): CTRL wants to perform today or tomorrow morning the installation of the new version. Nobody seems against it.
DA:
- Nothing to report.
EEE:
- EEE understand the team request regarding the 5 Hz trigger will be done tomorrow.
- EEE waits for confirmation of SEPS update, possibly happening today or tomorrow. XO mentions Z. Ansari will be able to confirm today after lunch time.
VAC:
- Nothing to report.
XPD:
- Not present.
DET:
- XO inquires on the DSSC activities in SCS: DET says it may be delayed a bit due to other activities; DET got a ticket regarding operation; DA believes the DSSC was
running and used.
ITDM:
- Comment on DSSC: it was used but we shall not forget that on the first day there was not a preparation and we could expect hiccups.
- A faulty memory module must be replaced and the exflgateway will be shut down shortly for this activity at around 12:00. An e-mail will be sent but nobody seems against
to it.
XRO:
- Nothing to report.
LAS:
- SA3 AMPHOS photodiodes still has problems. System is partially working and can be operated. LAS knows beam size and pointing do not change but not if loses power.
AMPHOS is contacted.
The consequence is that switching ON and OFF takes more time because it is done manually. Further measurements will be done and SA 3 instruments will be updated
accordingly.
- LAS informs its OCD team is not responsible for synchronization issues.
- SA1 and SA2 lasers are fine.
UNSYS:
- Small issues with SA3 PLC crashes: SA3 is operated without first cells. A re-start of the servers is needed, probably today or tomorrow. UNSYS will inform SCS and SQS
when this is fixed.
SPB:
- Nothing to report.
FXE:
- FXE says that SQS will need to perform some tests: this will be discussed in separate round.
MID:
- All fine.
HED:
- Data recording on Saturday night was not saved. Shift crew realized at the end of the shift that it was not saved - main data for the shift.
- ITDM investigated the issue after ticket arrived yesterday. It confirms data was not recorded and can't see why this happens as ticket arrived late. Even the folder of
the data was not recorded.
For the future: there is an assumption to look at the data every few minutes. ITDM will share procedures. Further discussions on flexibility for different systems/setups
have to be done.
- HED says the 5Hz test must be tomorrow by the morning. XO states that currently this is only a test then we will see what happens. HED has this understanding.
- HED has recorded several hundreds of microns in horizontal jitter. XRO says that it is important to work together for investigating the jitter.
HED will send the data to XRO and/or the coordinator of stability task force.
SCS:
- Not present.
SQS:
- Preparing next week user operation; still some problems with Zoom connection: SQS and ITDM will discuss the issue. |
Notes XO Integration Meeting 12.08.2020, posted by Frederik Wolff-Fabris on 12 Aug 2020, 10:52
|
General Announcement:
- Andreas G. thanks all instruments that have so far sent back to XO the cw35 Beam Delivery tables and reminds the missing ones to do so until Thursday (tomorrow).
- Jan G., as PRC, informs the access to SQS shutter will probably happen on next week. In case of possibility during this week he will directly inform the respective persons.
- Steve A. mentions on the timing related issues: it has been investigated with the help of instruments.
- XO asks groups for any time feedback on issues related to Covid rules, e.g. amount of masks.
- Jan G. informs several members of XPD team (and also CTRL and ITDM personnel) will be working at PetraIII while performing beamtime, from Thursday until mid-next week.
CTRL:
- Activities on GATT: Follow-up of Karabo update shows incompatibility with new version. CTRL, Raul and XO rolled back to the previous version and it works again.
- At around 8:00pm a notification service in SASE3 was sent: watchdogs went in unexpected state.
DA:
- Nothing special to report.
EEE:
- Today we bring back the crate for MID/STL and we start with HED requested changes.
- XO thanks Nerea on yesterday's work of Interlock conditions for the solid attenuators in the tunnels.
To protect the silicon screen first one has to insert the diamond screens. Nerea J. says this is in the motion loops and not EPS.
VAC:
- Raul V. thanks for the quick reaction of CTRL and XO to solve the code of GATT (which is 2.5years old).
If we have a chance during the next winter shutdown a change on the software system will be planned to permit the use of new Karabo versions.
- VAC has requested a couple of ZZ's for next days/weeks: on SQS shutter and other devices inspections.
XPD:
- No incidents since Monday to be reported. However, last Saturday there was a downtime on the XTD1/XGM due to a high voltage setting.
Probable cause was a wrong command at the DOOCS server. XPD believes it was a onetime incident and will look into it if it happens again.
DET:
- Working with Jungfrau in SPB; DSSC in SQS/SCS.
- Supporting other groups' activities.
ITDM:
- The DAQ update has been finished: so far it looks good.
- ITDM is ready to act but waits for more clarifications on the SRP requested work on Interlock.
XRO:
- Not present - group has helped with solid attenuators activities.
LAS:
- Everything is running and interlock is set.
UNSYS:
- Nothing to report.
SPB:
- Detector program ongoing with Junfgrau and AGIPD.
FXE:
- A ZZ was done to switch off the M3 cooling. The jitter has then reduced from 295u (RMS) to about 211u. FXE considers this only moderate jitter and will proceed with commissioning.
- Feedback system is fixed with parameters configurations – we are following it up to find when this was changed (if so).
MID:
- Issues with feedback Macro are observed and Robert S. investigates together with MID.
HED:
- Not present.
SCS:
- Not present.
SQS:
- Nothing to report.
|
Notes XO Integration Meeting 10.08.2020, posted by Frederik Wolff-Fabris on 11 Aug 2020, 11:01
|
General Announcement:
- Beam delivery starts this week. XO reminds the reason to send the Beam Delivery table is to provide machine with parameters. Please take in mind that we shall not fill
out tables during Friday's UOM.
CTRL:
- Two configurations changes planned for today if agreed:
1. In SA3 machine changes the 10Hz and may affect the archiving. This means time serving in Karabo has to be changed during the day today;
2. CTRL updated last week most of Karabo devices in instruments, but few tunnel instruments remain and shall be done today - few time down and then re-starts.
- HED issues with VAC/SA3 have been investigated together with EEE.
- InfluxDB update was done and OK.
- LAS comments a laser pattern is required in order to compare laser status between different runs. Follow-up discussion with beamlines, LAS, CTRL is needed.
- AG asks if SA2 and SA3 are aware of the 5Hz operation mode. Tomasz from LAS says all servers see the bits are changing.
- Steffen concludes that CTRL will proceed with updates today and by next Monday CTRL will switch on the archive again with the solution we will have by then.
DA:
- Nothing to report.
EEE:
- Taking out crate of MID; making changes on the request of HED.
VAC:
- Nothing to report - all running.
XPD:
- Nothing to report.
DET:
- Not present.
ITDM:
- DAQ recent bugs were mostly fixed: today FXE and HED will be fixed. Afterwards all are done.
XRO:
- Nothing to report.
LAS:
- SA3 laser had a probably on Saturday: warming up devices. LAS may be able to fix today (Sunday's effort did not work).
UNSYS:
- Issues in SA2 with cells #1 and #33 have been fixed by ZZ on last Wednesday.
- Also hardware problem in SA3 cell#14 was also fixed.
- Software development in response to MXL complaints is under work and needs a maintenance and/or shutdown time.
SPB:
- Nothing to report.
FXE:
- Meeting with EEE to look in the digitizer - issue could be solved, it is now running stably.
- Used Beam feedback to stablish the beam position was not working reliably: spontaneous jumps were observed. FXE will look together with CTRL on Wednesday.
One idea is to pick beam position from imager. No other instruments have this observation.
MID:
- Synchro will arrive today and thus SA2 laser will not be available.
- Train matching offset between AGIPD and XGM reported two weeks ago: MID is asking for an update. SA will check today and give information on Wednesday.
HED:
- HED had problems with MC2 axis in Beckhoff motors during last week and asks to have a follow up from EEE. Events of losing parameters are happening quite often.
- Karen A. mentions they added manually parameters and they had to reset again. HED-LOOP1 will not be reset today (Steffen). SA2 loops will have to be checked.
- Steffen and Marijan add the issue is now understood by sacrificing one of the HED PLCs. One has to fix the code of the framework and this can't be solved by Wednesday.
SCS:
- Motors of the DSSC support were also lost. They have been fixed.
- Issue with pump is also solved.
- New installed devices during shutdown (pressure gauges) were tested on Friday.
SQS:
- Not present. |
Notes XO Integration Meeting 05.08.2020, posted by Andreas Galler on 05 Aug 2020, 09:02
|
General Announcement:
CTRL:
- SA3 vac loop
- devices lose connection to the PLCs every now and then (every few days)
- karabo fix for small movement error on MC2 motors
- for the tunnel topics CTRL suggests to do that Thursday morning
- InfluxDB
DA:
- no issues
- upcoming commissioning
- da infrastructure up and ready
EEE:
- cw36, cable installations require
VAC:
- no issues, normal activities
XPD:
DET:
ITDM:
- continuing with Monday tasks
XRO:
LAS:
- increas priotity on karabo integration for lasermet interlock
- SH: for SA1 and SA2 much of the interlock data is
UNSYS:
- today ZZ --> cell 33 should be fixed today as well as cell1
SPB:
- nothing outstanding
- DAQ testing
- VAC intervention
FXE:
- FXE and XRO is switching off M3 cooling
- no update on the digitizer issue
MID:
HED:
- MC2 motor timeout problem, karabo fix seems to work
- there are other time out errors that ar not addressed by this
- SH: suggests a zoom session for demonstration of these
- TP: e.g. stop button gives time
SCS:
- no issues
- installation of girder
- bestec is working on hRIXS
- recommissioning of Mono
SQS:
- laser interlock issue
- Lasermet is visiting today
|
Notes XO Integration Meeting 03.08.2020, posted by Andreas Galler on 03 Aug 2020, 09:23
|
General Announcement:
CTRL:
- few issues on SA3 vacuum server, over the weekend.
- twice some devices were not running,
- when restarting Beckhoff loops from the GUI please remember that it takes time until they are back
- device servers with - in their name. restarting with the service manager cannot cope with this --> fix is underway
- scheduling:
- Beckhoff: small movement error fix is underway and it is suggested to update karabo Beckhoff servers later this week.
- influxDB data logging
- current version does not allow easy correlation data from two devices in Grafan (web front-end) an update to the next higher version would cure that
- suggest to update Wednesday morning early
-
DA:
EEE:
- utca OCD calls
- PLC no OCD calls
- some updates are planned for FXE, MID SCS on Beckhoff
VAC:
XPD:
DET:
- comm AGIPD detectors
- detectors are stable, but issues with online processing
ITDM:
- new version of DAQ had been tested
- should be deployed this week
- 1 OCD call for recording data with ADQdigitizer
- requires coord between CTRL and ITDM and EEE
- this operation mode had been tested
- 1 OCD call JUNGFRAU SPB in burst mode --> did not work
- SH: was the receiver crashing?
- KW: not clear, nothing was working in that config
- SH: the problem had been identified at FXE --> fix should be deployed at SPB as well
XRO:
- update:
- follwoing up the issue with the attenuator damage in SAx
-
LAS:
- all three laser are running
- some problems with shutters
- SQS: shutter in ILH, cooling water refilled --> but interlocks could not be set
- XO to coordinate discussion with Pouneh
UNSYS:
SPB:
- karabo sessions lost connections to GUI server on exflcon026
- JUNGFRAU receiver crashing
- DAQ took long time to change state
- even switching to ignore after run take a long time
- this relates to JUNGFRAU
- GOTTHARD specref device crasing multiple times
- restarting had to be done multiple times
- optics interlock hutch was observed to be broken on Thursady night
FXE:
- digitizers at some point stopped
- connection to boards and triggers
- BF: this has to be investigated!
- beam stability
- EXAFS within 700 eV had been measured
- PP Laser was very stable
MID:
- VAC incident in CRL2 --> which closed V0 --> VAC solved the issues
- due to high pulse energies small damage to attenuator and instrument component had been
- DAQ update scheduling
-
HED:
- beam is jittering (when collimated 100 um are observed)
- CRL scene optics hutch is mixed up for one or two motors --> Thomas Preston investigate
SCS:
SQS:
- interlock / shutter issue
|
Notes XO Integration Meeting 29.07.2020, posted by Andreas Galler on 29 Jul 2020, 08:59
|
General Announcement:
CTRL:
- INfluxDB migration is running smoothly, tomorrow the last topics will be switched to influx only
- some Basler / Andor cameras not working at 10 Hz is related to the data network --> depends
DA:
EEE:
- Company Wille is on-site for hardware installations
- hRIXS,
- HED cable installations
-
VAC:
XPD:
DET:
- commissioning shifts with AGIPD
- pn CCD at SQS also on commissioning
ITDM:
- issue from Monday wrt crippled DAQ when having no sow data in a datasource
- the fix reported on Monday only works partially --> earliest estimate for fixing is now next week
-
XRO:
LAS:
UNSYS:
SPB:
- OCD for Basler cameras
- OCD for MC2 stages (problem with encoders, EEE is informed)
FXE:
- Major issue with the machine wrt to big brother
- JUngFRAU seems to be more affected by the DAQ glitch
- GOTTHARD detector should be re-integrated into the DAQ
- K. Wrona clarifies that only a reconfig of datasources is needed
MID:
- beam looks unstable
- horizontal jitter is higher than usual
HED:
SCS:
- no issues
- Daniele: is the soft-mono solved?
- TM comments: the pre mirror is in place piezos are working
SQS:
- no issues
- pnCCD commissioning
|
Notes XO Integration Meeting 27.07.2020 , posted by Andreas Galler on 27 Jul 2020, 09:05
|
General Announcement:
- XO discusses the need for sharing information.
CTRL:
- support calls
- small movements cause errors on MC2 motors, fis is going to be tested today and it will be communicated when ti is going to beployed
- swithcing to influxDB only logging, if not heard otherwise then this should happen this afternoon (Monday)
- switching to influxDB in SA1, SA2 and SA3 to follow as well
- notification service updated such, that only more serious messafe are being send out
DA:
EEE:
- cable installations in this week at HED, should start from tomorrow, 1.5 days (Jörn)
- few OCD calls,
- soft mono in SASE3, motor not working
- mirror
- FXE motor runs into emergency stop --> fixed in the latest PC framework, but FXE is running an earlier version
-
VAC:
XPD:
DET:
- Thursday, OCD was not scheduled, despite some calls
ITDM:
- bug for big detectors in DAQ had been addressed and should be deployed soon
- this bug seems to also affect some slow detectors
- only happens when there is no slow data added to the aggregator
- data reduction effort
- one outcome: record bunch pattern in every run
- instruments are asked to include the datasource XXX_timing
- SH comments: in the future the bunch pattern information will be made available in the pipeline
XRO:
- some recommissioning last week
- sometimes mirrors are not moving
LAS:
UNSYS:
SPB:
- good run for the first wek
- trouble with detector, not being able to record darks -> fixed next day
- issue with a device that checks correct datasources
- FXE imager EPS interlock issue
FXE:
- FXE imager EPS interlock issue
MID:
- successful week
- a couple of OCD calls
- CTRL, EEE, DET, everything solved
- AGIPD trainID offset by 1 wrt to the XGM
- karabo update --> changes are incompatible with existing scenes
- e.g. number of digits
- SH comments that it would be appreciated if those issue were reported to CTRL
HED:
- no issues
- MC2 motor update, is that going to be done today?
- SH: yes and if possible for HED, then 11:30 - 13:30 would be the preferred time window.
SCS:
- exit slits did not move --> EEE OCD could help
- SCS requests tthat components are tested before tunnels are closed
- Navitar didn't work
- focussing lens lost position
- when component software is updated/upgraded, SCS would liek to see a test
- NG comments: motor parameters are sometimes wrong
- If things need to be modified, test beforehand
SQS:
|
XO integration meeting 22.07.2020, posted by Andreas Galler on 22 Jul 2020, 09:22
|
General Announcement:
- Default beam positions are being implemented for each SASE at the FEL imager to reduce re-alignment offers for the instrumetns after shutdowns and facility development weeks
CTRL:
- minor OCD calls
- few MC2 motors not working -> target velocity was set to 0
- in SA1 the bunch pattern changes @10 Hz --> data logging system (with timers) gets overloaded ...
- if this was to become a normal mode of operation, the system would need to be adpated --> as for now bunch pattterns are not logged anymore
- remote user access to karabo
- soo to be tested with SQS
- inlfuxDB
- FXE 95% migrated, probably 100% at the end of the day
- LAS SA1 100% migrated
- MID, SCS shall reply when migration is possible (causes a short down time of datalogging)
- SPB, HED shall reply s.a.
DA:
EEE:
- issue with encoders in SPB --> some errors had been disabled --> checks for cable lengths (OCD call)
- SQS interloop comunication working? EEE, yes, need to investigate
- N. Jardon reminds about the default parameter procedure
- pre shutter in SA3 works commence when Zunaira, Martin and Maurizio
VAC:
XPD:
- intervention Monday in SA3, PES is operational
DET:
ITDM:
- last week new DAQ deplyoed in HED, FXE and SPB and tunnels, the three other instruments had been delayed
- scratch space in gpfs --> running out of space --> data needs to be removed
- procedure not yet agreed upon
- all involved parties should identify data that can be removed from scratch
- 100 TB are left
- internal proposals for commissioning time needed
- in the future please prepare those in advance
XRO:
- during the weekend, XRO was recommissioning of the beam transport
- some qctivities were completed on Tuesday, M3 repositioned from damaged spot
- recommissioning is completed --> however small problems are to be expected over the frst few days, please contact XRO
LAS:
UNSYS:
- cell 33 in SA2 communication crippled, ZZ planned for next monday
SPB:
- reasonably succesful first night with beam, instrument aligned with alignment laser
- tunnel EPS system --> the FXE_XTD9_popin blocked SPB to 1 pulse
- MPS signals from SPB might be masked by the machine -> needs to be checked
- 1 OCD call to EEE
- dMigration to influxDB can be done
FXE:
- re align beam to FXE pop in
- mono re alignment
- issue with FastValva --> naming convention on the PLC level had been changed, now only capital letters are allowed --> Bernard is looking into this
- tunnel project using step function with motors throws an error on the slot
- CTRL is workig on a fix already, it happens for very small steps
- J. Nardon: was also reported by SQS
- all TView devices had to be restarted as they were out of sync
MID:
- alignment went smooth, until diagnostic endstation
- data loggers are fine today to migrate to influxDB
- Steffen aims at afternoon
- pre shutter
- for CRL2 not possible to laign as the pre shutter always moved in
HED:
- beamline aligned
- beam profile excellent in fornt of hutch
- spatial jitter present, and horizontal movement
- mirror feedback macro not working
- XGM investigation together twith Theo
- IPM didodes are not read out --> comment by Peter, check the timers
-
SCS:
SQS:
- beam from around noon, beamline aligned, set points on imagers need to be re found
|
Notes XO Integration Meeting 20.07.2020, posted by Frederik Wolff-Fabris on 20 Jul 2020, 14:18
|
General Announcement:
- Regarding Operation, VAC and PRC had a workaround to restore the North branch beam permission on Sunday.
- XTD9 Turnstiles were not working this morning for ZZ access planned by FXE.
During the meeting we heard the information that Michael Prollius / SRP has fixed it by rebooting the computer.
CTRL:
- Since Wednesday CTRL had one OCD call regarding the SA3/GATT gauges and controller configuration.
- SQS server - issues with interactions. Work ongoing.
- INFLUX roll out: test on CAS monitor topic is successful. For the double loggers, there will be an impact of 5-10 minutes down time when login.
This will happen on FXE (2x10min without data): it is OK for FXE and for LAS today. IF that is successful, CTRL will contact other instruments.
DA:
- Nothing to report
EEE:
- It has been working on the follow-up of HED topics.
- EEE OCD had a call on the SA3/GATT gauge controller.
VAC:
- Severe trouble with SA3 GATT: damage on the gauge controllers was a result of series of problems.
- Bearing of a downstream booster pump make vibration and developed two leaks; then pressure went bad;
vacuum gauges showed wrong numbers and resulting in electronic damage in the controller cards.
- The booster pump was exchanged on Friday; on Sunday a leak test found a broken vacuum welding and part was removed.
Currently ZM3 at Desy is fixing it. Expected to be ready for tunnel installation by 10AM.
- Sunday from 13:00: the GATT was in UHV mode to allow beam permission and lasing in SA1 and SA3 up to Monday 6AM.
- Currently XTD10 is open for GATT repair and PES exchange of turbo pump.
- CTRL mailing service worked quite well: an alarm e-mail of GATT fault was sent every 12 hours.
XPD:
- SA3/XGM valve was also OFF on Friday. Martin Dommach will contact Theo while in tunnel.
DET:
- Business as usual.
ITDM:
- Nothing to report.
XRO:
- Not present due to the night shift for beamline transport alignment activities.
- F. Wolff-Fabris informs the pre-shutters are commissioned by XRO during Sunday's shift.
LAS:
- Nothing to report
UNSYS:
- Nothing to report
SPB:
- Work is ongoing.
FXE:
- Preparing for beam operation: no big problems.
- Thursday we reported issues with motors not moving. EEE helped and it was found the current of the motors were not correctly migrated.
Now motors are working. Currents were set too low - this no risk of damage.
MID:
- Preparing for beam.
HED:
- Preparing and checking equipment.
SCS:
- Preparing for beam tomorrow.
- FFT end station problems reported last week are solved.
SQS:
- Waiting for the beam. |
Notes XO Integration Meeting 15.07.2020, posted by Frederik Wolff-Fabris on 17 Jul 2020, 10:19
|
General Announcement:
- SASE3 tuning will start this afternoon and possibly will last until Thursday afternoon. SCS and SQS will share the closures of their hutches.
CTRL:
- Swap server of one of the two Karabo GUI servers for SA1 (mail to readiness): suggestion is to do Friday 9-10AM. Please let CTRL know in case you have any issue.
- Status on INFLUX roll-out: testing to take place at the end of this week. A coordination with SA1 has to take place due to the running macros.
DA:
- Not present.
EEE:
- EEE is working in FXE and HED with the open issues.
- EEE gives an update of the discussed points on Monday:
Communication: From now on EEE will include the contact persons, as well as to the instrument mailing list.
SA1 motors in OFF state: after a PLC update the safest configuration for re-start is bringing them back in OFF state. EEE will continue with this procedure.
If instruments want differently, please make a list of devices that has to be changed.
HED issue: EEE is talking with Konstantin and found it was just a misconfiguration.
HED pump controller: Konstantin is already working together with VAC;
XPD turbo pump in SA3: no relation with PLC update. Yesterday's ZZ showed that the pump has to be replaced.
- Nerea and Patrick will provide a write up to ensure the problems are followed up and solved.
VAC:
- ZZ yesterday to install the Neon gas minican to allow the monochromator calibration in SCS/cw31.
- Exchange of PES turbo pump will take one day and break the interlock - North brunch will see no electrons and photons.
A date has to be found in agreement with involved groups and facility's operation.
XPD:
- Turbo pump in PES: pump itself is the problem. Vacuum beamline is safe but PES does not operate. We need the PES for some user experiments still in 2020.
- Operation of beam in XTD6 was not possible with multi-bunch due to devices in the beam. We do not know if in reality they were in beam or not.
Karen Appel confirms the devices were out of beam. This has to be followed up.
- This weekend, together with XRO, XPD will do the further recommissioning of beamlines and diagnostics.
DET:
- Vacuum issue reported on last Monday was confirmed to be related to the gauge.
- Started with DAQ tests with instruments. All ongoing.
ITDM:
- Not present.
XRO:
- Not present.
LAS:
- SA2 laser maintenance taking place on Friday.
UNSYS:
- Nothing to report.
SPB:
- Preparations are going smoothly. All good in the SPB's plan.
FXE:
- DAQ test with LPD deployment - all work fines but for one module need to be fixed in the detector. It is already under care;
- EEE/Bernard is finishing the loops.
- Monochromator commissioning next days (we need a ZZ to align the camera).
MID:
- It is scheduled MID DAQ test for Monday next week.
HED:
- Ulf will investigate with Konstantin on the EEE issues. In XTD6 there were two motors with wrong parameters configurations. HED will communicate directly with Nerea.
SCS:
- EEE is needed to solve issue with the FFT end station.
SQS:
- Not present. |
Notes XO Integration Meeting 13.07.2020, posted by Frederik Wolff-Fabris on 14 Jul 2020, 13:45
|
General Announcement:
- XO will reset the calendar invites for the XO Integration Meetings of Mondays and Wednesdays as well as for the User&Operation Meetings on Fridays. Usual times will be
kept.
- Machine is on startup and the RF window will be 400us instead of 600us in the first 2-3 weeks. Instruments can expect about 160 bunches in this time.
- Instruments must take tunnel components out of beam path, especially in case they are connected to the EPS which could potentially block beam operation.
- XPD and HED report on turbo pumps that failed last week. One has to investigate of it is linked to the Karabo re-start due to the DESY-NOC activities last Tuesday.
FXE also reports motors in tunnel components were OFF.
CTRL:
- Further investigations are ongoing with Basler cameras that did not start after Karabo reboot.
DA:
- Not present.
EEE:
- Waits for feedback on the FXE XGM commissioning.
VAC:
- Nothing to report.
XPD:
- SASE XGMs are started and being operated.
- Lasing in SASE2 was confirmed in the FEL-IMG; SA1 was lasing @2mJ last night.
- Re-alignment of SASE3 was not yet confirmed - no beam visible on the XGM or FEL IMG, only spontaneous radiation.
- XPD did a successful DAQ test.
- ZZ in XTD10 will be required to investigate failed PES pump. This will take place on Tuesday.
DET:
- Poor vacuum SPB/AGIPB last Friday: under investigation (may be a 40 gauge) and will be followed up next days.
ITDM:
- New DAQ version was released to fix detectors and slow data acquisition: tested in SPB and FXE (Agipd and LPD).
- This week ITDM makes upgrades in all instruments but MID takes place next week; SCS still needs to agree on the date for this week.
XRO:
- Plan for the week is the recommissioning of the beam transport. XRO has shift plans for the weekend.
LAS:
- All lasers are running.
UNSYS:
- Nothing to report.
SPB:
- Vacuum issue with the 40 gauge under investigation by DET.
- No big problems for the start of next week.
FXE:
- DAQ test taking place today.
- XGM is going with the plan but for a cable that may need to be exchanged.
- Upgrade of Beckhoff system: motors were not moving. There is a lack of communication between the Beckhoff upgrade, control support and instrument scientists.
MID:
- Not present.
HED:
- HED had to check and/or entry the motor parameters after the reboot of the control systems that place motors in OFF state.
CTRL informs the information must be stored in the PLCs / Karabo. It is solved but it demanded a lot of work during last week.
SCS:
- Not present.
SQS:
- Problems occurring with turbo pumps. There is a particular controller in one rack that had to be replaced 7 times in 2 years.
EEE and SQS will investigate and do inspections on this rack. |
Notes XO Integration Meeting 08.07.2020, posted by Frederik Wolff-Fabris on 10 Jul 2020, 13:50
|
General Announcement:
- Machine start-up continues to be ahead of schedule. Beam transmission to T4D and T5D has been done during the night (with open undulators).
- An Invitation for the User&Operation Meeting on next Friday 09:00 was sent to the user&opmeeting mailing list. If you did not receive it, please contact XO.
- XO Integration Meeting takes place only on Mondays and Wednesdays at 09:30AM.
CTRL:
- The synchronized shutdown, f/w update, configuration and restart of both CTRL network routers by DESY-NOC was performed yesterday.
This included the Karabo shutdown and restart in tunnels, instruments, etc.
- Problems linked to the re-start: in LAS a miss-behaving ADC and a faulty host cpu issue; in CTRL some obsolete deployment information.
- This Router update has to be schedule during the shutdown to avoid above mentioned issues.
- CTRL observed two Basler cameras which did not reestablish a network connection after the router updates: Please try power cycling the camera to restore system.
DA:
- No complications due to routers update.
EEE:
- Cabling installation in SCS is in progress.
- Technical staff will commission the FXE's two new brakes for XGM.
VAC:
- Not present.
XPD:
- Re-starting X-ray photon diagnostics devices this week.
- XGMs are ready for recommissioning.
DET:
- Nothing to report.
ITDM:
- Maintenance of ELOG server and update of application has been done last night. System has restarted.
XRO:
- Not present.
LAS:
- All LASERS are back today.
- Due to the router update, SA1 laser was the most affected due to loose of thermalization and was OUT of operation a full day. SA2 was OUT only for the afternoon.
UNSYS:
- Nothing to report.
SPB:
- Getting ready for beam: few DAQ issues still open but being worked out.
FXE:
- All motors in SA1 were OFF due to Karabo re-start linked to router update.
MID:
- Update of Linux computers: ITDM will communicate with MID for the re-installation.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 06.07.2020, posted by Frederik Wolff-Fabris on 07 Jul 2020, 13:23
|
General Announcement:
- Since the end of the shutdown last Friday, the current Program is "Access/Tuning/Setup";
- DESY/MXL has started ramping up the Gun, powering and testing the Magnets and phasing the RF stations.
- Currently it has electron beam transmission up to the TLD and is ahead of the planned schedule for these days.
CTRL:
- Data logger issue: Friday CTRL has rolled back to the old data login system. Meeting today at 11:00 with involved groups to define next steps.
Information will follow via readiness mailing list.
DA:
- No issues to report.
EEE:
- Issue with HED PLCs: a fuse has been tripped and it has been replaced. A CPU was broken and is believed to have caused the fuse tripping.
This is the first time this happens in the facility.
- Cabling activities in SCS is scheduled to take place this week.
VAC:
- Nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- ITDM has made a request to DESY-IT for a routers restart. This will cause the network to be interrupted and restarted. Also Karabo will restart.
A meeting today will define the proper time for this action, currently schedule for tomorrow morning. Information will be send to the readiness mailing list.
XRO:
- Not present.
LAS:
- All lasers are up and running again.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- Few problems with motors were already solved last Friday by EEE.
HED:
- All going fine.
SCS:
- Not present.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 03.07.2020, posted by Frederik Wolff-Fabris on 03 Jul 2020, 14:21
|
General Announcement:
- The Interlock tests are finished in all tunnels and XHEXP1 areas, following information from D3 and SRP.
- From next week we will continue with XO Integration Meeting only on Mondays and Wednesdays at 09:00. The Fridays' XO Int. Meetings will be terminated.
- Next Friday 10.07 the User and Operation Meeting will re-start at 09:00 in the virtual format. Invitation and corresponding Zoom link will be send in a separate e-mail.
CTRL:
- Preparation for operations has started. Three issues have been addressed:
1. XPD issues are currently been worked and will extend over next week;
2. XFEL gateway troubles seems linked to a router configuration;
3. Migration of old data logger: problem with MID LA2 and SA3 (running wrong migration).
- Recovering complex configurations: meeting 12:00 with ITDM - please wait until Monday for update.
DA:
- Not present.
EEE:
- Follow-up of the shutdown activities is ongoing; Modification of PLCs was done.
VAC:
- Vacuum at beamlines are running; Shutters are ready.
- VAC plans to open XTD6 again from next Monday up to Friday, allowing HED access and work.
- Please inform VAC if XTD09 is needed to be accessed. Currently a ZZ is foreseen for this tunnel (EEE request).
- XTD10 can be accessed by ZZ during next week.
XPD:
- Small bugs on the DOOCS side of XGM were observed: bridges were checked and looked fine.
- XPD is planning the re-start of devices and support for machine tuning in coming weeks.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Not present.
LAS:
- SA2 laser: Train-ID showed issues (Update software failed).
- SA1 laser: went down due a crate failure and is warming.
- SA3 laser is fine.
UNSYS:
- Nothing to report.
SPB:
- AGIPB update needs to be fixed.
FXE:
- LPD new DAQ could not save data.
- Calibration pipeline for LPD is OK.
- Calibration pipeline for JungFrau does not work - also CTRL fixed the crash with DAQ (how the image gets the train ID's). The crash is not observed anymore.
MID:
- Discovered problems with motors in Laser room: EEE is informed and investigates.
HED:
- Will need access to XTD6 for final checks from Monday. Please inform VAC when access is finished.
SCS:
- Not present.
SQS:
- Not present. |
Notes XO Integration Meeting 01.07.2020, posted by Frederik Wolff-Fabris on 01 Jul 2020, 13:53
|
General Announcement:
- Interlock tests are starting in SASE3 instruments right now. Zunaira, Eric and Wolfgang Clement are coordinating the TUV tests in the XHEXP1.
- MXL announced a draft for the machine start-up schedule which is published at the BKR e-log. The link to the schedule is:
https://ttfinfo.desy.de/XFELelog/show.jsp?dir=/doc/Schedule&pos=2020-06-29T15:40:57
CTRL:
- Group has a plan to switch the data loggers today; MID was already done in the past; Migration of old data was already done in advance.
- There was a report of issues with the GUI server. Currently it is under investigation. A workaround is to disconnect and reconnect again or move to a second GUI.
More information will follow.
DA:
- DA has two major plans for the upcoming days/weeks:
1. Implement auto triggering in pipelines for the instruments (used for data analysis); 2. Stablish a plan to have spread knowledge on development tasks.
EEE:
- Installation of a SCS component is arranged for July 8th.
- Cabling in HED is scheduled to be performed in cw31; EEE will contact FXE to fix a cabling installation date.
- More than 50% of Beckhoff modules are changed.
- Working on PLCs business as usual.
VAC:
- All fine: nothing to report.
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- DESY/NOC needs to perform software update of the control network routers for all SASEs.
This will take place next Tuesday, the 7th of July, between 6AM and 8AM in the morning. An e-mail will be sent to the Readiness list.
XRO:
- Not present.
LAS:
- All lasers are running.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to add.
MID:
- All running fine.
HED:
- Nothing to report.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 29.06.2020, posted by Frederik Wolff-Fabris on 30 Jun 2020, 08:50
|
General Announcement:
- The Photon Run Coordinator will re-start next Friday July 3rd. Please also feel free to contact him/her (9-4800) for daily coordination and communication with the
machine/BKR.
- Interlock / TUV tests are ongoing in tunnels. Next Wednesday it starts in the experimental Hall / SASE instruments.
CTRL:
- CTRL has a plan to switch the Data Loggers on Wednesday. This will cause a downtime on this service for about 5 minutes. A follow up will be announced afterwards.
DA:
- Not present.
EEE:
- EEE has made an agreement with HED on the cabling issues.
VAC:
- Tunnels are closed: maybe next week after Interlock tests the XTD6 and XTD9 tunnels will be open for a couple of days.
- If access in tunnels are needed (not only XTD6 and XTD9), please make a request by filling the table at the Desy/BKR elogbook
(https://ttfinfo.desy.de/XFELelog/index.jsp), located under "Access Requests".
XPD:
- Nothing to report.
DET:
- Nothing to report.
ITDM:
- Nothing to report.
XRO:
- Not present.
LAS:
- All lasers are fine.
UNSYS:
- Nothing to report.
SPB:
- It has tested the new Karabo for the Jungfrau detector and result is OK.
- This week SPB plans to test cameras.
FXE:
- Nothing to add.
MID:
- All fine.
HED:
- All fine.
SCS:
- Not present.
SQS:
- Nothing to report. |
Notes XO Integration Meeting 24.06.2020, posted by Frederik Wolff-Fabris on 25 Jun 2020, 10:27
|
General Announcement:
- Exceptionally next Friday the XO Intergration Meeting will not take place. The meeting will resume next Monday.
Please send us an e-mail to xo@xfel.eu in case of any issue.
- Details on the Interlock Tests:
1. D3/SRP has the responsibility to carry on the tests;
2. TS and XO (communication) are supporting groups;
3. There is no rehearsal date;
4. Tests on the Instruments/SASE wll take place on July 1st, 2nd and 3rd (reserve date);
5. Access to the instruments and optics hutches is forbidden;
6. Access to the laser rooms are not restricted, apart from the HED laser which needs clarification;
7. Emergency off will be disabled on July 1st at 08:00AM. As soon as the emergency offs are tested, they can be enabled again;
8. The participating EuXFEL Radiation Protection Officer can inform the instruments as soon as tests are ready with the respective SASEs for resuming access and work.
- Maurizio V. and Nerea J. mention that an EPS check must be scheduled prior beam delivery. XO will check with Zunaira/SRP on its status.
- Suren K. points the PKG in SA3 undulators is still switched off. TS shall switch it on prior tunnel closure.
- Suren K. asks on tunnel access for next weekend. XO states tunnels will be put in Interlock tomorrow (June 25th) and shall remain interlocked until the next shutdown.
Access to XTD6, XTD9 and (maybe) XTD10 can be done for few days after the interlock tests. Access to all other tunnels must be via ZZ.
Tunnels will not be open for the weekend. (after the meeting DESY/MXL confirms this procedure).
CTRL:
- Nothing to report.
DA:
- Working on small requirements with the instruments.
EEE:
- SA1 VAC signal's issue from last Monday: it has been fixed. There was problems related to filter in the train-ID communication: the addresses were not properly set.
- Cabling is ongoing on SCS; HED cabling waits for time/schedule for installation (to be discussed between EEE and HED);
Cabling works ongoing with DESY-MPS which connect MID to PPS.
VAC:
- Closing the activities - SA1 VAC signals are OK.
XPD:
- Nothing to report.
DET:
- XO asks for an status on the MID/AGIPD train-ID issue from last Monday. S. Aplin and K. Wrona say it has to be further investigated as is not clear where the problem
lays.
The DAQ gets data from AGIPB, DOOCS, Karabo, labelled with train-ID and the labeling have to be checked for consistence. This is not a new issue and has to be addressed
in the frame of data reduction.
- A clear independent signal has to be also checked. P. Gessler says it is needed to get various patterns for a complete understanding. A clear strategy to develop and
address has to be made.
- S. Aplin will overlook the efforts among different groups.
ITDM:
- DAQ installations: this morning a fix for big detectors is done; after leaving DAQ for longtime there are some issues that will be fixed next week.
- Majority of acceptance tests are done - slow data is fixed.
XRO:
- Everything is done. The last check on tunnels was performed.
LAS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to report.
MID:
- Nothing to report.
HED:
- Finished with the tunnel components check.
SCS:
- Nothing to report.
SQS:
- Nothing o report. |
Notes XO Integration Meeting 22.06.2020, posted by Frederik Wolff-Fabris on 23 Jun 2020, 14:12
|
General Announcement:
- The User & Operation Meeting that took place at beginning of this year on Fridays will re-start on July 10th.
Proper announcement and Zoom link will be sent. In this context, the XO Integration Meeting will continue only on Mondays and Wednesdays.
CTRL:
- Some of XFEL Elogs were not working during the weekend. ITDM is taking a look into the problem.
- SA1 Vacuum devices show some errors (not running). It seems the problems comes from the PLC and is under investigation.
The controllers in the tunnel were OK and pumps were placed in safe state.
This has affected all SA1 loop, at XTD2 and XTD9. Also the devices were not properly communicating.
DA:
- No issues to report.
EEE:
- Issues with the SA1 VAC loop.
- Updates of timing systems affecting LAS were successfully done on Friday. EEE encourages people to report in case of problems.
VAC:
- Reporting on the SA1 VAC loop issue.
XPD:
- Last Karabo update done on Thursday.
DET:
- No issues to report.
- MID observe a train mismatch between AGIPB and data sources such as SA2 XGM. The AGIPD has a shift of 1 train-ID.
This issue involves several groups and coordination and solution still miss.
- LPD/FXE also takes the train info from clock control but shows no mismatch.
ITDM:
- Investigating the Elog access issue.
XRO:
- Not present.
LAS:
- Maintenance in SA3 and SA2 are done. All lasers are running.
UNSYS:
- Nothing to report.
SPB:
- Nothing to report.
FXE:
- Nothing to report - checking the vacuum.
MID:
- Report on the AGIPD train-ID.
HED:
- Working with big lasers.
SCS:
- Nothing to report.
SQS:
- Nothing to report. |
|