General

• User operation ongoing in SPB/SFX and FXE

Instruments

• FXE:
  • User operation started during the night
  • Some concerns over running LPD as the sample chamber is from diamond and direct reflections could cause damage
  • Generally instrument worked very well
  • Synchronization with PP laser is now in a state where it can be tested
• SPB
  • Many data taken for detector calibration
  • Severe issues with KARABO
    • Response is very slow
    • Major crash this morning
    • Detector controls complex and instable
  • A failure of AGIPD related to FPGA not understood
  • Strange interlock issue: EPS interlocks appeared disabled

Tunnels

• L. Samoylova spent many hours on Wednesday to set up the DAQ in the tunnel
• With the help of many support people she managed to get the system to work
• The system is instable and also seems to record many more parameters than are needed
• Use of the system is very clumsy
• Comment from ITDM (L. Maia): The data sources were specified by the users. It is acknowledged that the system needs a lot more work
• Diagnostics is working with CAS in a squad to work down a list of open redmine tickets
• The DOOCS - KARABO bridge is now working. The FEL imager is routinely available in DOOCS to the BKR crew
• DESY set up a statistics server to help collect data about beam pointing instabilities

Laser

• Local pattern generation synchronized to machine timing is working
• Overlap finding together with FXE was done but has to be tested under real condition next week
• A few suddent shutoffs of the front end occured. The origin might be understood
• There are large timing drifts in the ADCs that are currently not understood (up to ms)

ITDM

• Next Wednesday there will be a 1h shutdown of the load balancer in the DESY RZ. This should affect mostly the web services
• It should be noted that the DAQ ran quite stably in SPB and FXE. More than 20 runs were collected in each case without a restart

CAS

• Two hot fixes are out (2.1.18.4/5). These address  very specific issues
• A major new release (2.2) is planned for the winter shutdown. This will be mandatory
• Comment from instrument: It must happen early so that wrinkles can be ironed out!)

General

• Preparation for the user's run starting tomorrow (until 7:00 Tuesday morning)

Instruments

• FXE:
  • On Monday night the DAQ seemed to be slow. This problem was caused by a misconfiguration in the DAQ which could be solved by ITDM once they were informed. In general both the Configuration understanding by the users and the communication with ITDM has to improve.
  • The Gotthard detector could not be used. The Greateyes detector is being used now and works properly.
  • PP laser is now synchronized but not commissioned yet. This is going on now (crucial point is the pulse width measure). It is not yet clear whether the PP laser will be used in the next users run. 
• SPB
  • AGIPD was moved 5 m downstream for the next experiments. Vacuum is OK. Alignment checked with the ALAS, will be checked with beam probably today evening.
  • A change in the DAQ configuration has to be (is being) done today (CAS/ITDM)

ITDM

• Upgrade switches today at 9:00 will be checked if there were problems right after this meeting.

General:

• User Operation ongoing until Tuesday 5 Dec 8 AM
• CAS shall present a release plan for KARABO 2.2 next week
• Dispatch meetings shall move to a Mo/Wed 13:00 schedule starting next Wedneday 6 Dec

Instruments

• FXE:
  • Scantool was successfully used to establish overlap with laser beam (Tangerine) 
  • Some problems with slow DAQ in particular also for tunnel components
  • Comment from ITDM: There are known issues with the slow data acquisiton. A new version exists that was intentionally not deployed before the end of UOP
• SPB
  • Just finished first shift of laser user experiment
  • No big problems
  • All goals in the optimistic schedule were achieved

Tunnels

• Currently preparing for the shutdown
• Sticking condition of mirror bender could be reproduced in the lab
• WP74 just installed the last missing pieces in XTD1

Laser

• Running
• temporal and spatial overlap with beam is achieved
• But now instruments do not intend to use the PP laser for the last experiment

Detectors

• Both detectors are working
• The DAQ for slow data is crashing occasionally
• Gotthard is fully operational but there problems getting the data out. Help from PSI is needed

AE

• No problems

ITDM

• No problems

CAS

• CAS-XPD Running:
  • We will finish next Monday
  • We have been addressing issues related to MCP, Karabo-DAQ, Cameras, and K-Mono
  • Deploy K-mono today
  • Review DAQ Groups in Karabo: Cyril (interface to ITDM), Liubov, Koch
  • Gotthard we have to address in a separate and more systematic discussion: trigger, operation, calibration...
• XGM memleak - we already have a fix to get it stable but have to check when we will be able to upgrade (only device).
• Internal discussion about CAS/Karabo priorities for the winter shutdown

General:

• User operation with good beam conditions over the weekend
• One interlock trip in the FXE branch in XTD9. Restored by CAS
• Shutdown starts tomorrow at 08:00.
• Next dispatch meeting takes place Wednesday at 13:00. A reminder will be sent.

Instruments

• SPB:
  • Started with the actual sample during the night shift
  • Some DAQ issues
  • 108TB data was recorded
  • Problems appear to occur between 01:00 and 02:00  and between 13:00 and 14:00 
• FXE
  • Running is ok. 
  • Temporal and spatial overlap with optical laser established
  • Currently taking pump-probe data
  • People on shift were uncertain about how to recover from interlock trip
    • Statement from Vacuum: Pumps can restarted/valves open if the respective button in KARABO can be clicked

CAS:

• Delay between data taking and "online" analysis in the "Hummingbird" software is about 0.3s

ITDM

• On Friday, there was a global network outage affecting the Office Network on the DESY site.
• The transponder system in the experiment hall was down
• SRP organised a guard at the hall entrance so that the hall had not to be evacuated
• The control network (XDAC) was not affected

General:

• Shutdown started.

Instruments

• SPB:
  • AGIPD prepared for exchange modules intervention
  • AE is in contact with SPB for crates issues fixing
  • The schedule for next year, especially when the experiment has to be ready before the users run has to be known as soon as possible.
• FXE
  • Starting loops update in the next days. Reason for the small delay is that FXE is checking all the default parameters to be able to restore them after the update.
  • In order to solve some DAQ issues, FXE requested to have their own topic. AE agrees, ITDM agrees, CAS is trying to find out how to do it and to do it before the next users run. Technical details have to be discussed offline with FXE.

ITDM

• in KW 51 (18-22 Dec.) the DAQ and online cluster will not be available due to recabling works in the balcony rooms.
• Update of the servers in the experiments has to be done. ITDM requested to know from CAS when the major update of Karabo is taking place to coordinate the effort and minimize the downtime. CAS will present a schedule concerning Karabo this Friday.
• 700 Tb of data are stored on the online cluster. It has to be communicated by the experiments what can be moved from there, since there is almost no space left.

Diagnostics

• Implementation of Gotthard detector in Karabo is still a problem. S.Hauf will invite the main groups involved for a meeting to find a solution.
• The data correlation with time stamps when using the Karabo DAQ has to be done carefully and contacting WP74 since there are still issues present. 

AE

• Activities on the Hirex in SASE1 will need the MOV loop to be brought down in the afternoon. An entry in the ELOG will be written.
• SPB loop 8 prepared. 
• Full list of new crates belonging to the motion loop in SASE3 tunnels has to be delivered to AE
• Full list of new crates belonging to the IPS loop in SASE3 tunnels has to be delivered to AE in order to be able to close the loop.
• Technosoft controller issues in SASE3 will be attacked once the SASE1 activities will allow it.

VAC

• VAC testing ongoing.
• 2 mirrors will be installed before Christmas
• Motion crates for the motion lopps and test of the VAC interlock will happen this week.

The Motion loop in SASE1 is totally operated again.

Please note following info from DESY:

Author: F. Saretzki



Ab sofort werden die Rack-Löschanlagen in den südlichen Tunneln in Betrieb genommen. Die Anlagen bleiben danach scharf.



 

General

• Starting from next week shutdown activities will be tracked using M. Aresté's list
• Dispatch meeting on Monday and Wednesday at 13:00
• Next Friday Readiness meeting also at 13:00 because of conflict with XDAC

Instruments

• FXE:
  • Currently focussing of implementing all default parameters into the next firmware release
• SPB:
  • Major detector work with AGIPD disassembly started

Tunnels

• Currently testing firmware and software in SASE3
• Installation of first mirror starts next week. Mirror installation takes about 1 wk/mirror
• CRL in XTD2 will be vented to align internal mask and fix stuck arm
• Drilling takes place at the end of XTD9 for the installation of the FXE diamond grating
• Mirror cooling will not be installed this shutdown. It is not needed for the planned bunch rates and might introduce unwanted effects/vibrations
• Activities of Diagnostics:
  • Photonics Science camera in SASE1 exchange.
  • Tests are being carried out in SASE3 together with J. Tolkiehn
  • Cabling is going on in XTD1 on the XGM
  • There is a noise problem on the Gotthardt detector in the HIREX. Need support from AE to find the origin

AE

• Motion loop in SASE3 will be split into mov1 and mov2. This will allow to sort components into more and less stable sets and keep one of the loops up most of the time

Laser

• Laser activities had to start late due to problems with AC and laser interlock (delays of 6 and 5 months respectively)
• Availability of trigger pattern from the machine was also a problem
• Specs could just be reached for the Nov user period
• Eventually the PP laser was not used for any of the experiments
• Work on the laser is far from finished and will require 3 more months to come to full funtionality
• Currently, the Karabo version is being upgraded and the most up-to-date PLC firmware is being installed
• These updates had intentionally been delayed until the end of the user period.

Detector

• Detector group not present at the meeting
• Information from SPB: major work is ongoing on AGIPD

AE

• Train builder is being tested as it was not used so far. This required re-cabling
• PLC motor loops are also being split in SASE1
• Many activities in SPB
  • Remaining modifications of modules have started
  • Export of input for firmware generation for loop 8 has been provided by EETF
• Waiting for go in FXE
  • Need to work on the interface for the EPS system
• No tasks currently from laser group
• New information available on SmarACT stages: The observed problems arise from a cable problem and can be solve by configuration
• The problems with movers in the sample chamber in SPB (non-functioning encoders) arise from cable cross talk at the vacuum feed through. This require hardware re-design
• In SASE2 there are tests coming up together with the diagnostics group

ITDM

• Train builder tests are scheduled next week
• Recabling of Infiniband network happens on 18/19/20 Dec. No DAQ and controls available on these days
• A summary of DAQ statistics was shown (see slides on INDICO)

CAS

• Release of KARABO 2.2 on 18 Dec
• Freeze of functionality on 21 Jan
• Discussion reveals that a discussion on specific deployment dates and needs of different groups is urgently needed (Instruments, Tunnels, DAQ)

FYI

Author: F. Saretzki

English version below

Ankündigung:

Aufgrund von Sachverständigenprüfungen wird es am folgenden Termin zu Auslösungen des Evakuierungsalarms kommen:

Gebäude: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 und 8
Datum: 13.12.2017 ca. 09:00 Uhr – 11:00 Uhr


Sie brauchen bei ertönen der Sirene in diesen Zeiträumen das Gebäude nicht verlassen.

============================================================================
Announcement:

Due to expert testing there will be an evacuation alarm on the following date:

Building: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 and 8
Date: 13th of December 2017 between 09:00 a.m. and 11:00 a.m.


You don’t need to exit the building while the alarm occurs.

 

General

• Meeting on Friday will take place at 1300 due to overlap with XDAC
• M. Areste shall ask for updates on task list on Thursday

Instruments

• SPB
  • Work on AGIPD is ahead of schedule. Closing of detector should happen at the beginning of next week
  • Additional installations for roughing pump system is going on and will probably finish this year

Tunnels

• Diagnostics group currently working in the Southern tunnels, cabling components and aligning imagers
• Colleagues from Russia are coming 3 - 20 Jan to work on MCP in SASE3
• KMono local tests in SASE3 should also happen in early Jan
• Diagnostics requests scenes for XGM in XTD1

AE

• Vacuum loop in SASE3 will be modified. Some components shall be moved to a separate loop, e. g. Soft Mono
• SPB tunnel loop will be re-deployed to include the attenuator
• The motion loop in SASE1 will also be split to allow more rapid updates for newly integrated components
• Z-motion crate for detector in SPB will be done until next week or early next year

CAS

• Currently working with AE and ITDM on refining release schedule for KARABO
• Testing the version for release next week. This will then first be used by ITDM and is needed for the firmware upgrade of AGIPD
• Work is going on on the XGM train ID problem
• FXE will be moved to a separate topic (broker) to disentangle some dependencies at least temporarily
• FEL imager performance problem was investigated. This runs on a server that is very busy and should be moved to a different server

PSPO:

• M. Areste presented a report about the progress so far achieved in the shutdown (see Indico for details).
• Most tasks are ongoing, on schedule.
• Few still have to be scheduled, mostly because support groups wait for feedback.
• C. Youngman does detailed coordination of SASE3tasks.Will organize meeting on Tuesday 9:30 for SASE3 tunnel only
• We will review the progress again in Mid-January!

SPB (M. Sikorski):

• AGIPD was tested.
• Roll back to crystallography configuration is ongoing.

FXE: No one attended.

Vacuum (M. Dommach): 

• 1st mirror M1 in SASE3 installed, chamber closed, pumped down & leak tested
• Preparations for 2^nd mirror in SASE3 installation ongoing, to be installed next week.
• MCP taken out of SASE3, put in parking position and pumped down again.
• Grating chamber from FXE was installed by JJ X-ray, is to be pumped in CW 51.
• SASE2 XS2 beam pipe behind shutter and lead wall installed, ready for TÜV inspection next week.

Diagnostics (A. Koch):

• XGM in XTD1 is almost cabled, more cabling was done in XTD10.
• Gas supply system in XTD1+6 was worked on.
• Photonics Science camera was tested.
• Trouble with the motion loop in XTD2 and XTD10, reported to AE.

AE (P. Gessler ):

• MCP motion and k-monochromator were tested.
• Motion loop was split in SASE1 tunnel.
• Many tests, lots of work in progress, need to act quick as time is short.
• T. Wilksen (DESY) is going to update the timing hardware è devices would have to switch to local timing, and switch back later to be operable.

Laser (M. Lederer):

• Further commissioning of the PP-laser was started.
• Upgraded Karabo and PLC firmwares. Works are on schedule.
• Measurements on the dependence of humidity are ongoing.
• DESY is working on the timing system, laser synchronization will be down until the 21.1.18.
• PP-laser pattern selection with machine timing:DESY is now pushing for the originally planned implementation. DESY will have to do some software development in this context.

Detectors (J. )

• Tested the AGIPD @ SPB, but 1 board was not working. Detector consortium could not provide a replacement board, so only half of the detector was tested before SPB/SFX had to close the detector vessel again.
• LPD was tested with train builder, but only tests using 2 memory cells could be conducted (2 images per train), so train builder was not tested with all memory cells.
• More tests with full load (i.e up to 510 images per train) still have to be performed.
• Test with DSSC will take place in HERA South in CW 51.

ITDM (K. Wrona)

• Monday til Wednesday there will be no DAQ due to extension of the Infini-band bandwidth, upgrades, firmware updates, recabling etc.
• DAQ up again on Thursday

Software (S. Brockhauser)

• CAS-People are currently working only in SASE3. In the future, they can either work on SASE3 or SASE2, but not both!
• Slides with Project-plan provided by Hugo Santos (see Indico)
• Release of Karabo 2.2.0 on the 18.12. (in the meantime shifted to the 20.12.)
• Testing by ITDM and consecutive deployment in tunnels and at the instruments takes until 22^nd of January
• Karabo 2.2.1 to be expected mid of January
• Further decisions will be taken mid of January

Dear all,

This is a reminder to inform you that Readiness meeting will resume today at 13:00.

Best regards and Happy New Year !!

PSPO:

- G. Wellenreuther inform that DESY is commissioning a new gun in the injector that could take longer than expected. Harald Sinn is in a meeting today to disccuss this issue, any change in the scheduled will be reported.

SPB:

- Lost motor positions in the tunnel, CRL in defined state. Good state in general no major problems expected.

- Detector was clabled again.

- Loop updates still pending, SPB request to have loops ready during this week.

FXE: No one attended

Vacuum:

- SASE1:

• Some works in CRL for XTD1.
• Working in FXE grating chamber, necessary to update Vacuum loop.

- SASE3:

• M1 and M2 mirrors were installed before X-mas. Mirrors installation in K-mono to be finished next week.
• Created new loop this affects Vacuum loop, update necessary.
• XGM loop need to be updated.
• Motion loop 1 is working, some works in MCP motors on going.

Diagnostics:

- Technical commissioning of MCP on going together with the Russian collegues.

- MCP SASE2 will be transported to XTD6 on Wednesday afternoon.

AE:

- Working to implement redundancy into FXE loops.

- To agree a date with FXE to work in the grating chamber at the end of XTD10.

- Motion loop for SASE1 will be interrupted this afternoon at 14:00 to add some terminals for mirror 2.

Software:

- Deployment of Karabo 2.2.0 will start this Wednesday in FXE.

- Next dates for Karabo deployment are: 15th January for the Tunnels,  22nd January for SPB.

TS:

- Torsten Schoen inform that it is necessary to carry out an 'Emergency off button' test during the Shutdown.

- SPB agreed to perfom the test the 22nd January in the morning.

- For FXE the proposal is to perfom the test the 17th January, to be confirmed with AE and FXE.

Dear all,

according to H. Sinn the closing of the XTDs 2, 4 and 10 will be delayed until the 30th of January.

Cheers, Gerd

Dear all,

we got the information from MKK, that they have to fix a leak in the main compressed air line. Therefore they will close the valves to the tunnels for approximately one hour starting today at 1 p.m.

Please do not use the compresed air system in the tunnels during that time with air driven tools or air guns, as the volume of the distribution systems acts as a buffer.

Regards,

Martin

PSPO:

- G.Wellenreuther inform that MKK is planning to test the Umschaltautomatik, this will affect XHE1, XHE3, XHPSC, XHEXP, XHVAC. Oleg Engelmann will send a mail with detailed information.

SPB:

- AGIPD is fully occupied again and restarted this morning as scheduled. Everything works well so far and it is expected to finish the test on time. Further test with new Microcontroller are scheduled for next week.

- Most of the loops that required modifications are updated or on going.

- Module BKF00128 will be modified this week, update of Loop 5 will folllow.

- Loop 12 is ready to be deployed, to confirm if modifications in the modules are finished and reorganization on rack UH2.12 is ready.

- It is necessary to correct an error in module BKF00220,  AE will agree with SPB  when this will be possible.

FXE:

- New Vacuum device in operation.

- Karabo 2.2.0 being deploy today for FXE.

AE:

- Replace one Micro-TC in SASE3, a cooling unit is failing and one module will be replaced by a new one today. This is providing timing for XGM and K-mono. AE will inform by the logbook before to start the replacement.

Dear Monica,

Just one small correction at "Karabo 2.2.0 being deploy today for FXE." The version is, actually, Karabo 2.2.1 that we released yesterday.

Hugo

Hi,

the FXE Mono's won't be available via Karabo from 15.30 onwards due to an update from CAS (karabo) and ITDM (server).

Best, Dennis

Hello everyone,

The main Timing System at DESY, which distributes all timing information to our systems, is right now configured to generate 1Hz trigger signals on Event #16. This is the trigger that is deliver to most detectors/cameras on our site.

DESY is testing a new Gun and wanted to use a slower trigger frequency during this evaluation.

We were not inform in advance about this reconfiguration. After a few colleagues from SPB approach us reporting this problem, we contacted DESY and got the update. We will take steps to make sure that future Timing changes done by DESY are communicated to XFEL ahead of time.

Dear all,

an update of the Timing boards firmware and doocs server is require. This operation should not take more than 10 minutes per device; during this time no timing signals will be generate (clocks/triggers/bunch patterns/etc.). Naturally, problems may occur during this update which will lead to increase down time.

I will perform this necessary update in the devices locates in SASE1 and SASE3 tunnels today at 11H30. If somebody sees a problem please contact me.

Concerning the devices in the Experiment Rack Rooms, I will contact a responsible person to schedule a time slot where this update can be perform. If are one of these people contact me.