General

• The latency observed by SPB in the GUI client has been almost completely fixed. Still some "freezing" from time to time remains.
• DAQ tests in FXE with LPD worked for 12/16 of the detector. Trigger's signal stability issue being addressed by AE.
• XTD 9 and 10 will remain open from 25.07. at 7:00 to 26.07. at 14:00  
• On Wednesday afternoon the machine will be prepare to lase with 30 bunches in order to allow the radiation measurements in SASE1 Exp. Hall on Thursday and Friday.

XTDs

• Fast valve in XTD9 still has issues with noise. Issue will be addressed tomorrow with AE support.

SPB

• Integration of the turbo pumps in  the VAC loop ongoing.
• Chiller control being included in Loop 5
• Firmware for the AGIPD hv power supply was updated. The spikes during ramping are gone
• SMARACT motors being implemented. (depend on Loop 13)
• Work on in vacuum interlock in SPB will probably start this week.
• Some problems with Karabo (While measuring profiles the level scaling does not work, Region of interest values wrong) have been reported to CAS.

General

• Today the machine is being prepared to lase with 30 bunches in order to allow the radiation measurements in SASE1 Exp. Hall on Thursday and Friday.
• Today GUI will be down for a couple of minutes at 11.30 to allow modifications of the servers for bugs fixing.
• Specific SASE1 areas will be off limits during the test, scheduled for    SPB OPT 27.07.17   12:00-16:00 ,  SPB EXP 28.07.17 9:00-15:00   and  FXE 03.08.17  9:00-15:00.  Please address Eric Boyd's presentation for details. A mail reminder will be also sent to notify the beginning and ending of the tests.
• The machine will be running with 30 bunches during the test and with 1 bunch during the night. Since no EPS is implemented, screens and sensitive components need to be taken out during the 30 bunches operation.
• New version of Karabo ready to be deployed. The time slot when this will happen will be discussed on Friday,
• Screens controlled by Karabo cannot be read from DOOCS. The Karabo/DOOCS interface problems will be be addressed with high priority by CAS.

XTDs

• Fast valve in XTD9 still has issues with noise. Issue will be addressed tomorrow with AE support.

FXE

• The detector group is updating the firmware for the LPD to fix the trigger signal problems. A test on just 1 quadrant will be done today together with ITDM
• Interlock implementation in Loop 1 ongoing.

SPB

• Integration of the turbo pumps in  the VAC loop ongoing. Interlock possible.
• Chiller control being included in Loop 5 - Crate has been modified this morning. Loop will be updated checking that this will not stop the turbo pumps.
• SMARACT motors being implemented. First motor is ready to be tested today after hardware problems have been fixed.
• Work on in vacuum interlock in SPB ongoing.
• The ELMO controllers for the motors of the liquid jet manipulator in Loop 13 should be brought and ready by CW 32. AE working on that. 

Tunnel Open.

Mirrors:

Status:

• Vacuum on M2 should be good enough to switch on ion pumps
• Surveying work going on 

Vacuum Loop:

Status:

• Several vacuum sections successfully tested in XTD9 yesterday
• Tests need to proceed in XTD2 today
• The protection switch on the ion pumps cannot be actuated remotely 
• Loop needs an update for the XGM
• XGM was in the vacuum interlock overnight. Will be taken out during the day but should be put back at the end of the day
• Interlock of High Voltage with vacuum in XGM not working yet
• V0 valve needs to be exercised in collaboration with DESY 

EPS Loop:

Status:

• The terminal exchanged for the FEL imager was not the broken one. Another terminal needs to be exchanged today
• The MCP needs to be moved out of the beam manually. This will be checked using the EPS switch. For this the EPS loop is needed

Motion Loop:

Status:

• An improved fix for the motors not stopping was deployed yesterday
• The loop needs to be reconfigured to replace the SRA motors 
• One broken motor was exchanged on the mirror chambers and needs to be tested

Imagers

Status:

• All motion tests in XTD2 done. Now continue in XTD9
• Photonics camera driver still not working but not critical for tunnel closure (there is also a workaround by the name of ebus player)
• EPS switch on FEL imager needs testing  after terminal replacement

Karabo:

Status:

• New version with TCP was deployed. Experience overnight is good

Tasks for today:

Note: Loop reconfiguration and updates of Karabo to be done in the morning. Testing can resume after lunch. Status updates should please be recorded in the log

• Reconfigure vacuum, motion loop (updates for XGM interlock, SRA motors)
• Replace broken terminal in EPS loop for FEL imager 
• Move MCP out of the beam manually and check using EPS
• Continue systematic test of the vacuum components (J. Eidam, V. Bondar)
• Test EPS switch on FEL imager after terminal exchange (A. Koch and W. Ehsan)
• Monitor stability and responsiveness of GUI Server and Log Servers (A. Silenzi)
• Set up monitoring of temperatures and pressures overnight (A. Silenzi)
• Test V0 and absorber

Tasks for tomorrow:

• Tunnels closed
• All loops should run stable
• No or only minimal interference with Karabo
• Continue tests from remote and check stability and robustness of systems

Other topics

• Tunnels open over the Easter weekend and also during the day on Tuesday 18 April
• There will be one more standup 09:30 meeting tomorrow (Wednesday)
• There will be a SASE Readiness Meeting at 09:00 on Thursday

Tunnel Status: XTD2/4/10 closed. XTD9 open. Controlled Access to 2/4/10 possible upon request

Vacuum (M. Dommach):

• Systematic tests of ion pumps in XTD9 concluded
• Tests of DPS in XTD9 should happen today with R. Villanueva
• Systematic tests of all interlock conditions in XTD9 should happen tomorrow

PBLMs (H. Sinn, P. Gessler):

• Debugging session for the ELMO controller was successful. The problem was traced to details in the parameter settings
• Proper setting of parameters is tricky and requires care and expert knowledge.
• Proper parameters now need to be programmed into the firmware. Manual setting is also possible but not favored.
• AE will set this up and deploy next week in order to maximise optimal use of resources (This week only one expert, TF, is here)

Imagers (A. Koch):

• Imagers in XTD2 and XTD9 are ready for beam
• It is important that images can be stored. This is not yet the case and will be required latest when PBS commissioning starts
• AK will be away from tomorrow until next Thursday

XGM (J. Grünert):

• Late yesterday, Karabo went into error triggering the interlock on the Vatterfly valves on both XGMs. Pressure is still low but slowly rising
• If the condition cannot be resolved remotely a local intervention is necessary. This can possibly wait until the scheduled interrupt tomorrow
• Today tests should occur with the DESY experts for gas injection
• Tests are currently hindered by the Karabo problem. AS suspects that this may have to do with the redeployment of Karabo yesterday. He is now working with high pressure on solving the problem

FXE (A. Galler):

• Currently commissioning loop 1. There is still a problem reading out encoders. It appears that this is a firmware issue. TF is working on this.

SPB (B Weinhausen):

• Fixes were deployed in the firmware for the Reinshaw encoder problem (16/32 bit issue). Still one encoder is in error. TF points out that this is a different problem. He will investigate

PLC (P. Gessler, T. Freyermuth):

• CRL in XTD2 was integrated into the PLC motion loop yesterday during the access.
• Meeting today with DESY experts to finalise the EPS logic
• Focus this week on SPB and AGIPD

Karabo (A. Silenzi):

• Karabo (Components Beckhoff-Device and Beckhoff-Com) were re-deployed yesterday.
• This maybe the origin of the problems observed in the evening 

Plans for today:

• XGM work on gas injection
• Bring back Karabo and clear XGM error codes
• Perform systematic test of DPS in XTD9

Other topics:

• No 09:30 meeting tomorrow due to access. 
• Request should be made to break the interlock in XTD10 on maintenance days in order to allow continuation of installation work

 General:

• DESY will try to lase again at higher energies during the next days
• XTD9 will searched an interlocked this afternoon at 1400 as PCT will try to transport photons as far as possible
• Company Wille has started to work on the radiation interlocks in SASE1. New TÜV date on 20 June and rehearsal on 14 June
• Instruments should be prepared that on these days

XTDs:

• CAS (A. Parenti) is still working on the instability problem of the SR imager. For the moment the workaround with reduced resolution (mono-8) seems to acceptable
• One (non-existing) pump needs to be removed from interlock conditions in XTD9. Otherwise interlock blocks one gate valve
• Filter was taken off the pop-in in XTD2
• Tests for PBLMs still need to be done in XTD9 when it re-opens next week

FXE:

• Problems exist with vacuum controls. J. Tolkiehn is currently working on it
• Firmware of commissioning crate was upgraded
• Test of Johan Spectrometer was done with test crate
• CRL test are next

SPB:

• Still waiting for firmware for loops 1&2. AE is almost ready and will deploy today
• All cables are ready for Optics hutch
• AE is currently pushing in the crates

PLC:

• DESY MPS claims that a cable from the interlock to an AE crate is still missing. 
• AE got vital information from MPS just a few days ago and is still waiting for info from the laser interlocks

KARABO:

• CAS is working on a version with bug fixes for stability. Will be deployed when appropriately test and when a break is available
• X2 timer interface still has issues. A. Sienzi is working on it

ITDM:

• An update of network switches is needed. This will be scheduled for next week. Details to be discussed during Monday 1500 BKR meeting

SPB

• Demo experiment done last week
• Flat fields taken with AGIPD over the weekend
• Focus was re-characterised. This required modifications to several beam parameters and then led to a controversial discussion with the RunCoor
• The sluggish behaviour of the GUI remains an issue and might pose a real risk if the injector freezes up and direct reflections into the detector occur
  • Kerstin and Gero looked at it last week
  • John is back from holidays and will also look
  • One observation is that the same information may update quickly on one server but not on another
• One motor is stuck in XTD9. A temporary access is requested
• On Sunday at beam hand over the CRL in XTD2 was not fully retracted. This was then done unannounced. Nothing bad happened this time but  this might cause real damage. 

FXE

• on shift and not represented at the meeting
• Harald mentions that 2 motors in the XTD2 CRL are stuck/in error. This should be fixed during this week's repair period
• Info by CB after the meeting: Scanned K_\beta and established machine energy scale within 40 eV

Tunnels

• Motion loop and SPB loop to be updated by AE tomorrow
• Tunnels and instruments are in different KARABO "Topics". K. Wrona states that this is a problem for DAQ
• SPB states that they have all information required already in their "Topic"
• It is unclear whether this might be an issue for FXE

CAS

• No specific issue

Just to clarify:

• Tunnel and SPB are in different Karabo Topics. It should not be a problem since we agreed that SPB does not need XTD2 slow data (from devices)
• Tunnel and FXE are in the same Karabo Topic
• We will keep prioritizing the configuration and deploy of Karabo-DAQ in SPB and FXE. Before going for Tunnel updates (we are mitigating risks of broadcast unknown issues just before the 1st User Experiments).  

FXE

• Still working on getting the laser on the sample
• Commissioning the recently received intensity monitor
• Work is also ongoing on the Gotthardt detector
• LPD is still in its own "Topic" but will be moved to the SASE1 "Topic" today

SPB

• Problem developed with one ELMO motor. It was moved to a test crate where it worked
• All loops were updated to allow slow data integration into DAQ
• AGIPD running improved significantly in stability after implementing the latest nightly build of KARABO
• ITDM team has various testing tools operational on the different channels of the detector
• Charge injection scan was possible for the first time without significant degradation of KARABO response

ITDM

• Slow data is now integrated for SPB
• Request is made for the list of data sources to integrate with FXE

AE

• PG points out that it was always foreseen that the timestamp/train ID integration would be done in software
• A HW solution was implemented during his absence to generate train IDs in a  PLC. He considers this to add significant risks.
• The HW solution will not be implemented in the tunnels to avoid unexpected behaviour during the user run
• Hence a work around solution needs to be found how to uniquely identify slow tunnel data with train IDs

Discussion about time stamps and train ID

• The summary of a longer discussion about the subject leads to the following conclusions:
• Slow tunnel data could be logged via a python script that also stores time stamps and train ids. An integration of this data with the fast data would have to be done offline post mortem
• Different opinions exist whether the association between time stamps and train ids in the KARABO data logger is unique and compatible with the train ids used in DAQ
• It is decided that this will be checked now and the result reported during the Friday S1R meeting

Announcement by ITDM

• The elog will be upgraded this afternoon
• The online cluster servers will be rebooted this afternoon

General

• Recomissioning of the FEL has been bumpy and up to this point only very weak lasing could be achieved. 
• So far SPB had only little beam and FXE had no beam at all. Both instruments state that the commissioning time for the up coming user run is insufficient

Instruments

• FXE:
  • First SW and DAQ stress test on Friday went well.
  • Some problems appeared with the slow data that should be fixed by now
  • K. Wrona states that these problem originated from missing permissions due to change of userid
• SPB 
  • Entire instruments needed re-alignment with beam since the CSS was moved during the shutdown
  • Very low intensity and not very stable beams were available on the Weekend. The beam could not be properly focussed
  • The bender for M2 cannot reach the flat condition. Is this the temperature problem again?
  • One pump was overheating
  • AGIPD tests with new bunch patterns started this morning
  • Significantly more time needed before user operation
  • There were some problems with the interlock system that trigger beam aborts

Tunnel

• Commissioning program was very limited in the last few days
• KMONO got some first interesting results
  • Rings and Kicks could be seen
  • Some wrong undulator settings were identified

Detectors

• Firmware update was done to record all 30 bunches. However this still has problems
• Need beam for calibrations
• The planned update by ITDM of the DAQ was not yet done

Advanced Electronics

• The mentioned interlock issues in SPB require an update to loop 7
• In loop 4 need to reset the "all valves open" signal by hand if it trips
• The SPB Master loop needs updating both for MPS and timing

Software

• Tests of the scantool took place with hardware. These tests continue today
• A new data model is being worked on for the scan tool but this is not available yet
• SPB states that at this point they will not make use of the scan tool as they first have to consolidate the instrument
• FXE is involved in the CAS tests

ITDM

• As mentioned before the DAQ version is still old and needs an update. This contains many improvements
• IBM discovered a bug in GPFS and published a patch. The bug could lead to data corruption
• Decision: DAQ update and GPFS patch will be installed today in the morning.

General

• There is currently no beam less than 24h before user operation is supposed to start in SPB
• The affects everybody (Instruments, Detector, Laser 

Instruments

• SPB:
  • Waiting for beam to come in a usable condition (500 uJ and pointing within aperture of optical elements)
• FXE
  • Similar

AE

• There was an interrupt of the vacuum loop in XTD2 due to a faulty pump controller. This was fixed by ZZ
• Rotational SmarACT stage and slits in the sample chamber for SPB are being worked on

CAS

• Due to change of beam schedule, now working on implementing more devices in DOOCS-KARABO for Jan Grünert

ITDM

• Will upgrade DAQ software today. Note: Still based on KARABO 2.1.16

General

• Intensity and availability of beam was good over the weekend
• Large drifts and jitter were observed that are significantly higher than previously
• Beam has two horizontal lobes as if it is too large for the mirrors

Instruments

• SPB:
  • Many tasks were achieved during the weekend, e. g. instrument alignment, detector calibration
  • Issue with encoders whose cables are close to motor cables persists. These encoders are being disabled
  • Rotational SmarACT stages still do not work and are needed for this week's user experiment
• FXE
  • At shift start on Saturday after reloading beam parameters there was no beam in the hutch.
  • Beam had to be brought back by hand. This was a large effort
  • This situation did not repeat on Sunday
  • LPD detector had problems: Only one super module correctly recorded data. Comment from S. Hauf: FXE should have called the detector expert
  • PP laser synchronisation does not work properly yet: Increments when adjusting timing are too coarse. Scan range is limited to 18ns

Detector

• Detector group was not informed about the LPD issues in FXE. 
• AGIPD calibrations were successfully taken on Sunday

ITDM

• Still work is needed for slow data DAQ
• DAQ behaves differently in SPB and FXE. In FXE it appears overloaded and in SPB it works fine
• Comment from CAS: 
  • Many changes were made in the KARABO framework to allow asynchronous operation
  • These changes have to be also pulled through to DAQ. This will only be done during the winter shutdown
  • Until then, DAQ is very vulnerable to any delay in the system

CAS

• No further comments

AE

• Currently working on problems with rotational stages in SPB

General:

• User operation with good beam conditions over the weekend
• One interlock trip in the FXE branch in XTD9. Restored by CAS
• Shutdown starts tomorrow at 08:00.
• Next dispatch meeting takes place Wednesday at 13:00. A reminder will be sent.

Instruments

• SPB:
  • Started with the actual sample during the night shift
  • Some DAQ issues
  • 108TB data was recorded
  • Problems appear to occur between 01:00 and 02:00  and between 13:00 and 14:00 
• FXE
  • Running is ok. 
  • Temporal and spatial overlap with optical laser established
  • Currently taking pump-probe data
  • People on shift were uncertain about how to recover from interlock trip
    • Statement from Vacuum: Pumps can restarted/valves open if the respective button in KARABO can be clicked

CAS:

• Delay between data taking and "online" analysis in the "Hummingbird" software is about 0.3s

ITDM

• On Friday, there was a global network outage affecting the Office Network on the DESY site.
• The transponder system in the experiment hall was down
• SRP organised a guard at the hall entrance so that the hall had not to be evacuated
• The control network (XDAC) was not affected

Radiation Interlock Tests are finished. Tunnels XTD9 and XTD10 up to the fence at the dump can be accessed right now. Magnet current interlock tests still takes a bit of time (ca. 1h). Once this is also done, I shall post a note 

Dear Colleagues,

in the future we would like to use this mechanism to record and exchange information that concerns the work done within the context of this task force. Please use this generously. We shall also put a link to the minutes after each weekly meeting.

regards,

Tobias 

Post on behalf of S. Brockhauser

 Since I am in Grenoble and visiting ESRF, I will not be able to give you the report in person. Instead Alessandro can give you a Summary.

In fact:
- we did collect the full list of the Karabo names for all the equipments involved
- we have started our sprint and we are on track
As next things:
- we are collecting all the required Karabo(!) default parameters
- we plan to finish the sprint as planned and presented last time.
Until now we have not experienced and obstacle.

Best wishes
Sandor

Note that plans for next week (points 2& 3) should refer to XTD9 and *NOT* XTD2

 Plans for next week:

• Consolidate vacuum in XTD2 so that V0 can be reliably opened when first beam should be transported
• Finish work in XTD9, in particular PBLMs and instrument components
• Finish systematic tests of ion pumps in XTD9
• Deploy fix for Renishaw encoders in firmware
•  

The current slow response times in the karabo GUI are the following (some measured times for reference):

- opening a project from the project manager : 45sec (just loading, not yet touching any device or scene)
- opening a scene and receiving the status for all devices e.g. XGM@XTD2 : between 90 and 120 sec
- beyond that, it's never clear if the displayed gauge values are true/actual and must be separately read.
- reading explicitly a gauge value (with command READ in configurator) gives currently an updated value after about 10 sec (should be almost immediate when requested)
-----> working with this (getting actual work done) is practically almost impossible

- this is using the "new" gui server "sa1-br-sys-con-gui2"
- this is after 19h30 (shouldn't less traffic help the performance ?)

Dear all,

we got the information from MKK, that they have to fix a leak in the main compressed air line. Therefore they will close the valves to the tunnels for approximately one hour starting today at 1 p.m.

Please do not use the compresed air system in the tunnels during that time with air driven tools or air guns, as the volume of the distribution systems acts as a buffer.

Regards,

Martin

This entry is on behalve of Frank Babies, if you have any doubt about it, contact him:

On 16.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from FXE
"fxe-rr-sys-utc-1"
"fxe-rr-sys-utc-2"
because of the Power Down issue on Monday.

On 17.Apr.2018 09:45am till Noon I have to shutdown the MicroTCA4 Crates Rackroom from SPB
"spb-rr-sys-utc-1"
"spb-rr-sys-utc-2"
because of the Power Down issue on Tuesday.

This will affect timing system and MPS communication.