Group Selection Page HELP DOC Controls Group Data Analysis Group ITDM WP76 DET Vacuum XRO EEE WP78 WP74 FXE SCS SPB MID HED SQS SXP Sample Environment Photon Commissioning Team Operation PSPO XO TS
Technical integration and coordination SASE1 Status and SASE3 Readiness
  Joint Operation and Readiness Logbook, Contact: T.Haas, A. Violante, G. Wellenreuther, Page 20 of 30  Not logged in ELOG logo
ID Date Author Location Severity Subjectdown
  518   25 Apr 2018, 18:10 Alessandro SilenziGeneralInformationRe: Status report after dispatch meeting 13:00

Regarding CAS Support for SASE2 Lasing:

CAS is happy to support in the frame of the On Call Duty system as it has been provided in the past.

Adriano Violante wrote:

SPB

  • Loops update: 1-4+8 this week. 5-7 Wed. next week. 11-13 end next week.
  • Smaract motors for PAM are needed and Nerea is on it.
  • It is not clear whether the AGIPD firmware has to be updated. To clarify with Jola when she is back. The info is important foe ITDM to plan the DAQ tests. Jola sick until 1st week of May. To be seen then.
  • During the last users run it happened that a valve in XTD9 was closed but the beam did not trip. It is not clear why. Will be clarified with AE. Interlock definitions  have to be checked next week when HSinn is back. 
  • During the switch on of the detector many Karabo devices had a problem even if no error messages were displayed. The problems disappeared restarting the devices. The errors could not be reproduced. Ot top of that it is possible to collect data only from 15 modules out of 16. The one missing changes every time the device is restarted. ASilenzi on it. 
  • Technofoft motor with magnetic switches do not work. In the tunnel the same configuration works, Will be investigated by AE/CAS/SPB
  • Motors upstream move sometimes by themselves. Discussion ongoing between AE SPB and CAS. To be adressed offline.

XPD

  • Access  of XTD1 today @14:00 to fix some EPS switches
  • SASE2 firsta lasing should be tried this week end + monday. CAS support requested. ASIlenzi checks with CAS whether this is possible.

AE

  • SCS exp: first loop running. 2 creates missing but loop is up.
  • MPS connection tunnels SASE2 done

ITDM

  • Online storage update patch to be installed today. The system should remain online but in some cases one could have some downtime (1-2 hrs)
  • MPS connection tunnels SASE2 done.

General

On Wednesday the meeting will be held at 13:00. Starting from next monday it will go back to 9.30

 

  478   21 Feb 2018, 13:49 Alessandro SilenziGeneralInformationRe: Status report after dispatch meeting 09.30

Small correction:

The coordinated motion is the KB-mirror's, not the Vacuum.

The postfix is 2.2.2.1 and includes features that were not included in 2.2.2

Karabo 2.2.2 is ready

 

Adriano Violante wrote:

SPB

  • KB chamber being pumped down
  • Smaract commissioning ongoing
  • Coordinate motion for VAC ready to be tested (probably tomorrow). It has to be agreed with Richard Bean
  • Clean-up slits Karabo middle layer deployed and working. - The same hase to be done for the power slits.
 FXE
  • DAQ test ongoing - fixing of bugs
  • After FXE has been moved into the new topic is not possible to read tunnel vacuum pressure anymore. Still the case.

 

  • XPD
  • XGM in SASE3 fixed (A turbo broke - replaced)
  • One exit slit Imager in XTD 10 commissioned

 

AE

  • PBLM moto problem was found to be due to a wrong Karabo-PLC linking. ==> FIXED
  • Soft mono commissioning started and AE support was required. DONE

ITDM

  •  Main problems with Train ID solved. Some problems still remain for some beckhoff loops in the tunnels==> AE on that.
  • Some way to document the change of data format for devices has and will be found by IDTM
  • Data source running for FXE. Changes have to be clearly communicated. 

CAS

  • Soft fix improving the GUI deployed. No 2.2.2 ready.
    
  • ​Issue with Karabo device returning the sensors data (e.g. cooling AGIPD) is being investigated. The Bug was found also in test environment but still random.
     

SASE3 - CY

  • GATT works
  • Inficon device to be improved (loop restart needed)
  • CAS will deliver a method to deal with the configuration changes asked by the experiments, redirecting the changes to the proper platform (Firmware (AE), Karabo...)

 

  496   29 Mar 2018, 09:48 Hugo Silva SantosGeneralInformationRe: Status report after dispatch meeting 09.30

Update regarding Digitizer:

----- Forwarded Message -----
From: "hamed sotoudi namin" <hamed.sotoudi.namin@xfel.eu>
To: "Sergey Esenov" <serguei.essenov@xfel.eu>, "andre" <andreas.galler@xfel.eu>
Cc: "Djelloul Boukhelef" <djelloul.boukhelef@xfel.eu>, "Dennis Goeries" <dennis.goeries@xfel.eu>, "Leonce Mekinda" <leonce.mekinda@xfel.eu>, "Patrick Gessler" <patrick.gessler@xfel.eu>, "Hugo Santos" <hugo.santos@xfel.eu>, "Krzysztof Wrona" <krzysztof.wrona@xfel.eu>, fxe-instrument@xfel.eu
Sent: Wednesday, 28 March, 2018 17:37:28
Subject: RE: adqDigitzer

Hi everybody,

Today morning me, Andreas and Leonce have tested digitizer with software in 
FXE hutch. We tried the same configuration as it was and checked the log 
file, we observed that software was missing synchronization, that means 
either packets from digitizer have some problems ( missed or corrupted) , or 
buffers are too small in software. We reinitialized everything and problem 
disappeared ( at least for synchronization).
This afternoon I tested digitizer in our lab in APD mode with Raw data mode 
and also checked values; everything was fine. Because till today we have 
tested digitizer in fully load and enabled APD and raw data for all channels 
we observed no issues. I tried to set the same configuration which Andreas 
applied in their system ( APD for one channel and raw data for all channels) 
I encountered the same problems that he had mentioned in Email. Conclusion 
is that Digitizer has no problem when Algorithms are enabled for all 
channels. But when one algorithm is enabled for only one channel there is 
problem in packets, or even there is no packet from digitizer sometimes. 
Which gives timeout error.
Current solution is that to enable required algorithm for all channels and 
ignore result from unwanted ones.
I do really apologize for this inconvenience caused and will do my best to 
solve the problem in Firmware as soon as possible.

Best,
Hamed

Adriano Violante wrote:

SPB

  • Sample delivery installed. Tonight will be commissioned. Experiment set-up starts tomorrow night

FXE

  • Digitizers were already implemented in the DAQ but there is an issue related to the Karabo device. CAS already looking into that.

AE

  • Follow up to the issue of re-setting pumps after interlock is triggered: loops updated. 
  • SASE2 MOV loop patched
  • Imagers in XTD1 have to be tested with Karabo

ITDM

  • On monday afternoon problems with the storage servers were reported. The issue is a defective board, whoch is meanwhile been deactivated. IBM is working on that. The system seems to be working in this state.

CAS

  • The next Karabo release will be able to handle better big loops. ASilenzi points out that the next update is really important since it adresses problems like the restart of time servers issue.

Readiness Meeting Issue list

  • Item 2 (XGM SASE3): After restarting of the CPP servers the XGM turbo pumps stopped, valves closed and it required a manual intervention to restore them. ==> Firmware updated. CLOSED

Operation Meeting Issuel List

  • Item 1 (Karabo rollout process): Testing steps of a new Karabo release have to be discussed with AE-ITDM-DET ==> Meeting took place. Closed

 

  534   13 Jun 2018, 11:07 Hugo Silva SantosGeneralInformationRe: Status report after dispatch meeting 09.30

Hi All,

Just adding more info:

CAS

  • Karabo version 2.2.4rc. being tested with PLC 1.27. Waiting for AE support to update the test environment to 1.28 (Jan, Nerea, and Bernard are aware).
Adriano Violante wrote:

SCS

  • Tests  continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

  • Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
  • Loop 3 - Motors test continue
  • Loop 2 is up and running.
  • Loop 4  probably will run tomorrow.
  • In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

AE

  • EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

  • Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

  • Karabo version 2.2.4. being tested with PLCs.

EET

  • Firmwares fo all the SCS loops have been generated.
  • Firmwares up to Loop 5 for SQS have been generated.

 

  564   17 Sep 2018, 14:30 Dennis GoeriesGeneralInformationRe: Status report after dispatch meeting 09.30

The GUI crashes reported by FXE will be fixed with the hotfix release coming ~tomorrow.

It is related to very high or low values in plotting widgets scratching the maximum floating point number (1e308 ~ infinity).

-- Wajid & Dennis

 

Adriano Violante wrote:

SPB

  • Bug in the AGIPD cooling macro (setpoint resets itself to -59 C)  -  Jola knows about the problem. AS: Macro will be fixed.

  • AGIPD dark calibration: when new darks were injected with calling Steffen's script "xfel-calibrate",  error message "Disk quota exceeded" popped up caused by a GPFS-related issue -> issue was solved. In general the script was not tested (scratch space was used!). One should not release script without ITDM knowing and haveing thested it before. 

FXE

  • Commissioning going on in the last days. Not able to get 14 keV beam for next users experiments. They will stick with 9.3 keV.
  • Karabo stability issue (GUI) reported in the commissionig logbook.

SQS

  • Problem with the digitizers bandwidth has now been solved.
  • Loop 6 running w/o smaract stages.
  • Flow switches issues solved . Rewiring of PLC needed
  • Pfeiifer flow valves ready to be stested this week.
  • Loop assignment changed. Change request handed in. As soon as it is approved it will be clear what loop to bring up next.

​SCS

  • Loop 7 is running. Needs to be connected to Karabo.
  • Loop 8 running too.
  • Readiness of loop 1 and 4 ==> Interlock conditions need to be implemented. 
  • It is unclear whether the homing capability has been implemented already and can be used. Discussion ongoing. The use case has to be clarified.
  • Network storage for the control PCs in the control room is requested. Scope definition and discussions with CAS and ITDM will be done offline.

MID

  • Loop 1 deployed and technical commissioning of the OPT hutch is ongoing. This week Slit and ATT will be commissioned. No big issues so far.
  • Loop 4 being prepared by AE. Will be ready next week.

HED

  • Work in tunnel ongoing
  • HRMono- ready to start grouting. Mono will be here next week.

DET

  • FXE/SPB GPUs crash issue  - to be checked with ITDM - ITDM was not contacted.

ITDM

  • DAQ was upgraded to the user friendly version in SASE3. After having discussed with FXE in the meeting, it was decided to install it also in FXE. 
  • DAQ training requested by SQS

CAS

  • Karabo 2.2.5. patch being prepared. Will solve minor bugs and the crash due to empty data files.
  • 28 Mpot update requests were received by CY. Need to be planned and coordinated with H.Santos.

 

 

  442   10 Jan 2018, 14:53 Hugo Silva SantosGeneralInformationRe: Status and plans after Readiness meeting

Dear Monica,

Just one small correction at "Karabo 2.2.0 being deploy today for FXE." The version is, actually, Karabo 2.2.1 that we released yesterday.

Hugo

Monica Aresté wrote:

PSPO:

- G.Wellenreuther inform that MKK is planning to test the Umschaltautomatik, this will affect XHE1, XHE3, XHPSC, XHEXP, XHVAC. Oleg Engelmann will send a mail with detailed information.

 

SPB:

- AGIPD is fully occupied again and restarted this morning as scheduled. Everything works well so far and it is expected to finish the test on time. Further test with new Microcontroller are scheduled for next week.

- Most of the loops that required modifications are updated or on going.

- Module BKF00128 will be modified this week, update of Loop 5 will folllow.

- Loop 12 is ready to be deployed, to confirm if modifications in the modules are finished and reorganization on rack UH2.12 is ready.

- It is necessary to correct an error in module BKF00220,  AE will agree with SPB  when this will be possible.

 

FXE:

- New Vacuum device in operation.

- Karabo 2.2.0 being deploy today for FXE.

 

AE:

- Replace one Micro-TC in SASE3, a cooling unit is failing and one module will be replaced by a new one today. This is providing timing for XGM and K-mono. AE will inform by the logbook before to start the replacement.

 

 

 

  555   28 Aug 2018, 11:36 Alessandro SilenziXTD10InformationRe: SASE3 loops will restarted at 11:30

 

Alessandro Silenzi wrote:

This announcement is to notify you that the intervention planned to configure the interlocks as read-only will be carried out at 11:30.

A following message will notify of the end of the intervention

Alessandro

Intervention completed.

SA3_SYS_EPS2MPS/DCTRL/BEAM_GATT_OK has now read-only interlock properties

  544   10 Jul 2018, 11:23 Hugo Silva SantosGeneralInformationRe: Karabo 2.2.4 released (mainly for ITDM, AE and DET) + Deployment dates

Dear All,

Please, see below the updates regarding Karabo 2.2.4.1 deployment:

  • SA3: 05.07, Thursday (references: Chris, Wajid, and Valerii) - Done
  • SA2: 06.07, Friday (references: Chris, Wajid, and Valerii) - Done
  • SA1 + FXE: 09.07, Monday (references: Wajidi) - Done
  • SPB: 10.07, Tuesday (references: Gabriele) - Done
  • BKR: 10.07, Tuesday (references: Wajid, Jan, Harald) ToDo
  • LAS, SQS and SCS dates have to be agreed with the respective contact persons. (references: Cyril, Riccardo, and David) In Progress
  • ITDM: We are working to understand and solve a problem reported by Nasser in Karabo 2.2.4 Configurator (references: Dennis, Cyril, and Nasser) In Progress
Hugo Silva Santos wrote:

Dear All,

I would like to inform that we released last Friday (22.06) the Karabo 2.2.4 version. We got around five days delayed to prioritize issues regarding Karabo-DAQ integration and MDL devices that were found in the last steps of the test cycle. However, fortunately, those issues are solved. 

Based on this run, the new dates that we (CAS) are proposing to start the deployment in operation will be:

  • SA3: 28.06, Thursday (CAS references: Chris, Wajid, and Valerii)
  • SA2: 02.07, Monday (CAS references: Chris, Wajid, and Valerii)
  • SA1 + SPB + FXE: 03.07, Tuesday (CAS references: Alessandro, Dennis, Gabriele, Wajid)
    • SPB deployment should not impact AGIPD firmware test week, as agreed before.
  • BKR: 28.06? (Wajid and Jan, could you please agree offline?)
  • LAS, SQS and SCS dates have to be agreed with the respective contact persons.

-
Hugo

 

 

  354   04 Aug 2017, 13:41 Eric BoydXHEXPInformationRadiation Measurements of FXE Monday 07.08

The control room thinks that they will be able to very easily provide us the desired parameters for commissioning  the FXE EXP Hutch this coming Monday. If that is the case we will be restricting access in SASE 1 in accordance to the attached presentation from 8:00 am until completion of the radiation measurements. A SASE 1 log book entry will be made when restrictions go into effect and are lifted.

We should find out Sunday evening from the control room if they don't think they can provide us with the proper parameters. If this happens I will send out an email informing you that the measurements have been canceled.  If you don't receive an email from me Sunday evening, assume that the measurements will be taking place beginning at 8:00 on Monday morning and will be in effect for most of the day.

Sorry for the short notice.

Attachment 1: Restricted_areas_for_FXE_EXP_measurments.pptx
  357   07 Aug 2017, 18:30 Eric BoydXHEXPInformationRadiaiton Measurements Complete for SASE 1

Radiaiton Measurements are complete for SASE 1 Normal access is restored.

 

  431   07 Dec 2017, 09:04 Tobias HaasGeneralInformationRack extinguishing system in SASE2

Please note following info from DESY:

 

Author: F. Saretzki



Ab sofort werden die Rack-Löschanlagen in den südlichen Tunneln in Betrieb genommen. Die Anlagen bleiben danach scharf.



 

  452   17 Jan 2018, 15:55 Bruno FernandesXTD2InformationRack 1.2 Timing Update

Dear all, 

following up from the message #449, the MicroTCA Crate in rack 1.2 has been rebooted. Timing system hardware is now running firmware v1.24.0.

  100   04 Apr 2017, 18:16 Tobias HaasGeneralInformationProgress on Vacuum Loop

J. Eidam, V. Bondor, T. Freyermuth checked Ion pump controller settings in XTD2 and XTD9.

Many controllers were found to have incorrect settings. All pumps in XTD2 now respond and give pressure readings. There remains the problem that the PLC is looking for the wrong channel in the case where 2 pumps are connected to one controller (Is 1 & 3 and PLC looks for 1 & 2). J. Eidam now checks channel numbers on all controller. T. Freyermuth then regenerates the firmware. A. Silenzi set up pressure recording for pumps P10050I, P10060I and P10200I to run overnight. 

  41   10 Feb 2017, 08:31 Tobias HaasGeneralInformationProgress Beam Transport

Post on behalf of Harald Sinn

Fortschritte, die ich letzte Woche beobachten konnte, sind: 

1.) Die Spiegelkammern wurden weiter lokal getestet. Benoit hat gestern abend den 
letzten Encoder am XTD9 vercrimmt. Wahrscheinlich muss der dann noch mal getestet werden, 
aber dann sollte das abgeschlossen sein. 
2.) Gestern haben wir zusammen mit Jan Tolkiehn den SRA und Filterkammer getetstet. 
SRA ist OK (incl. der Encoder), die Filterkammer auch, aber Wolfgang meinte, er müsste die 
LImit.switches noch mal etwas nachstellen (dafür sollte dann aber Karabo funktionieren, s.u.)
3.) Die PBLMs in XTD2 und die letzten beiden in XTD9 sich verkabelt und könnten von uns aus lokal 
getestet werden. Jan meinte allerdings, dass an den Elmo/technosoft (PBLM hat Elmo) Treibern noch etwas Arbeit
nötig ist, um zuverlässig Tests zu machen (MCI Library)
4.) Andreas Block hat mit Maik Neumann ausgemacht, dass sie sich Montag treffen und alle fehlenden Wasserverbindungen in 
SASE1 durchgehen und am besten sofort machen,. 
5.) Makus Schlösser hat bestätigt, dass nächste Woche (wohl leider nur) ein Team für uns arbeiten kann. 
Das wird zunächst aber in SASE3 hinten anfangen, weil wir da schnon einen Arbeitsstau in der Vakuumgruppe 
aufgebaut haben. Die relativ wenigen Komponenten in SASE1 werden natürlich auch fertig, aber Markus meinte, er würde vorhar 
lieber noch einige Daten konsolidieren. 
 
Zum lokalen testen: Das ist schon sehr wichtig, allerdings sollte es noch nicht mit dem technical commissioing 
verwechselt werden. Beim lokalen Testen wird nur festgestellt, ob sich der Motor in die richtige Richtung dreht 
und ob am Terminal die richtige LED leuchtet, wenn man den entsprechenden Endschalter drückt 
(Jan weiss dann, welche LED leuchten muss). Man sollte aber z.B. nicht auf den Endschalter fahren, weil in diesem 
Modus kein automatischer Stopp stattfindet, sondern man muss manuell eine Stopp Taste drücken. 
Auch werden nicht Motorgeschwindigkeit etc. optimiert oder gespeichert. 
Deswegen ist es wichtig, dass wir möglichst schnell Beckhoff+Karabo zum Laufen bringen, weil diese Tests 
essentiell sind und von den Komponentenverantwortlichen gemacht werden müssen. 
Falls die MCI Library tatsächlich ein Problem darstellt, was etwas dauert, würde ich vorschagen, die 
Beckhoff Motion Loop jetzt schon in Betrieb zu nehmen, mit den Motoren, die wir getestet haben. 
 
Grüße 
 
Harald 
 
 
  60   06 Mar 2017, 18:14 Andreas KochXTD2InformationPop-In II-45 Karabo test

6.3.2017, Karabo tests in XTD2 with Wajid, Andreas

1. Karabo all devices, all scenes working, with restrictions.

2. Power supply, motor, LED working.

3. Scenes tested: Motor scene and camera scene need additional parameters. AK will edit a list of suggestions, improvements and bugs.

4. Bugs, observations, details in improvement list.:

- Homing not working, tried also old parameter set from prototype crate.

- Encoder position not correct, dragging behind.

- Parameter for braking switch missing “ALowSpeed”.

  85   22 Mar 2017, 16:39 Andreas KochXTD2InformationPop-In II-45 Karabo test
Trigger signal now available at imager. First Karabo tests of camera device with trigger successful. Detailed
tests continue.
  104   05 Apr 2017, 14:14 Andreas KochXTD9FeedbackPlease inform on rack status 1.18

The rack 1.18 in XTD9 is not operational at the moment because Nicola is working on it. Nerea, Tobias or others let me know when it is fully operational again. This avoids checking it in the tunnel. Alternatively, let me know if I can check remotely.

  56   02 Mar 2017, 09:52 NEREA JARDONGeneralInformationPLC work

Dear all,

Right now the Motion PLC is up and running. We are still working in the Vacuum one, today by midday should be also working.

  194   02 May 2017, 10:05 Tobias FreyermuthXTD9InformationPBLM Motor drivers in PreOP

Hi all,

 

due to ongoing tests with the PBLM motor drivers I have to set the Elmo drivers in PreOp mode.

That should not affect any other terminal in the loop.

 

Cheers,

    Tobias

  396   04 Oct 2017, 10:31 Janusz SzubaGeneralInformationOnline storage update on Monday 9th Oct

We would like to take the oportunity of the shutdown period and update the online storage next week on Monday 9th Oct. This will take approx. half a day. During that time the online storage will not be available on the online computing cluster as well as on pclayer servers used for DAQ. In the same way, the nfs export for data logger is also affected.

Regarding data collected on the online cluster, although it should not have any influence, but as a precautionary measures, please make sure that all relevant data from all past proposals  are migrated to offline storage, which in practice means requesting migration via metadata catalog web interface. If there are still issues with particular runs not being able to be migrated, please contact ITDM.

regards,

Janusz 

ELOG V3.1.4-7c3fd00