ELOG SASE1 Status and SASE3 Readiness

Today late in the evening (~22:00), the exflgateway will be migrated to a new hardware. All users will be logged out before migration and home folders will be copied.

After migration is finished, you may experience ssh warning about ssh host key. For all centrally managed Linux this should be fixed the following morning. For all other, like Windows, or yellow desktops, you need to remove the old key and/or accept a changed one.

Best

Janusz

Re: exflgateway migration

The migration has finished, to avoid temporary problem with ssh you can use

ssh exflgateway -o StrictHostKeyChecking=no

until it will be resolved in the morning ( the usual update of all hosts ssh keys).

Home folder has been changed to /data/<user_name> and it is slightly bigger then before...

Let me know if there is still anything missing.

best, Janusz

ITDM Admin wrote:

Today late in the evening (~22:00), the exflgateway will be migrated to a new hardware. All users will be logged out before migration and home folders will be copied.

Best

Janusz

CAS - On-Call Duty Planning - Valerii, Gabriele and Wajid

Dear All,

I would like to share the CAS members on the OCD from 29/04 to 01/05:

Saturday 08:00 to Sunday 08:00 - Valerii - Phone: 95471
Sunday 08:00 to Monday 08:00 - Gabriele - Phone: 96894
Monday 08:00 to Tuesday 08:00 - Wajid - Phone: 95630

Best,

Hugo

ERRATA: CAS - On-Call Duty Planning - Valerii, Gabriele and Wajid

Dear All,

Still in time, I would like to correct out schedule.

The scheduled previous agreed for this weekend/holiday was from 10:00 to 17:00. Therefore,

Saturday 10:00 to 17:00 - Valerii - Phone: 95471
Sunday 10:00 to 17:00 - Gabriele - Phone: 96894
Monday 10:00 to 17:00 - Wajid - Phone: 95630

Best,

Hugo

CAS On-Call Duty Planning (Until 11/06)

Dear All,

I would like to share the CAS OCD Planning until 11/06: https://goo.gl/LL6vO1

In this planning, we show the numbers that can be called in case of incidents. Unfortunately, we are still trying to have a single number for all phones.

PS: I am sorry for the google.docs document, but once we are keeping it in a collaborative way in CAS, share the link with all of you will guarantee that you can see the updated version.

Best,

Hugo

Karabo down in tunnel

Dear All,

We have a issue on the control servers and we are already working on it...

Best,

Hugo

Karabo down in tunnel

Dear All,

We solved the incident and the services are running normally. We are still try to investigate, as far as we know we have some "zumbies" process running in the GUI servers and it was not consuming the HEATBEATs.

Hugo Silva Santos wrote:

Dear All,

We have a issue on the control servers and we are already working on it...

Best,

Hugo

Karabo 2.1.15 in Test

Dear All,

I would like to report that we started today to test (internally in CAS) the Karabo 2.1.15 version.

We got delayed to prioritise and solve a showstopper issue related to GUI Client. If our tests succeed, we are foreseeing to finish the tests and release tomorrow.

Best,

Hugo

Re: Status and plans after Readiness meeting

Dear Monica,

Just one small correction at "Karabo 2.2.0 being deploy today for FXE." The version is, actually, Karabo 2.2.1 that we released yesterday.

Hugo

Monica Aresté wrote:

PSPO:

- G.Wellenreuther inform that MKK is planning to test the Umschaltautomatik, this will affect XHE1, XHE3, XHPSC, XHEXP, XHVAC. Oleg Engelmann will send a mail with detailed information.

SPB:

- AGIPD is fully occupied again and restarted this morning as scheduled. Everything works well so far and it is expected to finish the test on time. Further test with new Microcontroller are scheduled for next week.

- Most of the loops that required modifications are updated or on going.

- Module BKF00128 will be modified this week, update of Loop 5 will folllow.

- Loop 12 is ready to be deployed, to confirm if modifications in the modules are finished and reorganization on rack UH2.12 is ready.

- It is necessary to correct an error in module BKF00220, AE will agree with SPB when this will be possible.

FXE:

- New Vacuum device in operation.

- Karabo 2.2.0 being deploy today for FXE.

AE:

- Replace one Micro-TC in SASE3, a cooling unit is failing and one module will be replaced by a new one today. This is providing timing for XGM and K-mono. AE will inform by the logbook before to start the replacement.

Re: Status report after dispatch meeting 09.30

Update regarding Digitizer:

----- Forwarded Message -----
From: "hamed sotoudi namin" <hamed.sotoudi.namin@xfel.eu>
To: "Sergey Esenov" <serguei.essenov@xfel.eu>, "andre" <andreas.galler@xfel.eu>
Cc: "Djelloul Boukhelef" <djelloul.boukhelef@xfel.eu>, "Dennis Goeries" <dennis.goeries@xfel.eu>, "Leonce Mekinda" <leonce.mekinda@xfel.eu>, "Patrick Gessler" <patrick.gessler@xfel.eu>, "Hugo Santos" <hugo.santos@xfel.eu>, "Krzysztof Wrona" <krzysztof.wrona@xfel.eu>, fxe-instrument@xfel.eu
Sent: Wednesday, 28 March, 2018 17:37:28
Subject: RE: adqDigitzer

Hi everybody,

Today morning me, Andreas and Leonce have tested digitizer with software in
FXE hutch. We tried the same configuration as it was and checked the log
file, we observed that software was missing synchronization, that means
either packets from digitizer have some problems ( missed or corrupted) , or
buffers are too small in software. We reinitialized everything and problem
disappeared ( at least for synchronization).
This afternoon I tested digitizer in our lab in APD mode with Raw data mode
and also checked values; everything was fine. Because till today we have
tested digitizer in fully load and enabled APD and raw data for all channels
we observed no issues. I tried to set the same configuration which Andreas
applied in their system ( APD for one channel and raw data for all channels)
I encountered the same problems that he had mentioned in Email. Conclusion
is that Digitizer has no problem when Algorithms are enabled for all
channels. But when one algorithm is enabled for only one channel there is
problem in packets, or even there is no packet from digitizer sometimes.
Which gives timeout error.
Current solution is that to enable required algorithm for all channels and
ignore result from unwanted ones.
I do really apologize for this inconvenience caused and will do my best to
solve the problem in Firmware as soon as possible.

Best,
Hamed

Adriano Violante wrote:

SPB

Sample delivery installed. Tonight will be commissioned. Experiment set-up starts tomorrow night

FXE

Digitizers were already implemented in the DAQ but there is an issue related to the Karabo device. CAS already looking into that.

Follow up to the issue of re-setting pumps after interlock is triggered: loops updated.
SASE2 MOV loop patched
Imagers in XTD1 have to be tested with Karabo

ITDM

On monday afternoon problems with the storage servers were reported. The issue is a defective board, whoch is meanwhile been deactivated. IBM is working on that. The system seems to be working in this state.

CAS

The next Karabo release will be able to handle better big loops. ASilenzi points out that the next update is really important since it adresses problems like the restart of time servers issue.

Readiness Meeting Issue list

Item 2 (XGM SASE3): After restarting of the CPP servers the XGM turbo pumps stopped, valves closed and it required a manual intervention to restore them. ==> Firmware updated. CLOSED

Operation Meeting Issuel List

Item 1 (Karabo rollout process): Testing steps of a new Karabo release have to be discussed with AE-ITDM-DET ==> Meeting took place. Closed

Re: Status report after dispatch meeting 09.30

Hi All,

Just adding more info:

CAS

Karabo version 2.2.4rc. being tested with PLC 1.27. Waiting for AE support to update the test environment to 1.28 (Jan, Nerea, and Bernard are aware).

Adriano Violante wrote:

SCS

Tests continue - List of issues sent to AE.(4/5 in total). Some of thema are related to cabling. EET will be notified.

SQS

Loop 1 - 2 issues with cables found. EET to act on that. Motors test continue
Loop 3 - Motors test continue
Loop 2 is up and running.
Loop 4 probably will run tomorrow.
In general, (motors, gauges...) parameters have to be saved in order to be restored when a loop is restarded. Info have to go from SQS to AE about which parameters have to be saved.

EPS in SASE2 running. MOV2 will be up until tomorrow.

ITDM

Transceiver changed ion SCS yesterday, for SQS will be done today. No rpoblems with the network have been noticed/reported by SCS.

CAS

Karabo version 2.2.4. being tested with PLCs.

EET

Firmwares fo all the SCS loops have been generated.
Firmwares up to Loop 5 for SQS have been generated.

Karabo 2.2.4 released (mainly for ITDM, AE and DET) + Deployment dates

Dear All,

I would like to inform that we released last Friday (22.06) the Karabo 2.2.4 version. We got around five days delayed to prioritize issues regarding Karabo-DAQ integration and MDL devices that were found in the last steps of the test cycle. However, fortunately, those issues are solved.

Based on this run, the new dates that we (CAS) are proposing to start the deployment in operation will be:

SA3: 28.06, Thursday (CAS references: Chris, Wajid, and Valerii)
SA2: 02.07, Monday (CAS references: Chris, Wajid, and Valerii)
SA1 + SPB + FXE: 03.07, Tuesday (CAS references: Alessandro, Dennis, Gabriele, Wajid)
- SPB deployment should not impact AGIPD firmware test week, as agreed before.
BKR: 28.06? (Wajid and Jan, could you please agree offline?)
LAS, SQS and SCS dates have to be agreed with the respective contact persons.

-
Hugo

Re: Karabo 2.2.4 released (mainly for ITDM, AE and DET) + Deployment dates

Dear All,

Please, see below the updates regarding Karabo 2.2.4.1 deployment:

~~SA3: 05.07, Thursday (references: Chris, Wajid, and Valerii) - Done~~
~~SA2: 06.07, Friday (references: Chris, Wajid, and Valerii) - Done~~
~~SA1 + FXE: 09.07, Monday (references: Wajidi) - Done~~
~~SPB: 10.07, Tuesday (references: Gabriele) - Done~~
BKR: 10.07, Tuesday (references: Wajid, Jan, Harald) ToDo
LAS, SQS and SCS dates have to be agreed with the respective contact persons. (references: Cyril, Riccardo, and David) In Progress
ITDM: We are working to understand and solve a problem reported by Nasser in Karabo 2.2.4 Configurator (references: Dennis, Cyril, and Nasser) In Progress

Hugo Silva Santos wrote:

Dear All,

Based on this run, the new dates that we (CAS) are proposing to start the deployment in operation will be:

SA3: 28.06, Thursday (CAS references: Chris, Wajid, and Valerii)
SA2: 02.07, Monday (CAS references: Chris, Wajid, and Valerii)
SA1 + SPB + FXE: 03.07, Tuesday (CAS references: Alessandro, Dennis, Gabriele, Wajid)
- SPB deployment should not impact AGIPD firmware test week, as agreed before.
BKR: 28.06? (Wajid and Jan, could you please agree offline?)
LAS, SQS and SCS dates have to be agreed with the respective contact persons.

-
Hugo

Status of Vacuum System

XTD2: All gate valves are now OK (not interlocked) after re-instantiating and re-setting the XGM_DP_OK.

The interlock errors of some valves in XTD9 are shown in the screen shots below (Janni and Valerii starting now in XTD9 to debug some more controllers)

XTD10 not open on Monday

Dear all,

contrary to the rumor that I spread last Tuesday in the PBS meeting, the magnet tests in XTD2/4 will be NOT done coming Monday and therefore the XTD10 will be be NOT open. It is still the plan to have one access day in XTD10 per week in the future (Monday or Wednesday to be clarified with the magnet group), but this will start only later this month.

Best regards

Harald

Restart vacuum loop now

We are in the BRK and corrected the vacuum interlock condition.

We will restart the SA1 vacuum loop in a few minutes

230

12 May 2017, 13:59