2. Inspecting and processing data during the experiment

2.1. Data display during experiment

Karabo provides a customised GUI scene that displays relevant data during an experiment. E.g. the last captured corrected-Xray image from the detector, and some crucial other parameters such as train and pulse ID, beam energy, detector position, and for pump-probe experiments the pump-probe laser state, pump-probe laser delay, etc.

Specific scenes can be developed for each experiment’s needs. An example is shown in Figure Fig. 2.1.

_images/lpd2tile_gui.png

Fig. 2.1 Example of scene in Karabo for LPD detector.

2.2. Integrating data analysis pipelines

Given enough prior notice, an on-the-fly pipeline can be created in collaboration with the users and made available to European XFEL experiments. Hence, calculation (like integration, correlations, dynamic ROI selection, peak detection, etc.) can be made available on-the-fly.

2.3. Data stream to user tools

Karabo bridge is a proxy interface to stream data to tools that are not integrated into the Karabo framework. It can be configured to provide any detector or control data. This interface is primarily for online data analysis (near real-time), but the karabo_data Python package can also stream data from files using the same protocol.

We provide Karabo Bridge Clients to receive this data in Python and C++, but you can also write your own code to receive the data if necessary.

2.3.1. Simulation

In order to test tools with Karabo bridge, we provide a simulated server in the karabo_bridge Python package. This sends nonsense data with the same structure as real data. To start a server, run the command:

karabo-bridge-server-sim 4545

The number (4545) must be an unused local TCP port above 1024.

2.4. How to use it at the EuXFEL

User tools running on the Online cluster can use the Karabo Bridge API to connect to karabo data pipelines and access data during an experiment.

Karabo bridge devices (from which karabo bridge clients can connect to and receive data) are running at all instruments.

Note

Data from MHz detectors at the EuXFEL (AGIPD, LPD, DSSC) might have a different data shape in file and in data streams. For each train data the shape in file follows: (pulse index, slow scan axis, fast scan axis). For performance constraints, the default shape offered for this data via data stream is: (module number, fast scan, slow scan, pulse index). This configuration can be changed on demand during beamtime and be switched between 2 shapes:

  • online: (module number, fast scan, slow scan, pulse index)
  • file-like: (pulse index, module number, slow scan, fast scan)

If you wish to use one or the other, communicate it to your contact person at the instrument.

Having file-like data shape online will add delay to the data stream, you might want to avoid this option if faster feedback is important for your experiment.

2.4.1. SPB instrument

2 bridge devices are permanently running and data can be requested from them. they are respectively sending raw and calibrated data of the AGIPD detector. The following table gives connection details:

data source host port
AGIPD RAW DATA 10.253.0.51 45000
AGIPD CORRECTED DATA 10.253.0.51
45010
45011
45012

2.4.2. FXE instrument

2 bridge devices are permanently running and data can be requested from them. they are respectively sending raw and calibrated data of the LPD detector. The following table gives connection details:

data source host port
LPD RAW DATA 10.253.0.53 4500
LPD CORRECTED DATA 10.253.0.53 4501

2.4.3. MID instrument

1 bridge device is permanently running and data can be requested from it. It sends calibrated data of the AGIPD detector. The following table gives connection details:

data source host port
AGIPD CORRECTED DATA 10.253.0.171
45051

Note

Any data pipeline can be temporarily set-up on request