Problem when trying to stream data

[FilipeMaia]

Hi,

I was trying to stream data using the following command:
karabo-bridge-serve-files /gpfs/exfel/exp/SPB/201901/p002316/raw/r0174 4545

And I got the following error, which I don't quite know where it's coming from:

Streamer started on: tcp://max-display002.desy.de:4545
Traceback (most recent call last):
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/bin//karabo-bridge-serve-files", line 8, in <module>
    sys.exit(main())
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/extra_data/export.py", line 190, in main
    use_infiniband=args.use_infiniband, sock=args.socket_type
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/extra_data/export.py", line 141, in serve_files
    for tid, data in _iter_trains(data, merge_detector=append_detector_modules):
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/extra_data/export.py", line 66, in _iter_trains
    for tid, train_data in data.trains():
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/extra_data/reader.py", line 1397, in __iter__
    yield tid, self._assemble_data(tid)
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/extra_data/reader.py", line 1372, in _assemble_data
    self._set_result(res, source, key, ds[pos])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 780, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/gpfs/exfel/sw/software/xfel_anaconda3/1.1/lib/python3.7/site-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
  File "h5py/_selector.pyx", line 272, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 151, in h5py._selector.Selector.apply_args
IndexError: Index (0) out of range for empty dimension

[takluyver]

Hi Filipe,

It looks like there's a problem with the DA02 files in that run - all the datasets under CONTROL are empty, but the indexes still say there should be data. We have a tool extra-data-validate to pick up problems like this - but at present, it doesn't pick up this one, because we only check the dataset size for INSTRUMENT data. I'll fix the validation tool so it can pick this up.

To avoid this, you can select the sources you want to stream, e.g. --source '*/DET/*' for AGIPD detector modules. Selecting a subset of data is usually a good idea anyway, because the default is to stream all keys of all sources recorded, which can be very slow.

Best wishes,
Thomas

[FilipeMaia]

I see, thanks!