Concurrent h5py

[woutdenolf]

Closes #3354

Tested with h5py 2.8, 2.9, 2.10, 3.1

Stress tests: https://gitlab.esrf.fr/denolf/concurrent_h5py

The main API:

• silx.io.h5py_utils.File: replace h5py.File mainly to handle file locking
• silx.io.h5py_utils.retry: retry the method whenever you get an HDF5 IO exception
• silx.io.h5py_utils.retry_in_subprocess: retry in a subprocess to capture segfaults
• silx.io.h5py_utils.retry_contextmanager: retry entering the context manager

The retry logic can also be used for other things than HDF5 reading:

• silx.utils.retry.retry
• silx.utils.retry.retry_in_subprocess
• silx.utils.retry.retry_contextmanager

Common use case of silx.io.h5py_utils for processing a Bliss Nexus file:

import silx.io.h5py_utils

@silx.io.h5py_utils.retry()
def process_scan(filename, name):
    """The method will be executed again if
    any HDF5 IO fails.
    """
    with silx.io.h5py_utils.File(filename) as h5file:
        scan = h5file[name]
        I0 = scan["measurement/I0"][()]
        It = scan["measurement/It"][()]
        return It/I0

scans = silx.io.h5py_utils.safe_top_level_names("myfile.h5")

for name in scans:
    process_scan("myfile.h5", name)

[t20100]

Nice to have the generic retry stuff apart from the HDF5 utility!

I hope that one day we won't need this any more, but in the (probably very long) meantime it makes it work.

A few minor "cosmetic" suggestions (up to you):

• silx/io/h5py_utils.py:
  • Make is_h5py_exception, retry_h5py_error, include_default private?
  • I would not put a open function since h5py does not provide one
• silx/utils/retry.py:
  • Make default_retry_on_error private?
  • Remove retry_ prefix from arguments retry_timeout, retry_period and retry_on_error since function are named retry*

[woutdenolf]

Thanks for the review!

Nice to have the generic retry stuff apart from the HDF5 utility!

I hope that one day we won't need this any more, but in the (probably very long) meantime it makes it work.

Exactly, we make it work like this and work on a better solution afterwards.

A few minor "cosmetic" suggestions (up to you):

* `silx/io/h5py_utils.py`:
  
  * Make `is_h5py_exception`, `retry_h5py_error`, `include_default` private?

Ok, except include_default as it may be useful for open_item as well (None is the default there). I will rename it to group_is_complete or something.

  * I would not put a `open` function since `h5py` does not provide one

Ok.

* `silx/utils/retry.py`:
  
  * Make `default_retry_on_error` private?

OK

  * Remove `retry_` prefix from arguments `retry_timeout`, `retry_period` and `retry_on_error` since function are named `retry*`

I added the prefix because you can override the timeout (and also the others) in each individual call while the argument names without the "retry_" prefix are rather generic and have higher probability of colliding with the method's named arguments. So for example:

@retry(retry_timeout=10)
def method(..., timeout=None):
     # "timeout" has nothing to do with retrying

# I want to override 10 by 1 while the method already has
# a timeout argument for some other purpose
method(retry_timeout=1, timeout=5)

[t20100]

I added the prefix because you can override the timeout (and also the others) in each individual call

OK. That's the kw.pops.

[payno]

pretty convenient to use (regarding the context :) and nice test suite)

I was wondering if it would be beneficial to have a redefinition of silx.io.utils.get_data(url, timeout) function in silx.io.h5py_utils.
This can be done in another PR by someone else.
But from my point of view it could be convenient to have
something simple like:

@retry()
def get_data(url):
    return silx_get_data(url)

or something more advance like:
get_data(url, retry_timeout, retry_invalid...) and call open_item if this is an hdf5 file otherwise call the original get_data. But maybe this is overkill.

[woutdenolf]

I was wondering if it would be beneficial to have a redefinition of silx.io.utils.get_data(url, timeout) function in silx.io.h5py_utils.

I'm not quite sure a method in silx.io.h5py_utils should be able to handle non-HDF5 things. It should be the other way around imo: silx.io.h5py_utils.get_data (HDF5-only with retry) which gets called by silx.io.utils.get_data but then it's probably wise to overwrite the default retry_timeout=None. Otherwise this call will block forever when the HDF5 file is corrupt.

Lets keep that for another PR.