H5md feature unavailable

[dpo854]

How does one go about fixing the unavailability of H5md? My espressomd.io.writer seems to have the vtf feature available in it. So, I am not sure where the error is coming from. A look at the source code for espressomd.io.writer.h5md suggested that 'H5md' is not part of the features of code_info. Any way out of this?

h5_file=h5md.H5md(filename="systeminfo.h5", write_pos=True, write_vel=False, write_force=False, write_species=True, write_mass=False, write_charge=True, write_ordered=True)

File "/home/dpo854/mycode/espresso/install/lib/python3.4/site-packages/espressomd/io/writer/h5md.py", line 29, in init
raise RuntimeError("H5md not available.")
RuntimeError: H5md not available.

[dpo854]

On further reading about H5md module, I found out that it's a feature that is external to Espressomd and must be invoked using "Note Requires H5MD external feature, enabled with -DWITH_HDF5=ON.". But, putting this option before the file name in the command "$ESPRESSO_PATH/pypresso -DWITH_HDF5=ON minimal.py 0.0002694 -0.2179 50.0" did not lead to any good.

[dpo854]

I figured that it's not compiled since it threw up the error message "H5MD not compiled in. Does it mean I will have to install EspressoMD again?
"

[KaiSzuttor]

please have a look at http://espressomd.org/html/doc4.1.2/installation.html. You have to enable certain features (like H5MD) at compile time.

[dpo854]

Hi Kai, does it mean I will have to install H5MD first before compiling espressomd with "H5MD" ON during ccmake?

[KaiSzuttor]

yes

[dpo854]

Ok. What version do you recommend? By the way, it's h5py I guess (http://docs.h5py.org/en/stable/).

[dpo854]

I recompiled Espressomd with " -DWITH_HDF5=ON" but the problem remains.

[dpo854]

When I put out all features using print(code_info.features()), I get the following:
features ['BOND_CONSTRAINT', 'COLLISION_DETECTION', 'CUDA', 'DIPOLAR_BARNES_HUT', 'DIPOLAR_DIRECT_SUM', 'DIPOLES', 'DP3M', 'DPD', 'EK_BOUNDARIES', 'ELECTROKINETICS', 'ELECTROSTATICS', 'ENGINE', 'EXCLUSIONS', 'EXTERNAL_FORCES', 'FFTW', 'GAUSSIAN', 'GSL', 'HAT', 'HERTZIAN', 'LANGEVIN_PER_PARTICLE', 'LB_BOUNDARIES', 'LB_BOUNDARIES_GPU', 'LENNARD_JONES', 'LENNARD_JONES_GENERIC', 'LJCOS', 'LJCOS2', 'LJGEN_SOFTCORE', 'MASS', 'MMM1D_GPU', 'NPT', 'P3M', 'PARTICLE_ANISOTROPY', 'ROTATION', 'ROTATIONAL_INERTIA', 'SMOOTH_STEP', 'SOFT_SPHERE', 'SWIMMER_REACTIONS', 'TABULATED', 'VIRTUAL_SITES', 'VIRTUAL_SITES_INERTIALESS_TRACERS', 'VIRTUAL_SITES_RELATIVE', 'WCA']
Clearly, H5MD is missing even with the option -DWITH_HDF5 ON.

[dpo854]

If I write "#define H5MD" in the myconfig.hpp file (the documentation in the above link counts H5MD as a general feature), I get the following error during make -j4. So, I believe it's an external feature which must be activated using " -DWITH_HDF5=ON". Either way, the problem still remains.
"
External feature 'H5MD' can not be defined in myconfig.
There were errors in '/home/dpo854/mycode/espresso/build/src/config/myconfig-final.hpp'
make[2]: *** [src/config/CMakeFiles/check_myconfig] Error 1
make[1]: *** [src/config/CMakeFiles/check_myconfig.dir/all] Error 2
"

[jngrad]

To enable H5MD, you'll need both the python package h5py version 2.6.0 or above, and the library libhdf5 (on Ubuntu, it can be installed via apt-get install libhdf5-openmpi-dev). The CMake log will look like this:

$> cmake .. -DWITH_HDF5=ON
[...]
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so;
               /usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;
               /usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so
               (found suitable version "1.10.0.1", minimum required is "1.8")
               found components:  C 
-- Found PY_h5py: /usr/lib/python3/dist-packages/h5py  
[...]
-- The following features have been enabled:
 * HDF5 (required version >= 1.8), parallel
-- The following OPTIONAL packages have been found:
 * HDF5 (required version >= 1.8), parallel

If h5py or libhdf5 were not found, CMake will carry on and let you compile ESPResSo without the H5MD feature.

[dpo854]

I checked the version of h5py on my cluster, it's '2.10.0'. And my cluster already has hdf5 since on running CMake, I got this message

-- The following OPTIONAL packages have been found:

• CUDA (required version >= 9.0)
• FFTW3
• HDF5 (required version >= 1.8)
• GSL
• Doxygen
• Git

[fweik]

@jngrad are you sure the python module is needed? I didn't think that would be the case, IIRC it's also all C++ code...

[dpo854]

To support my statement about hdf5 having already been installed, I attach here the snippet of the output of ccmake.. related to hdf5.

HDF5_C_COMPILER_EXECUTABLE /software/anaconda3/bin/h5cc
HDF5_C_LIBRARY_dl /usr/lib64/libdl.so
HDF5_C_LIBRARY_hdf5 /software/anaconda3/lib/libhdf5.so
HDF5_C_LIBRARY_m /usr/lib64/libm.so
HDF5_C_LIBRARY_pthread /usr/lib64/libpthread.so
HDF5_C_LIBRARY_rt /usr/lib64/librt.so
HDF5_C_LIBRARY_z /software/anaconda3/lib/libz.so
HDF5_DIFF_EXECUTABLE /software/anaconda3/bin/h5diff
HDF5_DIR HDF5_DIR-NOTFOUND

[dpo854]

It looks like a complicated technical problem. So, what are some other ways of storing the coordinates for computing density profiles (mind you I need to analyze thousands of frames to get smooth results)? Can .vtf files be read from a python script? I ask that because they are designed specifically for visualization through VMD.

[jngrad]

@fweik you're right, the Python package isn't required. There's actually no point in having it in the CMake logic. The H5MD CMake logic could be cleaned up a bit. We should also consider making CMake fail instead of carrying on silently if an external feature is requested but the libraries not found. The current behavior probably stems from CMake's option() command that only takes a binary value. We could use instead an enum ON/OFF/IF_AVAILABLE that would silently fail to install an external feature only for IF_AVAILABLE (and that would be the default).

@dpo854 in build/src/config/cmake_config.hpp, do you get #define H5MD or /* #undef H5MD */? Are you compiling ESPResSo 4.1.x or the current branch? The variable HDF5_DIR=HDF5_DIR-NOTFOUND is harmless, I also have it but H5MD is in still in my list of features.

[fweik]

@jngrad It's needed for a test, the corresponding cmake stuff should be at the point of usage imho.

[jngrad]

@fweik it took me a while to find out this command actually sets variable PY_H5PY to be used two levels down in the testsuite/python folder. We probably don't even need that variable anymore, now that we have unittest fixtures to skip tests when a python package is missing, like in test_MDAnalysisIntegration.py. I'll have a closer look once the CMake refactor PR gets merged.

[dpo854]

@jngrad I get "/* #undef H5MD */" in the cmake_config.hpp file. My espresso version is 4.1.2.

[jngrad]

Alright, looking through the Ubuntu repository, it looks like libhdf5 isn't the correct package. It offers the serial version, while ESPResSo requires the parallel version. The documentation should be clearer about that, and we should rewrite the CMake logic to throw and error if only the serial version is found, instead of printing "Found HDF5" and carrying on without HDF5.

To get the parallel version, install libhdf5-openmpi-dev if you're using OpenMPI, or libhdf5-mpich-dev if you're using MPICH. Let us know if this works for you.

[dpo854]

Thank you for tracking down the source of the error. A silly question. Since I am trying to install it on my cluster and sudo apt install or sudo yum install do not work there, is there a way to get its .tar.gz file? That's how I built fftw on my cluster.

[jngrad]

I've obtained the sources from the Ubuntu repo page, under "Download Source Package". Workflow:

wget http://archive.ubuntu.com/ubuntu/pool/universe/h/hdf5/hdf5_1.10.0-patch1+docs.orig.tar.gz
tar xfz hdf5_1.10.0-patch1+docs.orig.tar.gz 
cd hdf5-1.10.0-patch1+docs/
CC=$(which mpicc) ./configure --enable-parallel --prefix=/work/jgrad/hdf5-build
make -j$(nproc)
make check -j2 # just for testing
make install

then add set(HDF5_C_COMPILER_EXECUTABLE /work/jgrad/hdf5-build/bin/h5pcc) above find_package(HDF5 "1.8" COMPONENTS C) in the top-level CMakeLists.txt file. Change the prefix to any location you have write access to. I was able to compile 4.1.2 and run the h5md test with it:

-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /work/jgrad/hdf5-build/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libz.so;
               /usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so
               (found suitable version "1.10.0.1", minimum required is "1.8")
               found components:  C 
[...]
$> ./pypresso testsuite/python/h5md.py
Ran 13 tests in 0.184s
OK

[dpo854]

Do I have to delete " set(HDF5_PREFER_PARALLEL 1)", which is before "find_package(HDF5 "1.8" COMPONENTS C)"?

[dpo854]

Now, it throws up this error. I am sorry for causing you so much inconvenience. However, I did follow all the abovementioned steps.

import espressomd
File "/home/dpo854/mycode/espresso/install/lib/python3.4/site-packages/espressomd        /__init__.py", line 23, in <module>
import espressomd._init
ImportError: libhdf5.so.100: cannot open shared object file: No such file or directory

[dpo854]

I added this line "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/dpo854/mycode/hdf5-1.10.0-patch1+docs/build/lib" to my submission script but it did not help. Rather it gives me another error:

Traceback (most recent call last):
File "minimal.py", line 36, in <module>
import espressomd
File "/home/dpo854/mycode/espresso/install/lib/python3.4/site-packages/espressomd/__init__.py", line 23, in <module>
import espressomd._init
ImportError: libimf.so: cannot open shared object file: No such file or directory

[KaiSzuttor]

@dpo854 sorry, but we cannot give support for building dependencies from source. Please ask on the mailing list for hdf5.

[KaiSzuttor]

I'll close this for now, if you have any further questions please open another ticket.

[dpo854]

No problem. On a side note, is it okay to save manually the positions in text files as an alternative to hdf5?

[KaiSzuttor]

Of course you can do that, but it's quite slow. The best option is probably live analysis if you know beforehand what you want to look at.

[dpo854]

Pardon my ignorance, but isn't the computational cost involved in live analysis (I assume by this you mean determining profiles by looping over particles in each step as the simulation goes forward) the same as writing to a file in each step and analyzing it later from a separate script?

[KaiSzuttor]

no because file I/O slows you down significantly.

[dpo854]

Ok. That was quite helpful. Hopefully, now I can find a way around the hdf5 issue. Thanks