pyart.aux_io.read_gamic() failing with h5py 3.1.0 #977
Comments
cclopes
changed the title
pyart.aux_io.read_gamic() failing with h5py 3.1.0
|
@cclopes Thank you for the information! Yeah seems they are possibly decoding during reading and we are duplicating it in the newer versions. I'll add a try and except at the moment, for a workaround to work with earlier version. I'll submit a PR. Thanks again for bringing this to our attention |
|
This breaking change in |
|
@kmuehlbauer Ah gotcha, yeah we use hdf5 in alot of the aux ios , so have to do some digging to see where else this will arise. I'm assuming its not a bug on their part, just we have to now remove decoding on our end? |
|
@zssherman No, it's not a bug in that sense. The new string handling is described here: https://docs.h5py.org/en/stable/strings.html#reading-strings
So in general all strings are represented as bytes, but not attributes vlen_strings, those are |
|
@kmuehlbauer Makes sense and that link was helpful, I think I have a fix. Thanks again! |
|
@cclopes I implemented a fix and tested it with the file you provided. I accepted the PR and should be available using git. Thanks again for bringing this to our attention. |
|
That's great! Thanks for the quick fix 😄 |
When I try to open a file with
pyart.aux_io.read_gamic()in an environment with h5py 3.1.0, the following error occurs:Environment that caused the error
# Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge arm_pyart 1.11.3 py38h5c078b8_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge boost-cpp 1.74.0 hc6e9bd1_2 conda-forge brotlipy 0.7.0 py38h497a2fe_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.17.1 h36c2ea0_0 conda-forge ca-certificates 2020.12.5 ha878542_0 conda-forge cached-property 1.5.1 py_0 conda-forge cairo 1.16.0 h7979940_1007 conda-forge cartopy 0.18.0 py38h2516ec2_10 conda-forge certifi 2020.12.5 py38h578d9bd_1 conda-forge cffi 1.14.5 py38ha65f79e_0 conda-forge cfitsio 3.470 hb418390_7 conda-forge cftime 1.4.1 py38h5c078b8_0 conda-forge chardet 4.0.0 py38h578d9bd_1 conda-forge cryptography 3.4.4 py38h3e25421_0 conda-forge curl 7.71.1 he644dc0_8 conda-forge cycler 0.10.0 py_2 conda-forge decorator 4.4.2 py_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge expat 2.2.10 h9c3ff4c_0 conda-forge fontconfig 2.13.1 hba837de_1004 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.5 h516909a_1002 conda-forge gdal 3.2.1 py38hc0b2d6b_5 conda-forge geos 3.9.0 h9c3ff4c_0 conda-forge geotiff 1.6.0 h2b14fbe_4 conda-forge gettext 0.19.8.1 h0b5b191_1005 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge h5netcdf 0.10.0 pyhd8ed1ab_0 conda-forge h5py 3.1.0 nompi_py38hafa665b_100 conda-forge hdf4 4.2.13 h10796ff_1004 conda-forge hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge icu 68.1 h58526e2_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge ipykernel 5.4.3 py38h81c977d_0 conda-forge ipython 7.20.0 py38h81c977d_2 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.18.0 py38h578d9bd_2 conda-forge jpeg 9d h36c2ea0_0 conda-forge json-c 0.15 h98cffda_0 conda-forge jupyter_client 6.1.11 pyhd8ed1ab_1 conda-forge jupyter_core 4.7.1 py38h578d9bd_0 conda-forge kealib 1.4.14 he4dc956_1 conda-forge kiwisolver 1.3.1 py38h1fd1430_1 conda-forge krb5 1.17.2 h926e7f8_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge libblas 3.9.0 8_openblas conda-forge libcblas 3.9.0 8_openblas conda-forge libcurl 7.71.1 hcdd3856_8 conda-forge libdap4 3.20.6 hd7c4107_1 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_18 conda-forge libgdal 3.2.1 hbde00c8_5 conda-forge libgfortran-ng 9.3.0 hff62375_18 conda-forge libgfortran5 9.3.0 hff62375_18 conda-forge libglib 2.66.7 h1f3bc88_0 conda-forge libgomp 9.3.0 h2828fa1_18 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h02e6976_1012 conda-forge liblapack 3.9.0 8_openblas conda-forge libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge libnghttp2 1.43.0 h812cca2_0 conda-forge libopenblas 0.3.12 pthreads_h4812303_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.1 hfd2b0eb_1 conda-forge librttopo 1.1.0 hccdd1c9_5 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialite 5.0.1 h04c9dda_2 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.3.0 h6de172a_18 conda-forge libtiff 4.2.0 hdc55705_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp-base 1.2.0 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxml2 2.9.10 h72842e0_3 conda-forge lz4-c 1.9.3 h9c3ff4c_0 conda-forge matplotlib-base 3.3.4 py38h0efea84_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge netcdf4 1.5.6 nompi_py38h1cdf482_100 conda-forge numpy 1.20.1 py38h18fd61f_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 hf7af979_0 conda-forge openssl 1.1.1j h7f98852_0 conda-forge packaging 20.9 pyh44b312d_0 conda-forge pandas 1.2.2 py38h51da96c_0 conda-forge parso 0.8.1 pyhd8ed1ab_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.1.0 py38ha0e1e83_2 conda-forge pip 21.0.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge poppler 0.89.0 h2de54a5_5 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 13.1 h6303168_1 conda-forge proj 7.2.0 h277dcde_2 conda-forge prompt-toolkit 3.0.16 pyha770c72_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.8.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyshp 2.1.3 pyh44b312d_0 conda-forge pysocks 1.7.1 py38h578d9bd_3 conda-forge python 3.8.6 hffdb5ce_5_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2021.1 pyhd8ed1ab_0 conda-forge pyzmq 20.0.0 py38h3d7ac18_1 conda-forge readline 8.0 he28a2e2_2 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge scipy 1.6.0 py38hb2138dd_0 conda-forge semver 2.13.0 pyh9f0ad1d_0 conda-forge setuptools 49.6.0 py38h578d9bd_3 conda-forge shapely 1.7.1 py38h604dc72_2 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.34.0 h74cdb3f_0 conda-forge tiledb 2.2.4 hb9a9e87_0 conda-forge tk 8.6.10 h21135ba_1 conda-forge tornado 6.1 py38h497a2fe_1 conda-forge traitlets 5.0.5 py_0 conda-forge trmm_rsl 1.49 3 conda-forge tzcode 2021a h7f98852_0 conda-forge urllib3 1.26.3 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge wradlib 1.9.0 pyhd8ed1ab_0 conda-forge xarray 0.16.2 pyhd8ed1ab_0 conda-forge xerces-c 3.2.3 h9d8b166_2 conda-forge xmltodict 0.12.0 py_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.12 h516909a_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge zeromq 4.3.4 h9c3ff4c_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.8 ha95c52a_1 conda-forgeWhen using another environment with h5py 2.10.0, the file opens normally.
Environment that works
# Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge arm_pyart 1.11.3 py38h5c078b8_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge boost-cpp 1.74.0 hc6e9bd1_2 conda-forge brotlipy 0.7.0 py38h497a2fe_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.17.1 h36c2ea0_0 conda-forge ca-certificates 2020.12.5 ha878542_0 conda-forge cairo 1.16.0 h7979940_1007 conda-forge cartopy 0.18.0 py38h2516ec2_10 conda-forge certifi 2020.12.5 py38h578d9bd_1 conda-forge cffi 1.14.5 py38ha65f79e_0 conda-forge cfitsio 3.470 hb418390_7 conda-forge cftime 1.4.1 py38h5c078b8_0 conda-forge chardet 4.0.0 py38h578d9bd_1 conda-forge cryptography 3.4.4 py38h3e25421_0 conda-forge curl 7.71.1 he644dc0_8 conda-forge cycler 0.10.0 py_2 conda-forge decorator 4.4.2 py_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge expat 2.2.10 h9c3ff4c_0 conda-forge fontconfig 2.13.1 hba837de_1004 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.5 h516909a_1002 conda-forge gdal 3.2.1 py38hc0b2d6b_5 conda-forge geos 3.9.0 h9c3ff4c_0 conda-forge geotiff 1.6.0 h2b14fbe_4 conda-forge gettext 0.19.8.1 h0b5b191_1005 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge h5netcdf 0.10.0 pyhd8ed1ab_0 conda-forge h5py 2.10.0 nompi_py38h7442b35_105 conda-forge hdf4 4.2.13 h10796ff_1004 conda-forge hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge icu 68.1 h58526e2_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge ipykernel 5.4.3 py38h81c977d_0 conda-forge ipython 7.20.0 py38h81c977d_2 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.18.0 py38h578d9bd_2 conda-forge jpeg 9d h36c2ea0_0 conda-forge json-c 0.15 h98cffda_0 conda-forge jupyter_client 6.1.11 pyhd8ed1ab_1 conda-forge jupyter_core 4.7.1 py38h578d9bd_0 conda-forge kealib 1.4.14 he4dc956_1 conda-forge kiwisolver 1.3.1 py38h1fd1430_1 conda-forge krb5 1.17.2 h926e7f8_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge libblas 3.9.0 8_openblas conda-forge libcblas 3.9.0 8_openblas conda-forge libcurl 7.71.1 hcdd3856_8 conda-forge libdap4 3.20.6 hd7c4107_1 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 9.3.0 h2828fa1_18 conda-forge libgdal 3.2.1 hbde00c8_5 conda-forge libgfortran-ng 9.3.0 hff62375_18 conda-forge libgfortran5 9.3.0 hff62375_18 conda-forge libglib 2.66.7 h1f3bc88_0 conda-forge libgomp 9.3.0 h2828fa1_18 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h02e6976_1012 conda-forge liblapack 3.9.0 8_openblas conda-forge libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge libnghttp2 1.43.0 h812cca2_0 conda-forge libopenblas 0.3.12 pthreads_h4812303_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.1 hfd2b0eb_1 conda-forge librttopo 1.1.0 hccdd1c9_5 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialite 5.0.1 h04c9dda_2 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.3.0 h6de172a_18 conda-forge libtiff 4.2.0 hdc55705_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp-base 1.2.0 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxml2 2.9.10 h72842e0_3 conda-forge lz4-c 1.9.3 h9c3ff4c_0 conda-forge matplotlib-base 3.3.4 py38h0efea84_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge netcdf4 1.5.6 nompi_py38h1cdf482_100 conda-forge numpy 1.20.1 py38h18fd61f_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 hf7af979_0 conda-forge openssl 1.1.1j h7f98852_0 conda-forge packaging 20.9 pyh44b312d_0 conda-forge pandas 1.2.2 py38h51da96c_0 conda-forge parso 0.8.1 pyhd8ed1ab_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.1.0 py38ha0e1e83_2 conda-forge pip 21.0.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge poppler 0.89.0 h2de54a5_5 conda-forge poppler-data 0.4.10 0 conda-forge postgresql 13.1 h6303168_1 conda-forge proj 7.2.0 h277dcde_2 conda-forge prompt-toolkit 3.0.16 pyha770c72_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.8.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyshp 2.1.3 pyh44b312d_0 conda-forge pysocks 1.7.1 py38h578d9bd_3 conda-forge python 3.8.6 hffdb5ce_5_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytz 2021.1 pyhd8ed1ab_0 conda-forge pyzmq 20.0.0 py38h3d7ac18_1 conda-forge readline 8.0 he28a2e2_2 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge scipy 1.6.0 py38hb2138dd_0 conda-forge semver 2.13.0 pyh9f0ad1d_0 conda-forge setuptools 49.6.0 py38h578d9bd_3 conda-forge shapely 1.7.1 py38h604dc72_2 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sqlite 3.34.0 h74cdb3f_0 conda-forge tiledb 2.2.4 hb9a9e87_0 conda-forge tk 8.6.10 h21135ba_1 conda-forge tornado 6.1 py38h497a2fe_1 conda-forge traitlets 5.0.5 py_0 conda-forge trmm_rsl 1.49 3 conda-forge tzcode 2021a h7f98852_0 conda-forge urllib3 1.26.3 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge wradlib 1.9.0 pyhd8ed1ab_0 conda-forge xarray 0.16.2 pyhd8ed1ab_0 conda-forge xerces-c 3.2.3 h9d8b166_2 conda-forge xmltodict 0.12.0 py_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.12 h516909a_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge zeromq 4.3.4 h9c3ff4c_0 conda-forge zlib 1.2.11 h516909a_1010 conda-forge zstd 1.4.8 ha95c52a_1 conda-forgeTested with these GAMIC-HDF5 files: .mvol, .HDF5
I believe this issue is related to h5py because this function that uses h5py to read files is also failing because of a string that is read as
"b'[some string]'"in the older version and"[some string]"in the newer version.The text was updated successfully, but these errors were encountered: