Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to 1.14.5 #233

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

conda-forge-admin
Copy link
Contributor

@conda-forge-admin conda-forge-admin commented Oct 16, 2024

Updates to 1.14.5

Also removes the GIF tools that have CVEs #187

Closes #187

@hmaarrfk hmaarrfk changed the title MNT: rerender Update to 1.14.5 Oct 16, 2024
@conda-forge-admin
Copy link
Contributor Author

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@hmaarrfk
Copy link
Contributor

The osx arm patch needs to be adapted. It looked like it was partially integrated so it is harder than I wanted it to be.

@xylar
Copy link

xylar commented Oct 20, 2024

Just wanted to make you all aware that we're stuck on the 1.14.4 migration in kealib:
conda-forge/kealib-feedstock#65
We're seeing an issue in osx that H5::FileIException are not getting caught in catch clauses. This may be a clang issue but it doesn't seem to occur with the same clang version, but with hdf5 1.14.3 so it seems to be a clang issue brought in my hdf5 at the least.

@gillins and I would like to add a test to hdf5 to check for this but neither of us has had time to write one so far. It would be nice to have this addressed before 1.14.5 gets built, though.

@hmaarrfk
Copy link
Contributor

We're seeing an issue in osx that H5::FileIException are not getting caught in catch clauses. This may be a clang issue but it doesn't seem to occur with the same clang version, but with hdf5 1.14.3 so it seems to be a clang issue brought in my hdf5 at the least.

This might be pretty hard to troubleshoot. I might in a first step do the following locally if I were you:

  1. Get to a point where you can compile HDF5 locally on an OSX machine from source (not conda) but within your conda environment.
  2. Assume it is ABI compatible throughout 1.14.3 and 1.14.4.
  3. Use git bisect.
  4. get a clue as to what went wrong???

Those kinds of failures suck. Sorry you are hitting it....

@xylar
Copy link

xylar commented Oct 21, 2024

@hmaarrfk, that does certainly sound tricky. I don't have access to a non-ARM Mac and I have no expertise in building hdf5 from source. I'm just pointing out the issue encountered by kealib because it essentially snarled up the hdf5 migration before it could get to most downstream packages (notably gdal) and I expect the same to happen with this version unless we (as a community) can track this issue down.

My hope is to create a simple reproducer of the OSX issue (on CI here). I'll try to get to it sooner rather than later.

@gillins
Copy link
Contributor

gillins commented Oct 21, 2024

@xylar @petebunting have added test, can you please check this makes sense? I'm hoping this goes away with the various memory corruption fixes in this release of hdf5, fingers crossed.

{
H5::Exception::dontPrint();

H5::H5File *keaImgH5File = new H5::H5File("/tmp/test.hd5", H5F_ACC_TRUNC, H5::FileCreatPropList::DEFAULT);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gillins, probably name this something like h5file because the reference to keaImg is confusing.

}
catch (const H5::Exception &e)
{
std::cout << "Failed to open dataset (expected)\n";

This comment was marked as resolved.

std::cout << "Failed to open dataset (expected)\n";
}

return 0;

This comment was marked as resolved.

@xylar
Copy link

xylar commented Oct 21, 2024

@gillins, I have some suggested changes. We want your test to fail if the exception is not caught.

But the gist of the test is exactly what I had in mind. I hope it does the trick and either demonstrated the problem or shows that it has been fixed in this version.

It might be worth cherry-picking it into its own PR to see whether it has the expected effect on 1.14.4 as well.

@gillins
Copy link
Contributor

gillins commented Oct 21, 2024

Thanks @xylar have incorporated these changes. AFAICT the problem we were seeing was that the exception was not caught at all and the process was terminated by the C++ runtime (presumably with a non-zero exit code).

recipe/build.sh Outdated

# test for hdf5 C++ exceptions
$CXX $RECIPE_DIR/testhdf5exc.cpp -o testhdf5exc -L$PREFIX/lib -lhdf5_cpp -lhdf5 -I$PREFIX/include
LD_LIBRARY_PATH=$PREFIX/lib DYLD_LIBRARY_PATH=$PREFIX/lib ./testhdf5exc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
LD_LIBRARY_PATH=$PREFIX/lib DYLD_LIBRARY_PATH=$PREFIX/lib ./testhdf5exc
if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR}" != "" ]]; then
./testhdf5exc
fi

do you really need the LD_LIBRARY_PATH?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the tests CANNOT run on OSX (and later windows) for arm64 builds since we don't have emulators for those.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gillins and @hmaarrfk, as you may have seen, the test is failing to build under linux-64:

+ x86_64-conda-linux-gnu-c++ /home/conda/recipe_root/testhdf5exc.cpp -o testhdf5exc -L/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib -lhdf5_cpp -lhdf5 -I/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libcrypto.so.3, needed by /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so, not found (try using -rpath or -rpath-link)
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libcurl.so.4, needed by /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so, not found (try using -rpath or -rpath-link)
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libsz.so.2, needed by /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so, not found (try using -rpath or -rpath-link)
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: warning: libz.so.1, needed by /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so, not found (try using -rpath or -rpath-link)
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_easy_perform'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `EVP_sha256@OPENSSL_3.0.0'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_easy_init'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_slist_append'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `compress2'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `compressBound@ZLIB_1.2.0'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `HMAC@OPENSSL_3.0.0'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `inflate'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `inflateInit_'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `SHA256@OPENSSL_3.0.0'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_easy_cleanup'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `inflateEnd'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_slist_free_all'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `SZ_encoder_enabled'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_easy_setopt'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_global_init'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `SZ_BufftoBuffDecompress'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `SZ_BufftoBuffCompress'
/home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/13.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/hdf5_1729557442008/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libhdf5.so: undefined reference to `curl_global_cleanup'
collect2: error: ld returned 1 exit status

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might make more sense to add the test to run_tests.sh instead of build.sh and to build it with cmake similarly to:
https://github.com/conda-forge/hdf5-feedstock/blob/main/recipe/run_test.sh#L37-L40
https://github.com/conda-forge/hdf5-feedstock/blob/main/recipe/test/CMakeLists.txt
https://github.com/conda-forge/hdf5-feedstock/blob/main/recipe/test/h5_cmprss.c
That's what I originally had in mind. I think cmake will figure out linking in the necessary libs and it's not worth trying to duplicate that effort some other way.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand, maybe it makes sense to take the test back out since it didn't fail in #234

@xylar
Copy link

xylar commented Oct 21, 2024

Thanks @xylar have incorporated these changes. AFAICT the problem we were seeing was that the exception was not caught at all and the process was terminated by the C++ runtime (presumably with a non-zero exit code).

Ah, excellent point. Thanks!

@xylar
Copy link

xylar commented Oct 22, 2024

@gillins's test passes for osx-64 here. That's good news! But if it also passes in my test PR #234 with 1.14.4, it might mean we haven't found the culprit yet.

recipe/build.sh Outdated

# test for hdf5 C++ exceptions
if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR}" != "" ]]; then
$CXX $RECIPE_DIR/testhdf5exc.cpp -o testhdf5exc -L$PREFIX/lib -lhdf5_cpp -lhdf5 -I$PREFIX/include
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to add the standard CXXFLAGS

@gillins
Copy link
Contributor

gillins commented Oct 30, 2024

Should I remove my test from this PR? Didn't seem to actually help us...

@hmaarrfk hmaarrfk force-pushed the conda_forge_admin_187 branch from 5f9f955 to afcd1bb Compare December 8, 2024 14:24
@hmaarrfk
Copy link
Contributor

hmaarrfk commented Dec 8, 2024

I would like to continue this effort to get it merged.

Ultimately, HDF5 1.14.4 broke so much stuff for windows #240 that h5py is really hesitant to merge their migration PR conda-forge/h5py-feedstock#151 conda-forge/h5py-feedstock#152

I've created a migration branch for 1.14.3 and we can continue to use main to stay updated with the latest version of HDF5 to ensure we don't get too many changes all at once.

If you are invested in keeping HDF5 updated for windows, please please comment HDFGroup/hdf5#5037 to raise the urgency of reverting the changes.

@conda-forge-admin
Copy link
Contributor Author

conda-forge-admin commented Dec 8, 2024

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

  • ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). Your recipe may not receive automatic updates and/or may not be compatible with conda-forge's infrastructure. Please check the logs for more information and ensure your recipe can be parsed.
  • ℹ️ The recipe is not parsable by parser conda-recipe-manager. Your recipe may not receive automatic updates and/or may not be compatible with conda-forge's infrastructure. Please check the logs for more information and ensure your recipe can be parsed.

This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12222809928. Examine the logs at this URL for more detail.

@@ -23,6 +23,10 @@ h5c++ -showconfig
h5c++ ${CXXFLAGS} ${LDFLAGS} h5tutr_cmprss.cpp -o h5tutr_cmprss
./h5tutr_cmprss

echo "test for hdf5 C++ exceptions"
h5c++ ${CXXFLAGS} ${LDFLAGS} testhdf5exc.cpp -o testhdf5exc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gillins @xylar you can't use ${CXX} with HDF5 you should be using h5c++ to capture all the flags needed (i think....)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test can be removed, it didn't help prove anything...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok not doing any harm so i'll leave it for now, but I might just rebase things later.

Gotta get some time to look into the osx-arm failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Should we ship GIF Tools? Unaddressed CVE's and HDF5
4 participants