Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault when copying dataset with attributes #2414

Closed
takluyver opened this issue Jan 20, 2023 · 5 comments
Closed

Segfault when copying dataset with attributes #2414

takluyver opened this issue Jan 20, 2023 · 5 comments
Assignees
@vchoi-hdfgroup
Labels
Component - C Library Core C library issues (usually in the src directory) Priority - 1. High 🔼 These are important issues that should be resolved in the next release Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub UNCONFIRMED New issues are unconfirmed until a maintainer can duplicate them
Milestone
1.14.5

Comments

@takluyver
Copy link
Contributor

Describe the bug

We've come across a case where H5Ocopy causes a segfault when copying an object with attributes. We originally came across this via h5py, but I can replicate it with the h5copy command:

h5copy -i repacked2.h5 -o test2.h5 -s value -d value

Here is the input file to reproduce this (zipped to please Github): repacked2.zip

I suspect that this is related to dense attribute storage - when deleting attributes, I could reproduce it with 6 attributes, but not with 4 (6 is the default min_dense, so below this it will switch back to compound storage). Passing -f noattr to h5copy avoids the error.

I haven't been yet been able to reproduce it with a newly created file - the example file above is a heavily cut down version of a file from the European XFEL data acquisition system. I believe it's a valid HDF5 file - I can read all of the attributes with no errors. And ideally even if the file is corrupt, we'd like HDF5 to catch that without segfaulting.

We saw a segfault here on various HDF5 versions (1.10.6, 1.12.1, 1.12.2, 1.14.0), but I've focused on 1.14.0 for investigation.

GDB Backtrace from h5copy 1.14.0 
#0  0x00002aaaab271ec4 in H5T__conv_vlen (src_id=216172782113784125, dst_id=216172782113784126, cdata=0x699580, 
    nelmts=1, buf_stride=0, bkg_stride=0, buf=0x69ae78, bkg=0x0) at H5Tconv.c:3296
#1  0x00002aaaab256178 in H5T_convert (tpath=0x699510, src_id=216172782113784125, dst_id=216172782113784126, 
    nelmts=1, buf_stride=0, bkg_stride=0, buf=0x69ae78, bkg=0x0) at H5T.c:5449
#2  0x00002aaaaad5cc27 in H5A__attr_copy_file (attr_src=0x697830, file_dst=0x682710, 
    recompute_size=0x7fffffffd2ff, cpy_info=0x7fffffffd5a0) at H5Aint.c:2367
#3  0x00002aaaaad5dbbe in H5A__dense_post_copy_file_cb (attr_src=0x697830, _udata=0x7fffffffd310) at H5Aint.c:2556
#4  0x00002aaaaad4c930 in H5A__dense_iterate_bt2_cb (_record=0x696ac8, _bt2_udata=0x7fffffffd210)
    at H5Adense.c:1098
#5  0x00002aaaaad8e498 in H5B2__iterate_node (hdr=0x695ef0, depth=0, curr_node=0x696000, parent=0x695ef0, 
    op=0x2aaaaad4c51c <H5A__dense_iterate_bt2_cb>, op_data=0x7fffffffd210) at H5B2int.c:1702
#6  0x00002aaaaad75cbe in H5B2_iterate (bt2=0x695640, op=0x2aaaaad4c51c <H5A__dense_iterate_bt2_cb>, 
    op_data=0x7fffffffd210) at H5B2.c:423
#7  0x00002aaaaad4d0ba in H5A__dense_iterate (f=0x67be60, loc_id=0, ainfo=0x687d80, idx_type=H5_INDEX_NAME, 
    order=H5_ITER_NATIVE, skip=0, last_attr=0x0, attr_op=0x7fffffffd300, op_data=0x7fffffffd310)
    at H5Adense.c:1225
#8  0x00002aaaaad5e150 in H5A__dense_post_copy_file_all (src_oloc=0x7fffffffd720, ainfo_src=0x687d80, 
    dst_oloc=0x7fffffffd670, ainfo_dst=0x694880, cpy_info=0x7fffffffd5a0) at H5Aint.c:2621
#9  0x00002aaaab092095 in H5O__ainfo_post_copy_file (src_oloc=0x7fffffffd720, mesg_src=0x687d80, 
    dst_oloc=0x7fffffffd670, mesg_dst=0x694880, mesg_flags=0x7fffffffd424, cpy_info=0x7fffffffd5a0)
    at H5Oainfo.c:468
#10 0x00002aaaab0bbe3f in H5O__copy_header_real (oloc_src=0x7fffffffd720, oloc_dst=0x7fffffffd670, 
    cpy_info=0x7fffffffd5a0, obj_type=0x0, udata=0x0) at H5Ocopy.c:719
#11 0x00002aaaab0bcfff in H5O__copy_header (oloc_src=0x7fffffffd720, oloc_dst=0x7fffffffd670, 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:1008
#12 0x00002aaaab0bd3cd in H5O__copy_obj (src_loc=0x7fffffffd760, dst_loc=0x7fffffffd7d0, 
    dst_name=0x67a750 "value", ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:1062
#13 0x00002aaaab0b9585 in H5O__copy (loc=0x7fffffffd7e0, src_name=0x67a730 "value", dst_loc=0x7fffffffd7d0, 
    dst_name=0x67a750 "value", ocpypl_id=792633534417207348, lcpl_id=792633534417207349) at H5Ocopy.c:167
#14 0x00002aaaab3b981c in H5VL__native_object_copy (src_obj=0x67be60, loc_params1=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x682710, loc_params2=0x7fffffffd980, dst_name=0x67a750 "value", 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLnative_object.c:155
#15 0x00002aaaab38f617 in H5VL__object_copy (src_obj=0x67be60, src_loc_params=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x682710, dst_loc_params=0x7fffffffd980, dst_name=0x67a750 "value", 
    cls=0x642150, ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLcallback.c:5848
---Type <return> to continue, or q <return> to quit---
#16 0x00002aaaab38f8b9 in H5VL_object_copy (src_obj=0x681e60, src_loc_params=0x7fffffffd950, 
    src_name=0x67a730 "value", dst_obj=0x684d90, dst_loc_params=0x7fffffffd980, dst_name=0x67a750 "value", 
    ocpypl_id=792633534417207348, lcpl_id=792633534417207349, dxpl_id=792633534417207304, req=0x0)
    at H5VLcallback.c:5887
#17 0x00002aaaab07f7e6 in H5O__copy_api_common (src_loc_id=72057594037927936, src_name=0x67a730 "value", 
    dst_loc_id=72057594037927937, dst_name=0x67a750 "value", ocpypl_id=792633534417207348, 
    lcpl_id=792633534417207349, token_ptr=0x0, _vol_obj_ptr=0x0) at H5O.c:461
#18 0x00002aaaab07fb22 in H5Ocopy (src_loc_id=72057594037927936, src_name=0x67a730 "value", 
    dst_loc_id=72057594037927937, dst_name=0x67a750 "value", ocpypl_id=792633534417207348, 
    lcpl_id=792633534417207349) at H5O.c:553
#19 0x000000000040690e in main (argc=9, argv=0x7fffffffdc68) at h5copy.c:458

Expected behavior

Not segfaulting. 😉

Platform (please complete the following information)

  • HDF5 version (if building from a maintenance branch, please include the commit hash): 1.14.0
  • OS and version: CentOS 7
  • Compiler and version: gcc 4.8.5
  • Build system (e.g. CMake, Autotools) and version: Autotools
  • Any configure options you specified: --enable-build-mode=debug
@takluyver takluyver added the bug label Jan 20, 2023
@takluyver
Copy link
Contributor Author

HDF group made a JIRA issue for this: https://jira.hdfgroup.org/browse/HDFFV-11360

@derobins
Copy link
Member

derobins commented Mar 3, 2023

We're going to be moving our product-specific project planning to GitHub, so I've added this to the 1.14.1 release project.

@derobins derobins changed the title [BUG] Segfault when copying dataset with attributes Segfault when copying dataset with attributes Mar 3, 2023
@derobins derobins added Priority - 1. High 🔼 These are important issues that should be resolved in the next release Component - C Library Core C library issues (usually in the src directory) Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub UNCONFIRMED New issues are unconfirmed until a maintainer can duplicate them and removed bug labels Mar 3, 2023
@paulmueller paulmueller mentioned this issue Jun 30, 2023
Closed
@kyang2014
Copy link
Collaborator

Maybe this is known. I just tried 1.14.2 with the same file. It is still seg fault.

 "./h5copy -i repacked2.h5 -o test2.h5 -s value -d value
Segmentation fault (core dumped)"

valgrind doesn't give me what I want. However, it outputs the following message:

"h5copy error: Could not open output file <test2.h5>...Exiting"

@takluyver takluyver mentioned this issue Nov 27, 2023
Merged
vchoi-hdfgroup added a commit to vchoi-hdfgroup/hdf5 that referenced this issue Jan 29, 2024
@vchoi-hdfgroup
Fix for github issue HDFGroup#2414: segfault when copying dataset wit…
3169e59 
…h attributes.

This also fixes github issue HDFGroup#3241: segfault when copying dataset.
Need to set the location via H5T_set_loc() of the src datatype
when copying dense attributes.
Otherwise the vlen callbacks are not set up therefore causing seg fault
when doing H5T_convert() -> H5T__conv_vlen().
lrknox pushed a commit that referenced this issue Feb 6, 2024
@vchoi-hdfgroup
Fix for github issue #2414: segfault when copying dataset with attrib… (
49cce91 
#3967)

* Fix for github issue #2414: segfault when copying dataset with attributes.
This also fixes github issue #3241: segfault when copying dataset.
Need to set the location via H5T_set_loc() of the src datatype
when copying dense attributes.
Otherwise the vlen callbacks are not set up therefore causing seg fault
when doing H5T_convert() -> H5T__conv_vlen().
lrknox pushed a commit to lrknox/hdf5 that referenced this issue Feb 15, 2024
@vchoi-hdfgroup @lrknox
Fix for github issue HDFGroup#2414: segfault when copying dataset wit…
5153821 
…h attrib… (HDFGroup#3967)

* Fix for github issue HDFGroup#2414: segfault when copying dataset with attributes.
This also fixes github issue HDFGroup#3241: segfault when copying dataset.
Need to set the location via H5T_set_loc() of the src datatype
when copying dense attributes.
Otherwise the vlen callbacks are not set up therefore causing seg fault
when doing H5T_convert() -> H5T__conv_vlen().
lrknox added a commit that referenced this issue Feb 15, 2024
@lrknox
Merge pull request #4019 from lrknox/1_14_dev_sync2_lrk
413d10f 
* Update upload- artifact to match download version (#3929)

* Reorg and update options for doc and cmake config (#3934)

* Add binary build for linux S3 (#3936)

* Clean up Doxygen for szip functions and constants (#3943)

* Replace off_t with HDoff_t internally (#3944)

off_t is a 32-bit signed value on Windows, so we should use HDoff_t
(which is __int64 on Windows) internally instead.

Also defines HDftell on Windows to be _ftelli64().

* Fix chid_t to hid_t (#3948)

* Fortran API work. (#3941)

* - Added Fortran APIs:
      H5FGET_INTENT_F, H5SSELECT_ITER_CREATE_F, H5SSEL_ITER_GET_SEQ_LIST_F,
      H5SSELECT_ITER_CLOSE_F, H5S_mp_H5SSELECT_ITER_RESET_F

    - Added Fortran Parameters:
      H5S_SEL_ITER_GET_SEQ_LIST_SORTED_F, H5S_SEL_ITER_SHARE_WITH_DATASPACE_F

    - Added tests for new APIs
    - Removed H5F C wrapper stubs
    - Documentation misc. cleanup.

* Add the user test program in HDFFV-9174 for committed types. (#3937)

Add the user test program for committed types in HDFFV-9174

* Remove cached datatype conversion path table entries on file close (#3942)

* fixed BIND name (#3957)

* update H5Ssel_iter_reset_f test

* Change 'extensible' to 'fixed' in H5FA code (#3964)

* RF: move codespell configuration into .codespellrc so could be used locally as well (#3958)

* Add RELEASE.txt note for the fix for issue #1256 (#3955)

* Fix doxygen errors (#3962)

* Add API support for Fortran MPI_F08 module definitions. (#3959)

* revert to using c-stub for _F08 MPI APIs

* use mpi compiler wrappers for cmake and nvhpc

* Added a GitHub Codespaces configuration. (#3966)

* Fixed XL and gfortran errors (#3968)

* h5 compiler wrappers now pass all arguments passed to it to the compile line (#3954)

* The issue was that the "allargs" variable was not being used in the final command of the compiler wrapper. Any entries containing an escaped quote (\", \') or other non-matching argument (*) would not be passed to the compile line. I have fixed this problem by ensuring all arguments passed to the compiler wrapper are now included in the compile line.

* Add binary testing to CI testing (#3971)

* Replace 'T2' with ' ' to avoid failure to match expected output due to (#3975)

* Clarify vlen string datatype message (#3950)

* append '-WF,' when passing C preprocessor directives to the xlf compiler (#3976)

* Create CITATION.cff (#3927)

Add citation source based on http://web.archive.org/web/20230610185232/https://portal.hdfgroup.org/display/knowledge/How+do+I+properly+cite+HDF5%The space difference in the Fortran examples must be fixed to match the expected output for compression filter examples.

* corrected warning: implicit conversion changes signedness (#3982)

* Skip mac bintest until more reliable (#3983)

* Make platform specific test presets for windows and macs (#3988)

* chore: fix typo (#3989)

* Add a missing left parenthesis in RELEASE.txt. (#3990)

* Remove ADB signature from RELEASE.txt. (#3986)

* Bump the github-actions group with 6 updates (#3981)

* Sync API tests with vol-tests (#3940)

* Fix for github issue #2414: segfault when copying dataset with attrib… (#3967)

* Fix for github issue #2414: segfault when copying dataset with attributes.
This also fixes github issue #3241: segfault when copying dataset.
Need to set the location via H5T_set_loc() of the src datatype
when copying dense attributes.
Otherwise the vlen callbacks are not set up therefore causing seg fault
when doing H5T_convert() -> H5T__conv_vlen().

* Fix broken links caused by examples relocation. (#3995)

* Add abi-complience check and upload to releases (#3996)

* Fix h5watch test failures to ignore system warnings on ppc64le. (#3997)

* Remove oneapi/clang compiler printf() type warning. (#3994)

* Updated information about obtaining the HDF5 source code to use the repos. (#3972)

* Fix overwritten preset names (#4000)

* Fix incompatible pointer type warnings in object reference examples (#3999)

* Fix build issue and some warnings in H5_api_dataset_test.c (#3998)

* Modern C++ dtor declarations (#1830)

* C++ dtor modernization

- Replaced a bunch of empty dtors with `= default`
- Removed deprecated `throw()`. In C++11, dtors are `noexcept` by default.

* remove incorrect check for environ (#4002)

* Add a missing file into Makefile.am for MinGW Autotools build error. (#4004)

* Issue #1824: Replaced most remaining sprintf with safer snprint (#4003)

* Add hl and cpp ABI reports to daily build (#4006)

* Don't add files and directories with names that begin with ., or that match *autom4te* to release tar & zip files. (#4009)

* Fix some output issues with ph5diff (#4008)

* Update install texts (#4010)

* Add C in project line for CMake to fix #4012. (#4014)

* separate out individual checks for string removal (#4015)

* Add compound subset ops on attributes to API tests (#4005)

---------
@derobins derobins added this to the 1.14.5 milestone Mar 28, 2024
@takluyver takluyver mentioned this issue Apr 22, 2024
Open
@ajelenak
Copy link
Contributor 

h5copy -i repacked2.h5 -o test2.h5 -s value -d value

Works for me with 1.14.4.

@takluyver
Copy link
Contributor Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vchoi-hdfgroup vchoi-hdfgroup

Labels
Component - C Library Core C library issues (usually in the src directory) Priority - 1. High 🔼 These are important issues that should be resolved in the next release Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub UNCONFIRMED New issues are unconfirmed until a maintainer can duplicate them
Projects
None yet
Milestone
1.14.5
Development

No branches or pull requests
5 participants
@derobins @takluyver @ajelenak @kyang2014 @vchoi-hdfgroup