Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor H5FD and package initialization #4934

Merged
merged 58 commits into from
Oct 24, 2024

Conversation

qkoziol
Copy link
Contributor

@qkoziol qkoziol commented Oct 6, 2024

  • Reverts PR#1024, which (unnecessarily) switched from deferred package initialization to centralized initialization of all packages and introduced H5FDperform_init() to wrap an internal routine to initialize VFD plugins.
  • Went back to deferred package initialization (primarily), to eliminate unnecessary resource use. (Performance has been verified to be the same either way)
  • Switched VFD plugins to use “#define (H5OPEN, )” pattern, with registration of internal VFD plugins at library initialization time. Eliminates calling API routine (H5FDperform_init) from within library, which was deadlocking threadsafe concurrency. And also eliminates exposing internal library routines in a public header file.
  • Removed copy-and-paste replicas of the H5OPEN macro and put a (better) single definition in H5public.h
  • Separated API and internal routine calls in stdio and multi VFD plugins into separate source files, so that the library doesn’t invoke API routines internally (also a deadlock problem for threadsafe concurrency). Also needed a “private” header for these plugins.
  • Separated registering/unregistering a VFD plugin from initializing/finalizing the plugin, instead of blurring those ideas together. Defers the VFD plugin init to when it’s actually used, which reduces resource usage, especially for the MPI-based plugins like the subfiling, etc.
  • Refactored the copy-and-pasted check for locking into a central location in the H5FD.c code.
  • Fixed a bunch of compiler warnings, especially ones that trigger CI failures with -Werror

qkoziol and others added 30 commits September 27, 2024 17:45
Gets rid of public API call (H5FDperform_init) within the library code and
returns to original way of initializing packages in the library.

Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Update comment about interface initialization routines in H5_init_library

Signed-off-by: Quincey Koziol <[email protected]>
Move all VFDs into the 'H5FD' package, instead of their own individual
packages.

Stop using VFD's 'term' callback to reset the ID for the VFD.  Use explicit
'register' and 'unregister' calls to do that.  Use the 'term' callback only
for dealing with VFD internal resource cleanup.

A _ton_ of other misc. changes, but mainly to update and align the VFDs more
with current best practices in the library.

Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
Signed-off-by: Quincey Koziol <[email protected]>
@derobins derobins changed the title Refactor h5fd and package initialization Refactor H5FD and package initialization Oct 21, 2024
jhendersonHDF and others added 4 commits October 22, 2024 13:52
Ignores some of the older Autotools platform files, since the Autotools
will be dropped in the next major release (and we don't have
compilers on which to test, anyway).

Also drops support for the old, non-compliant MSVC
preprocessor.
The high-level GIF tools, h52gif and gif2h5, have unfixed CVE issues
(with no proof-of-concept files). They are not critical tools, are not
well maintained, and are an odd fit for building with the library.
Because of this, they have been removed. We may move them to a separate
repository in the future.
src/H5.c Show resolved Hide resolved
src/H5.c Outdated Show resolved Hide resolved
src/H5CX.c Show resolved Hide resolved
src/H5private.h Outdated Show resolved Hide resolved
qkoziol and others added 5 commits October 23, 2024 12:44
This VFD will need to be reworked. Disabling it for now.
H5FD__ros3_init() doesn't emit errors and needs FUNC_ENTER_PACKAGE_NOERR
@derobins derobins merged commit 2c58357 into HDFGroup:develop Oct 24, 2024
62 checks passed
@qkoziol qkoziol deleted the refactor_h5fd_and_packages branch October 24, 2024 14:12
brtnfld pushed a commit to brtnfld/hdf5 that referenced this pull request Oct 31, 2024
- Reverts PR#1024, which (unnecessarily) switched from deferred
  package initialization to centralized initialization of all packages
  and introduced H5FDperform_init() to wrap an internal routine
  to initialize VFD plugins.
- Went back to deferred package initialization (primarily), to
  eliminate unnecessary resource use. (Performance has been
  verified to be the same either way)
- Switched VFD plugins to use “#define (H5OPEN, )” pattern,
  with registration of internal VFD plugins at library initialization
  time. Eliminates calling API routine (H5FDperform_init) from
  within the library, which was deadlocking threadsafe concurrency.
  And also eliminates exposing internal library routines in a public
  header file.
- Removed copy-and-paste replicas of the H5OPEN macro and put
  a (better) single definition in H5public.h
- Separated API and internal routine calls in stdio and multi VFD
  plugins into separate source files, so that the library doesn’t
  invoke API routines internally (also a deadlock problem for
  threadsafe concurrency). Also needed a “private” header for these
  plugins.
- Separated registering/unregistering a VFD plugin from initializing
  /finalizing the plugin, instead of blurring those ideas together.
  Defers the VFD plugin init to when it’s actually used, which reduces
  resource usage, especially for the MPI-based plugins like the subfiling,
  etc.
- Refactored the copy-and-pasted check for locking into a central
  location in the H5FD.c code.
- Fixed a bunch of compiler warnings, especially ones that trigger CI
  failures with -Werror
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component - C Library Core C library issues (usually in the src directory) Merge - Develop Only This cannot be merged to previous versions of HDF5 (file format or breaking API changes) Priority - 0. Blocker ⛔ This MUST be merged for the release to happen Type - Bug / Bugfix Please report security issues to [email protected] instead of creating an issue on GitHub Type - Improvement Improvements that don't add a new feature or functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants