Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add filter plugin user guide text. Fix registered URL in docs #4169

Merged
merged 2 commits into from
Mar 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion doxygen/aliases
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,6 @@ ALIASES += sa_metadata_ops="\sa \li H5Pget_all_coll_metadata_ops() \li H5Pget_co
################################################################################

ALIASES += ref_cons_semantics="<a href=\"https://portal.hdfgroup.org/display/HDF5/Enabling+a+Strict+Consistency+Semantics+Model+in+Parallel+HDF5\">Enabling a Strict Consistency Semantics Model in Parallel HDF5</a>"
ALIASES += ref_dld_filters="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+Dynamically+Loaded+Filters\">HDF5 Dynamically Loaded Filters</a>"
ALIASES += ref_file_image_ops="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+File+Image+Operations\">HDF5 File Image Operations</a>"
ALIASES += ref_filter_pipe="<a href=\"https://portal.hdfgroup.org/display/HDF5/HDF5+Data+Flow+Pipeline+for+H5Dread\">Data Flow Pipeline for H5Dread()</a>"
ALIASES += ref_group_impls="<a href=\"https://portal.hdfgroup.org/display/HDF5/Groups\">Group implementations in HDF5</a>"
Expand Down
2 changes: 2 additions & 0 deletions doxygen/dox/UsersGuide.dox
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ HDF5 Release 1.14
<li> \ref subsubsec_dataset_transfer_props
<li> \ref subsubsec_dataset_transfer_store
<li> \ref subsubsec_dataset_transfer_partial
<li> \ref subsubsec_dataset_transfer_dyn_filter
</ul>
\li \ref subsec_dataset_allocation
<ul>
Expand All @@ -147,6 +148,7 @@ HDF5 Release 1.14
<li> \ref subsubsec_dataset_filters_nbit
<li> \ref subsubsec_dataset_filters_scale
<li> \ref subsubsec_dataset_filters_szip
<li> \ref subsubsec_dataset_filters_dyn
</ul>

\ref sec_datatype
Expand Down
4 changes: 2 additions & 2 deletions doxygen/examples/H5.format.1.1.html
Original file line number Diff line number Diff line change
Expand Up @@ -5436,8 +5436,8 @@ <h4><a name="FilterMessage">Name: Data Storage - Filter Pipeline</a></h4>
filters requested and supported by third parties.
Filters supported by The HDF Group are documented immediately
below. Information on 3rd-party filters can be found at
<a href="/services/contributions.html#filters">
<code>https://support.hdfgroup.org/services/contributions.html#filters</code></a>.
<a href="https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html">
<code>https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html</code></a>.
<a href="#Footnote1Change"><sup><small>1</small></sup></a>
<p>
To request a filter identifier, please contact
Expand Down
10 changes: 4 additions & 6 deletions doxygen/examples/H5.format.2.0.html
Original file line number Diff line number Diff line change
Expand Up @@ -12598,9 +12598,8 @@ <h4>
HDF5 Library and for filters requested and supported by third
parties. Filters supported by The HDF Group are documented
immediately below. Information on 3rd-party filters can be found at
The HDF Group&rsquo;s <a
href="http://www.hdfgroup.org/services/contributions.html">
Contributions</a> page.
The HDF Group&rsquo;s <a href="https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html">
Registered Filters</a> page.
</p>

<p>
Expand Down Expand Up @@ -12854,9 +12853,8 @@ <h4>
HDF5 Library and for filters requested and supported by third
parties. Filters supported by The HDF Group are documented
immediately below. Information on 3rd-party filters can be found at
The HDF Group&rsquo;s <a
href="http://www.hdfgroup.org/services/contributions.html">
Contributions</a> page.
The HDF Group&rsquo;s <a href="https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html">
Registered Filters</a> page.
</p>

<p>
Expand Down
8 changes: 4 additions & 4 deletions doxygen/examples/H5.format.html
Original file line number Diff line number Diff line change
Expand Up @@ -14226,8 +14226,8 @@ <h4><a name="FilterMessage">IV.A.2.l. The Data Storage - Filter
Filters supported by The HDF Group are documented immediately
below. Information on 3rd-party filters can be found at
The HDF Group&rsquo;s
<a href="http://www.hdfgroup.org/services/contributions.html">
Contributions</a> page.</p>
<a href="https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html">
Registered Filters</a> page.</p>

<p>
To request a filter identifier, please contact
Expand Down Expand Up @@ -14488,8 +14488,8 @@ <h4><a name="FilterMessage">IV.A.2.l. The Data Storage - Filter
Filters supported by The HDF Group are documented immediately
below. Information on 3rd-party filters can be found at
The HDF Group&rsquo;s
<a href="http://www.hdfgroup.org/services/contributions.html">
Contributions</a> page.</p>
<a href="https://portal.hdfgroup.org/documentation/hdf5-docs/registered_filter_plugins.html">
Registered Filters</a> page.</p>

<p>
To request a filter identifier, please contact
Expand Down
83 changes: 71 additions & 12 deletions src/H5Dmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -815,19 +815,10 @@
* <table>
* <caption>Data pipeline filters</caption>
* <tr>
* <th>Filter</th>
* <th>Built-in Filter</th>
* <th>Description</th>
* </tr>
* <tr>
* <td>gzip compression</td>
* <td>Data compression using zlib.</td>
* </tr>
* <tr>
* <td>Szip compression</td>
* <td>Data compression using the Szip library. See The HDF Group website for more information
* regarding the Szip filter.</td>
* </tr>
* <tr>
* <td>N-bit compression</td>
* <td>Data compression using an algorithm specialized for n-bit datatypes.</td>
* </tr>
Expand All @@ -844,6 +835,19 @@
* <tr>
* <td>Fletcher32</td>
* <td>Fletcher32 checksum for error-detection.</td>
* </tr>
* <tr>
* <th>Optional Built-in Filter</th>
* <th>Description</th>
* </tr>
* <tr>
* <td>gzip compression</td>
* <td>Data compression using zlib.</td>
* </tr>
* <tr>
* <td>szip compression</td>
* <td>Data compression using the szip library. The HDF Group now uses the libaec library for the szip
filter.</td>
* </tr>
* </table>
*
Expand All @@ -861,6 +865,44 @@
* \li @see @ref subsubsec_dataset_filters_nbit
* \li @see @ref subsubsec_dataset_filters_scale
*
* \subsubsection subsubsec_dataset_transfer_dyn_filter Data Pipeline Dynamically Loaded Filters
* While the HDF5 “internal” compression methods work reasonably well on users’
* datasets, there are certain drawbacks to this implementation. First, the “internal” compression
* methods may not provide the optimal compression ratio, as do some newly developed or specialized
* compression methods. Secondly, if a data provider wants to use a “non-internal” compression for
* storing the data in HDF5, they have to write a filter function that uses the new compression method
* and then register it with the library. Data consumers of such HDF5 files will need to have the new filter
* function and use it with their applications to read the data, or they will need a modified version of the
* HDF5 Library that has the new filter as a part of the library.
*
* If a user of such data does not have a modified HDF5 Library installed on his system, command-line tools
* such as h5dump or h5ls will not be able to display the compressed data. Furthermore, it would be
* practically impossible to determine the compression method used, making the data stored in HDF5
* useless.
*
* It is clear that the internal HDF5 filter mechanism, while extensible, does not work well with third-party
* filters. It would be a maintenance nightmare to keep adding and supporting new compression methods
* in HDF5. For any set of HDF5 “internal” filters, there always will be data with which the “internal”
filters
* will not achieve the optimal performance needed to address data I/O and storage problems. Thus the
* internal HDF5 filter mechanism is enhanced to address the issues discussed above.
*
* We have a feature of HDF5 called “dynamically loaded filters in HDF5.” This feature
* makes the HDF5 third-party filters available to an application at runtime. The third-party HDF5 filter
* function has to be a part of the HDF5 filter plugin installed on the system as a shared library or DLL.
*
* To use a third-party filter an HDF5 application should call the #H5Pset_filter function when setting the
* filter pipeline for a dataset creation property. The HDF5 Library will register the filter with the library
* and the filter will be applied when data is written to the file.
*
* When an application reads data compressed with a third-party HDF5 filter, the HDF5 Library will search
* for the required filter plugin, register the filter with the library (if the filter function is not
registered) and
* apply it to the data on the read operation.
*
* For more information,
* \li @see @ref sec_filter_plugins
*
* \subsubsection subsubsec_dataset_transfer_drive File Drivers
* I/O is performed by the HDF5 virtual file layer. The file driver interface writes and reads blocks
* of data; each driver module implements the interface using different I/O mechanisms. The table
Expand Down Expand Up @@ -2685,8 +2727,25 @@ allocated if necessary.
* and minimum values, and they will get a much larger minimum-bits (poor compression)
* </li></ol>
*
* \subsubsection subsubsec_dataset_filters_szip Using the Szip Filter
* See The HDF Group website for further information regarding the Szip filter.
* \subsubsection subsubsec_dataset_filters_szip Using the SZip Filter
* See The HDF Group website for further information regarding the SZip filter.
*
* \subsubsection subsubsec_dataset_filters_dyn Using Dynamically-Loadable Filters
* \see \ref sec_filter_plugins for further information regarding the dynamically-loadable filters.
*
* HDF has a filter plugin repository of useful third-party plugins that can used
* <table>
* <tr><th>Filter</th><th>SetFilter Params</th></tr>
* <tr><td>BLOSC</td><td>UD=32001,0,0</td></tr>
* <tr><td>BSHUF</td><td>UD=32004,0,0</td></tr>
* <tr><td>BZIP2</td><td>UD=307,0,1,9</td></tr>
* <tr><td>JPEG</td><td>UD=32019,0,4,q,c,r,t</td></tr>
* <tr><td>LZ4 </td><td>UD=32004,0,1,3</td></tr>
* <tr><td>LZF</td><td>UD=32000,1,3,0,0,0</td></tr>
* <tr><td>SZ</td><td>UD=32017,1,5,2,7,20,40,0</td></tr>
* <tr><td>ZFP</td><td>UD=32013,1,0,0</td></tr>
* <tr><td>ZSTD</td><td>UD=32015,0,0</td></tr>
* </table>
*
* Previous Chapter \ref sec_group - Next Chapter \ref sec_datatype
*
Expand Down
Loading
Loading