Skip to content

Commit

Permalink
Incorporate remaining parts of the filters.html file (HDFGroup#4966)
Browse files Browse the repository at this point in the history
  • Loading branch information
@byrnHDF @brtnfld
byrnHDF authored and brtnfld committed Oct 31, 2024
1 parent a23f20b commit b1576ab
Show file tree
Hide file tree
Showing 4 changed files with 125 additions and 21 deletions.
10 changes: 0 additions & 10 deletions doxygen/dox/ExamplesAPI.dox
View file
Original file line number Diff line number Diff line change
Expand Up @@ -800,16 +800,6 @@ FORTRAN
<td><a href="https://\SRCURL/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lzf.ddl">h5ex_d_lzf.ddl</a></td>
</tr>
<tr>
<td>Read / Write Dataset using MAFISC Compression</td>
<td>
<a href="https://\SRCURL/HDF5Examples/C/H5FLT/h5ex_d_mafisc.c">C</a>
FORTRAN Java JavaObj MATLAB PyHigh PyLow
</td>
<td><a href="https://\SRCURL/HDF5Examples/C/H5FLT/tfiles/h5ex_d_mafisc.h5">h5ex_d_mafisc.h5</a></td>
<td><a href="https://\SRCURL/HDF5Examples/C/H5FLT/tfiles/h5ex_d_mafisc.tst">h5ex_d_mafisc.tst</a></td>
<td><a href="https://\SRCURL/HDF5Examples/C/H5FLT/tfiles/h5ex_d_mafisc.ddl">h5ex_d_mafisc.ddl</a></td>
</tr>
<tr>
<td>Read / Write Dataset using ZFP Compression</td>
<td>
<a href="https://\SRCURL/HDF5Examples/C/H5FLT/h5ex_d_zfp.c">C</a>
Expand Down
3 changes: 2 additions & 1 deletion src/H5Dmodule.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1612,7 +1612,8 @@ allocated if necessary.
* care must be taken to assure that all the external files are accessible in the new location.
*
* \subsection subsec_dataset_filters Using HDF5 Filters
* This section describes in detail how to use the n-bit, scale-offset filters and szip filters.
* This section describes in detail how to use the n-bit, scale-offset filters and szip filters. For
* details on the how filters are used in the read / write of data, see /def subsubsec_dataset_transfer_pipe.
*
* \subsubsection subsubsec_dataset_filters_nbit Using the N‐bit Filter
* N-bit data has n significant bits, where n may not correspond to a precise number of bytes. On
Expand Down
95 changes: 85 additions & 10 deletions src/H5PLmodule.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,23 @@
* available to the application. For example, if the application intends to apply the HDF5 bzip2 compression
* filter that was registered with The HDF Group and has an identification number 307
* (<a href="https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md">Registered
* Filters</a>) then the application would follow the steps as outlined below: \code dcpl = H5Pcreate
* (H5P_DATASET_CREATE); status = H5Pset_filter (dcpl, (H5Z_filter_t)307, H5Z_FLAG_MANDATORY, (size_t)6,
* cd_values); dset = H5Dcreate (file, DATASET, H5T_STD_I32LE, space, H5P_DEFAULT, dcpl, status = H5Dwrite
* (dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata[0]); \endcode
* Filters</a>) then the application would follow the steps as outlined below:
* \code
* dcpl = H5Pcreate (H5P_DATASET_CREATE);
* status = H5Pset_filter (dcpl, (H5Z_filter_t)307, H5Z_FLAG_MANDATORY, (size_t)6, cd_values);
* dset = H5Dcreate (file, DATASET, H5T_STD_I32LE, space, H5P_DEFAULT, dcpl, H5P_DEFAULT);
* status = H5Dwrite (dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata[0]);
* \endcode
*
* \subsubsection subsubsec_filter_plugins_model_read Reading Data with an Applied Third-party Filter
* An application does not need to do anything special to read the data with a third-party filter applied. For
* example, if one wants to read data written in the previous example, the following regular steps should be
* taken: \code file = H5Fopen (FILE, H5F_ACC_RDONLY, H5P_DEFAULT); dset = H5Dopen (file, DATASET,
* H5P_DEFAULT); H5Dread (dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata[0]); \endcode
* taken:
* \code
* file = H5Fopen (FILE, H5F_ACC_RDONLY, H5P_DEFAULT);
* dset = H5Dopen (file, DATASET, H5P_DEFAULT);
* H5Dread (dset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata[0]);
* \endcode
*
* The command-line utility h5dump, for example, will read and display the data as shown:
* \code
Expand Down Expand Up @@ -161,10 +168,78 @@
* plugin on the system.
*
* \subsubsection subsubsec_filter_plugins_prog_write Writing a Filter Function
* The HDF5 filter function for the dynamically loaded filter feature should be written as any custom filter
* described in <a href="https://\DOXURL/_f_i_l_t_e_r.html">Custom Filters</a>. See the
* “Example” section, section 5, of that document to get an idea of the simple filter function, and see the
* example of the more sophisticated HDF5 bzip2 filter function in the “Building an HDF5 bzip2 Plugin Example”
* The HDF5 filter function for the dynamically loaded filter feature should be written as a custom filter.
* This example shows how to define and register a simple filter
* that adds a checksum capability to the data stream.
*
* The function that acts as the filter always returns zero (failure) if the <code>md5()</code> function was
* not detected at configuration time (left as an exercise for the reader). Otherwise the function is broken
* down to an input and output half. The output half calculates a checksum, increases the size of the output
* buffer if necessary, and appends the checksum to the end of the buffer. The input half calculates the
* checksum on the first part of the buffer and compares it to the checksum already stored at the end of the
* buffer. If the two differ then zero (failure) is returned, otherwise the buffer size is reduced to exclude
* the checksum. /code size_t md5_filter(unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[],
* size_t nbytes, size_t *buf_size, void **buf)
* {
* \c \#ifdef HAVE_MD5
* unsigned char cksum[16];
*
* if (flags & H5Z_REVERSE) {
* // Input
* assert(nbytes >= 16);
* md5(nbytes-16, *buf, cksum);
* // Compare
* if (memcmp(cksum, (char*)(*buf)+ nbytes- 16, 16)) {
* return 0; // fail
* }
* // Strip off checksum
* return nbytes - 16;
* }
* else {
* // Output
* md5(nbytes, *buf, cksum);
* // Increase buffer size if necessary
* if (nbytes + 16 > *buf_size) {
* *buf_size = nbytes + 16;
* *buf = realloc(*buf, *buf_size);
* }
* // Append checksum
* memcpy((char*)(*buf)+nbytes, cksum, 16);
* return nbytes+16;
* }
* \c \#else
* return 0; // fail
* \c \#endif
* }
* /endcode
*
* Once the filter function is defined it must be registered so
* the HDF5 library knows about it. Since we're testing this
* filter we choose one of the #H5Z_filter_t numbers
* from the reserved range. We'll randomly choose 305.
*
* /code
* \c \#define FILTER_MD5 305
* herr_t status = H5Zregister(FILTER_MD5, "md5 checksum", md5_filter);
* /endcode
*
* Now we can use the filter in a pipeline. We could have added
* the filter to the pipeline before defining or registering the
* filter as long as the filter was defined and registered by time
* we tried to use it (if the filter is marked as optional then we
* could have used it without defining it and the library would
* have automatically removed it from the pipeline for each chunk
* written before the filter was defined and registered).
*
* /code
* hid_t dcpl = H5Pcreate(H5P_DATASET_CREATE);
* hsize_t chunk_size[3] = {10,10,10};
* H5Pset_chunk(dcpl, 3, chunk_size);
* H5Pset_filter(dcpl, FILTER_MD5, 0, 0, NULL);
* hid_t dset = H5Dcreate(file, "dset", H5T_NATIVE_DOUBLE, space, dcpl);
* /endcode
*
* See the example of a more sophisticated HDF5 bzip2 filter function in the /ref subsec_filter_plugins_build
* section. The HDF5 bzip2 filter function is also available for download from <a
* href="https://github.com/HDFGroup/hdf5_plugins">Filter Plugin Repository</a>.
*
Expand Down
38 changes: 38 additions & 0 deletions src/H5Zmodule.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@
* shuffling algorithm) and error checking (Fletcher32 checksum). For further
* flexibility, the library allows a user application to extend the pipeline
* through the creation and registration of customized filters.
* See \ref sec_filter_plugins
*
* The flexibility of the filter pipeline implementation enables the definition
* of additional filters by a user application. A filter
Expand All @@ -83,6 +84,43 @@
* the difficulty of implementing random access for partial I/O. Compact dataset
* filters are not supported because they would not produce significant results.
*
* HDF5 allows chunked data to pass through user-defined filters
* on the way to or from disk. The filters operate on chunks of an
* #H5D_CHUNKED dataset can be arranged in a pipeline
* so output of one filter becomes the input of the next filter.
*
* Each filter has a two-byte identification number (type
* #H5Z_filter_t) allocated by The HDF Group and can also be
* passed application-defined integer resources to control its
* behavior. Each filter also has an optional ASCII comment
* string.
*
* <table>
* <tr>
* <th>Values for <code>#H5Z_filter_t</code></th><th>Description</th>
* </tr>
* <tr>
* <td><code>0-255</code></td>
* <td>These values are reserved for filters predefined and
* registered by the HDF5 library and of use to the general
* public.</td>
* </tr>
* <tr>
* <td><code>256-511</code></td>
* <td>Filter numbers in this range are used for testing only
* and can be used temporarily by any organization. No
* attempt is made to resolve numbering conflicts since all
* definitions are by nature temporary.</td>
* </tr>
* <tr>
* <td><code>512-65535</code></td>
* <td>Reserved for future assignment. Please contact the
* <a href="mailto:[email protected]">HDF5 development team</a>
* to reserve a value or range of values for
* use by your filters.</td>
* </tr>
* </table>
*
* Filter identifiers for the filters distributed with the HDF5
* Library are as follows:
* <table>
Expand Down

0 comments on commit b1576ab

Please sign in to comment.