Lossy compression (sandialabs#464)

* EXODUS: Add support for lossy compression via netCDF quantize method * IOSS: Add support for lossy compression specifying NSD for netCDF quantize method * EXODUS: Update version date * EXODUS: Add some error checking on nsd values * EXODUS: Check if Quantization supported; include in config output * EXODUS: Cleaning up new compression support * EXODUS: Fix error handling code * EXODUS: Better error checking on compression * IOSS: Support for zstd compression * IOSS: Handle building with no netCDF-4 capability * IOSS: Change gold files to netcdf-3 so work without hdf5-supported netCDF * CI: See if can figure out intel build failure * CI: Fix hdf5 version warning message * CI: Turn on plugins; support new syntax * EXODUS: Only flush stderr if written to Signed-off-by: Greg Sjaardema <[email protected]> * Tweak compression additions * CI: Fix intel build script syntax error * EXODUS: Support new netCDF with changed _FillValue * IOSS: Fix so latest fmt version works * IOSS: Catalyst API 2 (sandialabs#463) * IOSS: Catalyst API 2 Added support to database for using my_processor and processor_count IOSS properties when operating in serial parallel mode and reading Conduit files. * IOSS: Catalyst API 2 Changed Conduit serial parallel test to use files from the one MPI rank can.ex2 test. --------- Co-authored-by: Greg Sjaardema <[email protected]> * Fix export symbols (sandialabs#465) * IOSS: Catalyst API 2 (sandialabs#466) Create the cell_ids and cell_node_ids fields for a StructuredBlock when asked for by a reading application, if these fields are not already stored in the Conduit data. Uses the get_cell_ids() and get_cell_node_ids() methods on StructuredBlock. * CI: Remove hdf5-1.8 testing * IOSS: Catalyst API 2 (sandialabs#463) * IOSS: Catalyst API 2 Added support to database for using my_processor and processor_count IOSS properties when operating in serial parallel mode and reading Conduit files. * IOSS: Catalyst API 2 Changed Conduit serial parallel test to use files from the one MPI rank can.ex2 test. --------- Co-authored-by: Greg Sjaardema <[email protected]> * NEM_SPREAD: Add missing include file * CI: Enable netCDF quantize option * CI: Support newest netCDF; fmt * Enable new compression options * EXODUS: Fix non-hdf5 build * EXODUS: Do not commit the kluge to repository * IOSS: See if can fix spack ci build * CI: Spack build should not run on PR * CI: Spack build on master only [ci skip] * CI: Fix spack workflow syntax [ci skip] * IOSS: Add progress output to chain/face generation * IOSS: Reduce Face memory * EXODUS: Better control over exodus verbosity * EXODUS:, IOSS: Add bzip2 compression support * Further testing of non-lossy compression with plugins * CI: Update cmake-sems to work with current sems * Default netcdf should be 4.9.2 Signed-off-by: Greg Sjaardema <[email protected]> * Clean up copyright dates Signed-off-by: Greg Sjaardema <[email protected]> * Clean up copyright dates Signed-off-by: Greg Sjaardema <[email protected]> * accidentally added to pr; removed Signed-off-by: Greg Sjaardema <[email protected]> --------- Signed-off-by: Greg Sjaardema <[email protected]> Signed-off-by: Greg Sjaardema <[email protected]> Co-authored-by: Greg Sjaardema <[email protected]> Co-authored-by: tjotaha <[email protected]> Co-authored-by: Spiros Tsalikis <[email protected]>
tokusanya · Sep 11, 2024 · 963c159 · 963c159
1 parent 2fa97fc
commit 963c159
Show file tree

Hide file tree

Showing 34 changed files with 762 additions and 189 deletions.
diff --git a/TPL/hdf5/runcmake.sh b/TPL/hdf5/runcmake.sh
@@ -61,6 +61,7 @@ cmake .. -DCMAKE_C_COMPILER:FILEPATH=${CC} \
          -DCMAKE_INSTALL_PREFIX=${INSTALL_PATH} \
 	 -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
 	 -DDEFAULT_API_VERSION=V18 \
+	 -DH5_DEFAULT_PLUGINDIR:PATH=${INSTALL_PATH}/lib/hdf5/lib/plugin \
 	 -DHDF5_ENABLE_PARALLEL:BOOL=${MPI} \
 	 -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON \
 	 -DHDF5_BUILD_CPP_LIB:BOOL=${H5CPP} \

diff --git a/TPL/netcdf/runcmake.sh b/TPL/netcdf/runcmake.sh
@@ -15,6 +15,8 @@ else
   BUILD_TYPE="RELEASE"
 fi
 
+PREFIX="NETCDF_"
+
 SHARED="${SHARED:-YES}"
 if [[ "$SHARED" == "ON" || "$SHARED" == "YES" ]]
 then
@@ -31,9 +33,10 @@ fi
 
 if [[ "$HDF5" == "ON" || "$HDF5" == "YES" ]]
 then
-    HDF5_INFO="-DHDF5_ROOT:PATH=${INSTALL_PATH} -DHDF5_DIR:PATH=${INSTALL_PATH} -DENABLE_NETCDF4:BOOL=ON"
+    export HDF5_PLUGIN_PATH=${INSTALL_PATH}/lib/hdf5/lib/plugin
+    HDF5_INFO="-DHDF5_ROOT:PATH=${INSTALL_PATH} -DHDF5_DIR:PATH=${INSTALL_PATH} -D${PREFIX}ENABLE_NETCDF4:BOOL=ON -D${PREFIX}PLUGIN_INSTALL_DIR=YES"
 else
-    HDF5_INFO="-DENABLE_HDF5=OFF -DENABLE_NETCDF4:BOOL=OFF"
+    HDF5_INFO="-DENABLE_HDF5=OFF -D${PREFIX}ENABLE_NETCDF4:BOOL=OFF"
 fi
 
 NEEDS_ZLIB="${NEEDS_ZLIB:-NO}"
@@ -46,10 +49,13 @@ NEEDS_SZIP="${NEEDS_SZIP:-NO}"
 if [ "$NEEDS_SZIP" == "YES" ]
 then
    LOCAL_SZIP="-DSZIP_INCLUDE_DIR:PATH=${INSTALL_PATH}/include -DSZIP_LIBRARY:FILEPATH=${INSTALL_PATH}/lib/libsz.${LD_EXT}"
+else
+   LOCAL_SZIP="-D${PREFIX}ENABLE_FILTER_SZIP=NO"
 fi
 
 . ${ACCESS}/TPL/compiler.sh
 
+
 # If using an XLF compiler on an IBM system, may need to add the following:
 # -DCMAKE_Fortran_FLAGS="-qfixed=72" \
 # -DCMAKE_EXE_LINKER_FLAGS:STRING="-lxl -lxlopt"
@@ -62,20 +68,24 @@ cmake .. -DCMAKE_C_COMPILER:FILEPATH=${CC} \
          -DCMAKE_INSTALL_PREFIX=${INSTALL_PATH} \
          -DCMAKE_INSTALL_LIBDIR:PATH=lib \
 	 -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-         -DENABLE_PNETCDF:BOOL=${MPI} \
-         -DENABLE_CDF5=ON \
-         -DENABLE_MMAP:BOOL=ON \
-         -DENABLE_DAP:BOOL=OFF \
-         -DENABLE_BYTERANGE:BOOL=OFF \
-	 -DENABLE_NCZARR:BOOL=OFF \
-         -DENABLE_V2_API:BOOL=OFF \
-	 -DENABLE_FILTER_TESTING:BOOL=OFF \
-	 -DENABLE_TESTS:BOOL=OFF \
+	 -DCMAKE_PREFIX_PATH=${INSTALL_PATH} \
+         -D${PREFIX}ENABLE_PNETCDF:BOOL=${MPI} \
+         -D${PREFIX}ENABLE_CDF5=ON \
+         -D${PREFIX}ENABLE_MMAP:BOOL=ON \
+         -D${PREFIX}ENABLE_DAP:BOOL=OFF \
+         -D${PREFIX}ENABLE_BYTERANGE:BOOL=OFF \
+	 -D${PREFIX}ENABLE_NCZARR_FILTERS:BOOL=OFF \
+	 -D${PREFIX}ENABLE_NCZARR:BOOL=OFF \
+         -D${PREFIX}ENABLE_V2_API:BOOL=OFF \
+	 -D${PREFIX}ENABLE_FILTER_TESTING:BOOL=OFF \
+	 -DENABLE_PLUGIN_INSTALL:BOOL=ON \
+	 -D${PREFIX}ENABLE_TESTS:BOOL=OFF \
+         -D${PREFIX}ENABLE_QUANTIZE:BOOL=ON \
          ${LOCAL_ZLIB} \
          ${LOCAL_SZIP} \
          ${EXTRA_DEPS} \
 	 ${HDF5_INFO} \
-         -DENABLE_CONVERSION_WARNINGS:BOOL=OFF
+         -D${PREFIX}ENABLE_CONVERSION_WARNINGS:BOOL=OFF
 
 echo ""
 echo "         MPI: ${MPI}"

diff --git a/install-tpl.sh b/install-tpl.sh
@@ -425,7 +425,7 @@ then
 	elif [ "${H5VERSION}" == "develop" ]; then
             hdf_version="develop"
 	else
-            echo 1>&2 ${txtred}Invalid HDF5 version specified: ${H5VERSION}.  Must be one of V110, V112, or V114. exiting.${txtrst}
+            echo 1>&2 ${txtred}Invalid HDF5 version specified: ${H5VERSION}.  Must be one of V110, V112, or V114 [default]. exiting.${txtrst}
             exit 1
 	fi
 
@@ -539,7 +539,7 @@ then
 #   net_version="v4.9.1"
     net_version="v4.9.2"
 #   net_version="v4.8.1"
-#   net_version="master"
+#   net_version="main"
 
     echo "${txtgrn}+++ NetCDF ${net_version} ${txtrst}"
     cd $ACCESS || exit
@@ -551,6 +551,11 @@ then
         git clone --depth 1 --branch ${net_version} https://github.com/Unidata/netcdf-c netcdf-c
     fi
 
+    if [ net_version == "main" ]
+    then
+	PREFIX="NETCDF_"
+    fi
+
     if [ "$BUILD" == "YES" ]
     then
         echo "${txtgrn}+++ Configuring, Building, and Installing...${txtrst}"
@@ -562,7 +567,7 @@ then
 	then
            export HDF5_PLUGIN_PATH=${INSTALL_PATH}/lib/hdf5/lib/plugin
 	fi
-        CRAY=${CRAY} SHARED=${SHARED} DEBUG=${DEBUG} HDF5=${HDF5} NEEDS_ZLIB=${NEEDS_ZLIB} MPI=${MPI} bash -x ../../runcmake.sh
+        PREFIX=${PREFIX} CRAY=${CRAY} SHARED=${SHARED} DEBUG=${DEBUG} HDF5=${HDF5} NEEDS_ZLIB=${NEEDS_ZLIB} MPI=${MPI} bash -x ../../runcmake.sh
         if [[ $? != 0 ]]
         then
             echo 1>&2 ${txtred}couldn\'t configure cmake for NetCDF. exiting.${txtrst}

diff --git a/packages/seacas/applications/conjoin/CJ_ExodusFile.C b/packages/seacas/applications/conjoin/CJ_ExodusFile.C
@@ -1,4 +1,4 @@
-// Copyright(C) 1999-2021, 2023 National Technology & Engineering Solutions
+// Copyright(C) 1999-2021, 2023, 2024 National Technology & Engineering Solutions
 // of Sandia, LLC (NTESS).  Under the terms of Contract DE-NA0003525 with
 // NTESS, the U.S. Government retains certain rights in this software.
 //
@@ -202,6 +202,16 @@ bool Excn::ExodusFile::create_output(const SystemInterface &si)
     else if (si.zlib()) {
       ex_set_option(outputId_, EX_OPT_COMPRESSION_TYPE, EX_COMPRESS_ZLIB);
     }
+    else if (si.zstd()) {
+      ex_set_option(outputId_, EX_OPT_COMPRESSION_TYPE, EX_COMPRESS_ZSTD);
+    }
+    else if (si.bz2()) {
+      ex_set_option(outputId_, EX_OPT_COMPRESSION_TYPE, EX_COMPRESS_BZ2);
+    }
+
+    if (si.quantize()) {
+      ex_set_option(outputId_, EX_OPT_QUANTIZE_NSD, si.quantize_nsd());
+    }
   }
 
   fmt::print("IO Word size is {} bytes.\n", ioWordSize_);

diff --git a/packages/seacas/applications/conjoin/CJ_SystemInterface.C b/packages/seacas/applications/conjoin/CJ_SystemInterface.C
@@ -65,18 +65,27 @@ void Excn::SystemInterface::enroll_options()
   options_.enroll("64-bit", GetLongOption::NoValue,
                   "True if forcing the use of 64-bit integers for the output file", nullptr);
 
-  options_.enroll(
-      "zlib", GetLongOption::NoValue,
-      "Use the Zlib / libz compression method if compression is enabled (default) [exodus only].",
-      nullptr);
+  options_.enroll("zlib", GetLongOption::NoValue,
+                  "Use the Zlib / libz compression method if compression is enabled (default) "
+                  "[exodus only, enables netcdf-4].",
+                  nullptr);
 
   options_.enroll("szip", GetLongOption::NoValue,
                   "Use SZip compression. [exodus only, enables netcdf-4]", nullptr);
+  options_.enroll("zstd", GetLongOption::NoValue,
+                  "Use Zstd compression. [exodus only, enables netcdf-4, experimental]", nullptr);
+  options_.enroll("bzip2", GetLongOption::NoValue,
+                  "Use Bzip2 compression. [exodus only, enables netcdf-4, experimental]", nullptr);
+
+  options_.enroll("compress", GetLongOption::MandatoryValue,
+                  "Specify the compression level to be used.  Values depend on algorithm:\n"
+                  "\t\tzlib/bzip2:  0..9\t\tszip:  even, 4..32\t\tzstd:  -131072..22",
+                  nullptr);
 
-  options_.enroll(
-      "compress", GetLongOption::MandatoryValue,
-      "Specify the hdf5 (netcdf4) compression level [0..9] to be used on the output file.", nullptr,
-      nullptr, true);
+  options_.enroll("quantize_nsd", GetLongOption::MandatoryValue,
+                  "Use the lossy quantize compression method.  Value specifies number of digits to "
+                  "retain (1..15) [exodus only]",
+                  nullptr, nullptr, true);
 
   options_.enroll("sort_times", GetLongOption::NoValue,
                   "Sort the input files on the minimum timestep time in the file.\n"
@@ -242,17 +251,61 @@ bool Excn::SystemInterface::parse_options(int argc, char **argv)
   omitNodesets_      = options_.retrieve("omit_nodesets") != nullptr;
   omitSidesets_      = options_.retrieve("omit_sidesets") != nullptr;
 
-  if (options_.retrieve("szip") != nullptr) {
-    szip_ = true;
-    zlib_ = false;
-  }
   zlib_ = (options_.retrieve("zlib") != nullptr);
+  szip_ = (options_.retrieve("szip") != nullptr);
+  zstd_ = (options_.retrieve("zstd") != nullptr);
+  bz2_  = (options_.retrieve("bzip2") != nullptr);
+
+  if (szip_ + zlib_ + zstd_ + bz2_ > 1) {
+    fmt::print(stderr,
+               "ERROR: Only one of 'szip' or 'zlib' or 'zstd' or 'bzip2' can be specified.\n");
+  }
+
+  {
+    const char *temp = options_.retrieve("compress");
+    if (temp != nullptr) {
+      compressionLevel_ = std::strtol(temp, nullptr, 10);
+      if (!szip_ && !zlib_ && !zstd_ && !bz2_) {
+        zlib_ = true;
+      }
 
-  if (szip_ && zlib_) {
-    fmt::print(stderr, "ERROR: Only one of 'szip' or 'zlib' can be specified.\n");
+      if (zlib_ || bz2_) {
+        if (compressionLevel_ < 0 || compressionLevel_ > 9) {
+          fmt::print(stderr,
+                     "ERROR: Bad compression level {}, valid value is between 0 and 9 inclusive "
+                     "for gzip/zlib/bzip2 compression.\n",
+                     compressionLevel_);
+          return false;
+        }
+      }
+      else if (szip_) {
+        if (compressionLevel_ % 2 != 0) {
+          fmt::print(
+              stderr,
+              "ERROR: Bad compression level {}. Must be an even value for szip compression.\n",
+              compressionLevel_);
+          return false;
+        }
+        if (compressionLevel_ < 4 || compressionLevel_ > 32) {
+          fmt::print(stderr,
+                     "ERROR: Bad compression level {}, valid value is between 4 and 32 inclusive "
+                     "for szip compression.\n",
+                     compressionLevel_);
+          return false;
+        }
+      }
+    }
   }
 
-  compressionLevel_ = options_.get_option_value("compress", compressionLevel_);
+  {
+    const char *temp = options_.retrieve("quantize_nsd");
+    if (temp != nullptr) {
+      quantizeNSD_ = std::strtol(temp, nullptr, 10);
+      if (!szip_ && !zlib_ && !zstd_ && !bz2_) {
+        zlib_ = true;
+      }
+    }
+  }
 
   if (options_.retrieve("copyright") != nullptr) {
     fmt::print("{}", copyright("2009-2021"));

diff --git a/packages/seacas/applications/conjoin/CJ_SystemInterface.h b/packages/seacas/applications/conjoin/CJ_SystemInterface.h
@@ -17,11 +17,16 @@ namespace Excn {
     SystemInterface();
     bool parse_options(int argc, char **argv);
 
-    int  debug() const { return debugLevel_; }
-    int  screen_width() const { return screenWidth_; }
+    int debug() const { return debugLevel_; }
+    int screen_width() const { return screenWidth_; }
+
+    int  quantize_nsd() const { return quantizeNSD_; }
     int  compress_data() const { return compressionLevel_; }
     bool zlib() const { return zlib_; }
     bool szip() const { return szip_; }
+    bool zstd() const { return zstd_; }
+    bool bz2() const { return bz2_; }
+    bool quantize() const { return quantizeNSD_ > 0; }
 
     bool omit_nodesets() const { return omitNodesets_; }
     bool omit_sidesets() const { return omitSidesets_; }
@@ -60,8 +65,11 @@ namespace Excn {
     int  debugLevel_{0};
     int  screenWidth_{0};
     int  compressionLevel_{0};
-    bool zlib_{true};
+    int  quantizeNSD_{0};
+    bool zlib_{false};
     bool szip_{false};
+    bool zstd_{false};
+    bool bz2_{false};
     bool omitNodesets_{false};
     bool omitSidesets_{false};
     bool ints64Bit_{false};

diff --git a/packages/seacas/applications/cpup/CP_SystemInterface.C b/packages/seacas/applications/cpup/CP_SystemInterface.C
@@ -124,19 +124,6 @@ void Cpup::SystemInterface::enroll_options()
       "large subcycle runs.",
       nullptr);
 
-  options_.enroll(
-      "zlib", GetLongOption::NoValue,
-      "Use the Zlib / libz compression method if compression is enabled (default) [exodus only].",
-      nullptr);
-
-  options_.enroll("szip", GetLongOption::NoValue,
-                  "Use SZip compression. [exodus only, enables netcdf-4]", nullptr);
-
-  options_.enroll("compress_data", GetLongOption::MandatoryValue,
-                  "The output database will be written using compression (netcdf-4 mode only).\n"
-                  "\t\tValue ranges from 0..9 for zlib/gzip or even values 4..32 for szip.",
-                  nullptr, nullptr, true);
-
   options_.enroll("append", GetLongOption::NoValue,
                   "Append to database instead of opening a new database.\n"
                   "\t\tTimestep transfer will start after last timestep on database",
@@ -301,18 +288,7 @@ bool Cpup::SystemInterface::parse_options(int argc, char **argv)
 
   append_ = options_.retrieve("append") != nullptr;
 
-  if (options_.retrieve("szip") != nullptr) {
-    szip_ = true;
-    zlib_ = false;
-  }
-  zlib_ = (options_.retrieve("zlib") != nullptr);
-
-  if (szip_ && zlib_) {
-    fmt::print(stderr, "ERROR: Only one of 'szip' or 'zlib' can be specified.\n");
-  }
-
   append_ = options_.retrieve("append") != nullptr;
-  compressData_ = options_.get_option_value("compress_data", compressData_);
   //subcycle_        = options_.get_option_value("subcycle", subcycle_);
   cycle_           = options_.get_option_value("cycle", cycle_);
   subcycleJoin_    = options_.retrieve("join_subcycles") != nullptr;

diff --git a/packages/seacas/applications/cpup/CP_SystemInterface.h b/packages/seacas/applications/cpup/CP_SystemInterface.h
@@ -65,9 +65,6 @@ namespace Cpup {
     bool map_element_ids() const { return mapIds_; }
     bool omit_nodesets() const { return omitNodesets_; }
     bool omit_sidesets() const { return omitSidesets_; }
-    int  compress_data() const { return compressData_; }
-    bool zlib() const { return zlib_; }
-    bool szip() const { return szip_; }
     bool subcycle_join() const { return subcycleJoin_; }
     bool keep_temporary() const { return keepTemporary_; }
     bool verify_valid_file() const { return verifyValidFile_; }
@@ -132,10 +129,7 @@ namespace Cpup {
     int  partCount_{-1};
     int  subcycle_{-1};
     int  cycle_{-1};
-    int  compressData_{0};
     int  maxOpenFiles_{0};
-    bool zlib_{true};
-    bool szip_{false};
     bool sumSharedNodes_{false};
     bool addProcessorIdField_{false};
     bool addProcessorIdMap_{false};