Fix append mode double attributes

CHANGELOG.rst

@@ -25,6 +25,7 @@ Bug Fixes
 
 Other
 """""
+- Catch2: updated to 2.13.9 #1299
 
 
 0.14.3

CMakeLists.txt

@@ -536,9 +536,9 @@ if(openPMD_BUILD_TESTING)
         target_include_directories(openPMD::thirdparty::Catch2 SYSTEM INTERFACE
             $<BUILD_INTERFACE:${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/catch2/include>
         )
-        message(STATUS "Catch2: Using INTERNAL version '2.13.4'")
+        message(STATUS "Catch2: Using INTERNAL version '2.13.9'")
     else()
-        find_package(Catch2 2.13.4 REQUIRED CONFIG)
+        find_package(Catch2 2.13.9 REQUIRED CONFIG)
         target_link_libraries(openPMD::thirdparty::Catch2
             INTERFACE Catch2::Catch2)
         message(STATUS "Catch2: Found version '${Catch2_VERSION}'")

NEWS.rst

@@ -12,6 +12,7 @@ Building openPMD-api now requires a compiler that supports C++17 or newer.
 Python 3.10 is now supported.
 openPMD-api now depends on `toml11 <https://github.com/ToruNiina/toml11>`__ 3.7.1+.
 pybind11 2.9.1 is now the minimally supported version for Python support.
+Catch2 2.13.9 is now the minimally supported version for tests.
 
 The following backend-specific members of the ``Dataset`` class have been removed: ``Dataset::setChunkSize()``, ``Dataset::setCompression()``, ``Dataset::setCustomTransform()``, ``Dataset::chunkSize``, ``Dataset::compression``, ``Dataset::transform``.
 They are replaced by backend-specific options in the JSON-based backend configuration.

README.md

@@ -103,7 +103,7 @@ Required:
 * C++17 capable compiler, e.g., g++ 7+, clang 7+, MSVC 19.15+, icpc 19+, icpx
 
 Shipped internally in `share/openPMD/thirdParty/`:
-* [Catch2](https://github.com/catchorg/Catch2) 2.13.4+ ([BSL-1.0](https://github.com/catchorg/Catch2/blob/master/LICENSE.txt))
+* [Catch2](https://github.com/catchorg/Catch2) 2.13.9+ ([BSL-1.0](https://github.com/catchorg/Catch2/blob/master/LICENSE.txt))
 * [pybind11](https://github.com/pybind/pybind11) 2.9.1+ ([new BSD](https://github.com/pybind/pybind11/blob/master/LICENSE))
 * [NLohmann-JSON](https://github.com/nlohmann/json) 3.9.1+ ([MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT))
 * [toml11](https://github.com/ToruNiina/toml11) 3.7.1+ ([MIT](https://github.com/ToruNiina/toml11/blob/master/LICENSE))
@@ -271,7 +271,7 @@ The following options allow to switch to external installs:
 
 | CMake Option                    | Values     | Library       | Version |
 |---------------------------------|------------|---------------|---------|
-| `openPMD_USE_INTERNAL_CATCH`    | **ON**/OFF | Catch2        | 2.13.4+ |
+| `openPMD_USE_INTERNAL_CATCH`    | **ON**/OFF | Catch2        | 2.13.9+ |
 | `openPMD_USE_INTERNAL_PYBIND11` | **ON**/OFF | pybind11      |  2.9.1+ |
 | `openPMD_USE_INTERNAL_JSON`     | **ON**/OFF | NLohmann-JSON |  3.9.1+ |
 | `openPMD_USE_INTERNAL_TOML11`   | **ON**/OFF | toml11        |  3.7.1+ |

docs/source/backends/adios2.rst

@@ -10,12 +10,43 @@ For further information, check out the :ref:`installation guide <install>`,
 :ref:`build dependencies <development-dependencies>` and the :ref:`build options <development-buildoptions>`.
 
 
-I/O Method
-----------
+I/O Method and Engine Selection
+-------------------------------
 
 ADIOS2 has several engines for alternative file formats and other kinds of backends, yet natively writes to ``.bp`` files.
-The openPMD API uses the BP4 engine as the default file engine and the SST engine for streaming support.
-We currently leverage the default ADIOS2 transport parameters, i.e. ``POSIX`` on Unix systems and ``FStream`` on Windows.
+The openPMD API uses the File meta engine as the default file engine and the SST engine for streaming support.
+
+The ADIOS2 engine can be selected in different ways:
+
+1. Automatic detection via the selected file ending
+2. Explicit selection of an engine by specifying the environment variable ``OPENPMD_ADIOS2_ENGINE`` (case-independent).
+   This overrides the automatically detected engine.
+3. Explicit selection of an engine by specifying the JSON/TOML key ``adios2.engine.type`` as a case-independent string.
+   This overrides both previous options.
+
+Automatic engine detection supports the following extensions:
+
+.. list-table::
+   :header-rows: 1
+
+   * - Extension
+     - Selected ADIOS2 Engine
+   * - ``.bp``
+     - ``"file"``
+   * - ``.bp4``
+     - ``"bp4"``
+   * - ``.bp5``
+     - ``"bp5"``
+   * - ``.sst``
+     - ``"sst"``
+   * - ``.ssc``
+     - ``"ssc"``
+
+Specifying any of these extensions will automatically activate the ADIOS2 backend.
+The ADIOS2 backend will create file system paths exactly as they were specified and not change file extensions.
+Exceptions to this are the BP3 and SST engines which require their endings ``.bp`` and ``.sst`` respectively.
+
+For file engines, we currently leverage the default ADIOS2 transport parameters, i.e. ``POSIX`` on Unix systems and ``FStream`` on Windows.
 
 Steps
 -----

docs/source/dev/buildoptions.rst

@@ -68,7 +68,7 @@ The following options allow to switch to external installs of dependencies:
 ================================= =========== ======== ============= ========
 CMake Option                      Values      Installs Library       Version
 ================================= =========== ======== ============= ========
-``openPMD_USE_INTERNAL_CATCH``    **ON**/OFF  No       Catch2         2.13.4+
+``openPMD_USE_INTERNAL_CATCH``    **ON**/OFF  No       Catch2         2.13.9+
 ``openPMD_USE_INTERNAL_PYBIND11`` **ON**/OFF  No       pybind11        2.9.1+
 ``openPMD_USE_INTERNAL_JSON``     **ON**/OFF  No       NLohmann-JSON   3.9.1+
 ``openPMD_USE_INTERNAL_TOML11``   **ON**/OFF  No       toml11          3.7.1+

docs/source/dev/dependencies.rst

@@ -17,7 +17,7 @@ Shipped internally
 
 The following libraries are shipped internally in ``share/openPMD/thirdParty/`` for convenience:
 
-* `Catch2 <https://github.com/catchorg/Catch2>`_ 2.13.4+ (`BSL-1.0 <https://github.com/catchorg/Catch2/blob/master/LICENSE.txt>`__)
+* `Catch2 <https://github.com/catchorg/Catch2>`_ 2.13.9+ (`BSL-1.0 <https://github.com/catchorg/Catch2/blob/master/LICENSE.txt>`__)
 * `pybind11 <https://github.com/pybind/pybind11>`_ 2.9.1+ (`new BSD <https://github.com/pybind/pybind11/blob/master/LICENSE>`_)
 * `NLohmann-JSON <https://github.com/nlohmann/json>`_ 3.9.1+ (`MIT <https://github.com/nlohmann/json/blob/develop/LICENSE.MIT>`_)
 * `toml11 <https://github.com/ToruNiina/toml11>`_ 3.7.1+ (`MIT <https://github.com/ToruNiina/toml11/blob/master/LICENSE>`__)

include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

@@ -30,6 +30,7 @@
 #include "openPMD/IO/InvalidatableFile.hpp"
 #include "openPMD/IterationEncoding.hpp"
 #include "openPMD/auxiliary/JSON_internal.hpp"
+#include "openPMD/auxiliary/Mpi.hpp"
 #include "openPMD/backend/Writable.hpp"
 #include "openPMD/config.hpp"
 
@@ -131,12 +132,16 @@ class ADIOS2IOHandlerImpl
         AbstractIOHandler *,
         MPI_Comm,
         json::TracingJSON config,
-        std::string engineType);
+        std::string engineType,
+        std::string specifiedExtension);
 
 #endif // openPMD_HAVE_MPI
 
     explicit ADIOS2IOHandlerImpl(
-        AbstractIOHandler *, json::TracingJSON config, std::string engineType);
+        AbstractIOHandler *,
+        json::TracingJSON config,
+        std::string engineType,
+        std::string specifiedExtension);
 
     ~ADIOS2IOHandlerImpl() override;
 
@@ -145,6 +150,8 @@ class ADIOS2IOHandlerImpl
     void
     createFile(Writable *, Parameter<Operation::CREATE_FILE> const &) override;
 
+    void checkFile(Writable *, Parameter<Operation::CHECK_FILE> &) override;
+
     void
     createPath(Writable *, Parameter<Operation::CREATE_PATH> const &) override;
 
@@ -211,6 +218,7 @@ class ADIOS2IOHandlerImpl
 
 private:
     adios2::ADIOS m_ADIOS;
+    std::optional<auxiliary::Mock_MPI_Comm> m_communicator;
     /*
      * If the iteration encoding is variableBased, we default to using the
      * 2021_02_09 schema since it allows mutable attributes.
@@ -220,6 +228,11 @@ class ADIOS2IOHandlerImpl
      * The ADIOS2 engine type, to be passed to adios2::IO::SetEngine
      */
     std::string m_engineType;
+    /*
+     * The filename extension specified by the user.
+     */
+    std::string m_userSpecifiedExtension;
+
     ADIOS2Schema::schema_t m_schema = ADIOS2Schema::schema_0000_00_00;
 
     enum class UseSpan : char
@@ -309,7 +322,7 @@ class ADIOS2IOHandlerImpl
     // use m_config
     std::optional<std::vector<ParameterizedOperator> > getOperators();
 
-    std::string fileSuffix() const;
+    std::string fileSuffix(bool verbose = true) const;
 
     /*
      * We need to give names to IO objects. These names are irrelevant
@@ -1289,15 +1302,17 @@ class ADIOS2IOHandler : public AbstractIOHandler
         Access,
         MPI_Comm,
         json::TracingJSON options,
-        std::string engineType);
+        std::string engineType,
+        std::string specifiedExtension);
 
 #endif
 
     ADIOS2IOHandler(
         std::string path,
         Access,
         json::TracingJSON options,
-        std::string engineType);
+        std::string engineType,
+        std::string specifiedExtension);
 
     std::string backendName() const override
     {

include/openPMD/IO/ADIOS/CommonADIOS1IOHandler.hpp

@@ -52,6 +52,7 @@ class CommonADIOS1IOHandlerImpl : public AbstractIOHandlerImpl
 public:
     void
     createFile(Writable *, Parameter<Operation::CREATE_FILE> const &) override;
+    void checkFile(Writable *, Parameter<Operation::CHECK_FILE> &) override;
     void
     createPath(Writable *, Parameter<Operation::CREATE_PATH> const &) override;
     void createDataset(

include/openPMD/IO/AbstractIOHandlerHelper.hpp

@@ -35,6 +35,8 @@ namespace openPMD
  * @param   access      Access mode describing desired operations and
  permissions of the desired handler.
  * @param   format      Format describing the IO backend of the desired handler.
+ * @param   originalExtension The filename extension as it was originally
+ *                            specified by the user.
  * @param   comm        MPI communicator used for IO.
  * @param   options     JSON-formatted option string, to be interpreted by
  *                      the backend.
@@ -47,6 +49,7 @@ std::shared_ptr<AbstractIOHandler> createIOHandler(
     std::string path,
     Access access,
     Format format,
+    std::string originalExtension,
     MPI_Comm comm,
     JSON options);
 #endif
@@ -58,6 +61,8 @@ std::shared_ptr<AbstractIOHandler> createIOHandler(
  * @param   access      Access describing desired operations and permissions
  * of the desired handler.
  * @param   format      Format describing the IO backend of the desired handler.
+ * @param   originalExtension The filename extension as it was originally
+ *                            specified by the user.
  * @param   options     JSON-formatted option string, to be interpreted by
  *                      the backend.
  * @tparam  JSON        Substitute for nlohmann::json. Templated to avoid
@@ -66,9 +71,16 @@ std::shared_ptr<AbstractIOHandler> createIOHandler(
  */
 template <typename JSON>
 std::shared_ptr<AbstractIOHandler> createIOHandler(
-    std::string path, Access access, Format format, JSON options = JSON());
+    std::string path,
+    Access access,
+    Format format,
+    std::string originalExtension,
+    JSON options = JSON());
 
 // version without configuration to use in AuxiliaryTest
-std::shared_ptr<AbstractIOHandler>
-createIOHandler(std::string path, Access access, Format format);
+std::shared_ptr<AbstractIOHandler> createIOHandler(
+    std::string path,
+    Access access,
+    Format format,
+    std::string originalExtension);
 } // namespace openPMD

include/openPMD/IO/AbstractIOHandlerImpl.hpp

@@ -58,6 +58,12 @@ class AbstractIOHandlerImpl
                         deref_dynamic_cast<Parameter<Operation::CREATE_FILE> >(
                             i.parameter.get()));
                     break;
+                case O::CHECK_FILE:
+                    checkFile(
+                        i.writable,
+                        deref_dynamic_cast<Parameter<Operation::CHECK_FILE> >(
+                            i.parameter.get()));
+                    break;
                 case O::CREATE_PATH:
                     createPath(
                         i.writable,
@@ -198,6 +204,17 @@ class AbstractIOHandlerImpl
                     break;
                 }
             }
+            catch (std::exception const &e)
+            {
+                std::cerr << "[AbstractIOHandlerImpl] IO Task "
+                          << internal::operationAsString(i.operation)
+                          << " failed with exception. Removing task"
+                          << " from IO queue and passing on the exception. "
+                          << "Original exception was: \n"
+                          << e.what() << std::endl;
+                (*m_handler).m_work.pop();
+                throw;
+            }
             catch (...)
             {
                 std::cerr << "[AbstractIOHandlerImpl] IO Task "
@@ -220,6 +237,17 @@ class AbstractIOHandlerImpl
     virtual void
     closeFile(Writable *, Parameter<Operation::CLOSE_FILE> const &) = 0;
 
+    /**
+     * Check if the file specified by the parameter is already present on disk.
+     * The Writable is irrelevant for this method.
+     * A backend can choose to ignore this task and specify FileExists::DontKnow
+     * in the out parameter.
+     * The consequence will be that some top-level attributes might be defined
+     * a second time when appending to an existing file, because the frontend
+     * cannot be sure that the file already has these attributes.
+     */
+    virtual void checkFile(Writable *, Parameter<Operation::CHECK_FILE> &) = 0;
+
     /** Advance the file/stream that this writable belongs to.
      *
      * If the backend is based around usage of IO steps (especially streaming

include/openPMD/IO/Format.hpp

@@ -30,7 +30,9 @@ enum class Format
 {
     HDF5,
     ADIOS1,
-    ADIOS2,
+    ADIOS2_BP,
+    ADIOS2_BP4,
+    ADIOS2_BP5,
     ADIOS2_SST,
     ADIOS2_SSC,
     JSON,

include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp

@@ -25,6 +25,7 @@
 #include "openPMD/IO/AbstractIOHandlerImpl.hpp"
 
 #include "openPMD/auxiliary/JSON_internal.hpp"
+#include "openPMD/auxiliary/Mpi.hpp"
 
 #include <hdf5.h>
 #include <optional>
@@ -43,6 +44,7 @@ class HDF5IOHandlerImpl : public AbstractIOHandlerImpl
 
     void
     createFile(Writable *, Parameter<Operation::CREATE_FILE> const &) override;
+    void checkFile(Writable *, Parameter<Operation::CHECK_FILE> &) override;
     void
     createPath(Writable *, Parameter<Operation::CREATE_PATH> const &) override;
     void createDataset(
@@ -92,6 +94,9 @@ class HDF5IOHandlerImpl : public AbstractIOHandlerImpl
     hid_t m_H5T_CDOUBLE;
     hid_t m_H5T_CLONG_DOUBLE;
 
+protected:
+    std::optional<auxiliary::Mock_MPI_Comm> m_mockedMpiComm;
+
 private:
     json::TracingJSON m_config;
     std::string m_chunks = "auto";

include/openPMD/IO/IOTask.hpp

@@ -44,7 +44,8 @@ Writable *getWritable(Attributable *);
 /** Type of IO operation between logical and persistent data.
  */
 OPENPMDAPI_EXPORT_ENUM_CLASS(Operation){
-    CREATE_FILE,      OPEN_FILE,      CLOSE_FILE,    DELETE_FILE,
+    CREATE_FILE,      CHECK_FILE,     OPEN_FILE,     CLOSE_FILE,
+    DELETE_FILE,
 
     CREATE_PATH,      CLOSE_PATH,     OPEN_PATH,     DELETE_PATH,
     LIST_PATHS,
@@ -118,6 +119,32 @@ struct OPENPMDAPI_EXPORT Parameter<Operation::CREATE_FILE>
     IterationEncoding encoding = IterationEncoding::groupBased;
 };
 
+template <>
+struct OPENPMDAPI_EXPORT Parameter<Operation::CHECK_FILE>
+    : public AbstractParameter
+{
+    Parameter() = default;
+    Parameter(Parameter const &p)
+        : AbstractParameter(), name(p.name), fileExists(p.fileExists)
+    {}
+
+    std::unique_ptr<AbstractParameter> clone() const override
+    {
+        return std::unique_ptr<AbstractParameter>(
+            new Parameter<Operation::CHECK_FILE>(*this));
+    }
+
+    std::string name = "";
+    enum class FileExists
+    {
+        DontKnow,
+        Yes,
+        No
+    };
+    std::shared_ptr<FileExists> fileExists =
+        std::make_shared<FileExists>(FileExists::DontKnow);
+};
+
 template <>
 struct OPENPMDAPI_EXPORT Parameter<Operation::OPEN_FILE>
     : public AbstractParameter

include/openPMD/IO/JSON/JSONIOHandlerImpl.hpp

@@ -160,6 +160,8 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
     void
     createFile(Writable *, Parameter<Operation::CREATE_FILE> const &) override;
 
+    void checkFile(Writable *, Parameter<Operation::CHECK_FILE> &) override;
+
     void
     createPath(Writable *, Parameter<Operation::CREATE_PATH> const &) override;