Joined arrays in ADIOS2 #1382

franzpoeschel · 2023-02-27T17:06:56Z

This implements ornladios/ADIOS2#3452 in openPMD and (partially) fixes #1374.

We will need a fallback implementation for backends other than ADIOS2. Quoting the discussion in #1374:

Since this feature is relevant for basically all parallel code that writes particles, we should (as a second step) implement this also in openPMD, so that those codes still stay compatible with HDF5.
Using MPI collectives to determine local block offsets inside the particle list is a really common pattern for using openPMD in such code, and this is a great chance to eliminate this pattern, while still staying compatible with all openPMD backends.

TODO

More extensive testing, parallel tests
documentation
Merge RecordComponent: Properly handle uninitialized datasets #1316 first
Don't use .joinedDimension, magic number instead
~~Needs a standard update: numParticlesOffset may optionally be skipped, particlePatches must then be in order so they can be reconstructed from numParticles~~ Not updating the standard yet
Merge Refactor: Extract ADIOS2 BufferedActions struct to own file, rename to ADIOS2File #1577 first

test/SerialIOTest.cpp

+    s.close();
+}
+
+TEST_CASE("joined_dim", "[serial]")


franzpoeschel · 2023-03-03T12:56:37Z

Except for the documentation, this is now somewhat ready.
I see one problem: With this, it's practically impossible to write the particlePatchesOffset as that is determined by ADIOS2 and we don't know it before it's written.

test/SerialIOTest.cpp

+         */
+        for (size_t i = 0; i < length_of_patch * patches_per_rank; ++i)
+        {
+            REQUIRE(float(i) == particleData.get()[i]);


test/SerialIOTest.cpp

+        for (size_t i = 0; i < patches_per_rank; ++i)
+        {
+            REQUIRE(length_of_patch * i == numParticlesOffset.get()[i]);
+            REQUIRE(type(length_of_patch * i) == patchOffset.get()[i]);


guj · 2023-03-09T01:20:56Z

test/ParallelIOTest.cpp

+    }
+}
+
+TEST_CASE("joined_dim", "[serial]")


I think you mean "parallel"?

Yep, good catch

guj · 2023-03-09T01:24:49Z

test/ParallelIOTest.cpp

+        auto patchOffset = it.particles["e"].particlePatches["offset"]["x"];
+        auto patchExtent = it.particles["e"].particlePatches["extent"]["x"];
+        Dataset particlePatchesDS(determineDatatype<float>(), {0});
+        particlePatchesDS.joinedDimension = 0;


Is a sanity check needed here? do we want to throw error if someone set an invalid value of joinedDimension?

We will probably switch to the same API as ADIOS2, i.e. have some magic number JOINED_DIMENSION=std::numeric_limits<>::max and then use something like Dataset ds(<type>, {JOINED_DIMENSION, 128 , 128}). We should add a sanity check there.

The constructor of Dataset now verifies this.

franzpoeschel

see inline comments

franzpoeschel · 2023-04-25T09:57:28Z

include/openPMD/RecordComponent.hpp

+    template <typename T>
+    void verifyChunk(Offset const &, Extent const &) const;
+
+    void verifyChunk(Datatype, Offset const &, Extent const &) const;


Earlier versions of this PR added additional calls to RecordComponent::storeChunk, so instead of duplicating the verification logic, I extracted it.
Now no longer necessary, but it's still a useful refactoring.

include/openPMD/RecordComponent.tpp

@@ -397,7 +376,7 @@
 } // namespace detail

 template <typename Visitor, typename... Args>
-constexpr auto RecordComponent::visit(Args &&...args)
+auto RecordComponent::visit(Args &&...args)


include/openPMD/IO/ADIOS/ADIOS2File.hpp

+
+    void markActive(Writable *);
+
+    // bool isActive(std::string const & path);


src/IO/JSON/JSONIOHandlerImpl.cpp

src/IO/HDF5/HDF5IOHandler.cpp

ax3l · 2024-01-30T19:00:56Z

This looks great! Potentially can you add a Python test, to make sure the new enum exposed to the bindings works well?

src/binding/python/RecordComponent.cpp

+            extent.at(curAxis) =
+                undo_mocked_extent && slicelength == PYTHON_JOINED_DIMENSION
+                ? Dataset::JOINED_DIMENSION
+                : slicelength; // stop - start;


Use magic number instead of API call (impl)

Only test if ADIOS2 version at least 2.9

for more information, see https://pre-commit.ci

ax3l

Looks great!!

ax3l · 2024-02-27T21:57:46Z

docs/source/usage/workflow.rst

+
+   (Currently) only supported in ADIOS2 no older than v2.9.0 under the conditions listed in the `ADIOS2 documentation on joined arrays <https://adios2.readthedocs.io/en/latest/components/components.html#shapes>`_.
+
+   In some cases, the concrete chunk within a dataset does not matter and the computation of indexes is a needless computational and mental overhead.


Suggested change

In some cases, the concrete chunk within a dataset does not matter and the computation of indexes is a needless computational and mental overhead.

In some cases, the concrete chunk within a dataset does not matter and the computation of indexes is a needless computational, (MPI) communication and mental overhead.

ax3l · 2024-02-28T01:46:16Z

examples/5_write_parallel.py

+    if USE_JOINED_DIMENSION:
+        # explicit API
+        # mymesh.store_chunk(local_data, [], [10, 300])
+        mymesh[:, :] = local_data
+        # or short:
+        # mymesh[()] = local_data


Fancy!
Good examples even in the commented code.

franzpoeschel added backend: ADIOS2 api: new additions to the API labels Feb 27, 2023

franzpoeschel marked this pull request as draft February 27, 2023 17:07

franzpoeschel mentioned this pull request Feb 27, 2023

Fully Support JoinedArrays ornladios/ADIOS2#3452

Closed

github-advanced-security bot found potential problems Feb 27, 2023

View reviewed changes

test/SerialIOTest.cpp Fixed Show fixed Hide fixed

franzpoeschel marked this pull request as ready for review February 28, 2023 16:48

franzpoeschel force-pushed the joined-arrays branch 2 times, most recently from d0d1e7a to 4d68621 Compare March 1, 2023 12:04

github-advanced-security bot found potential problems Mar 1, 2023

View reviewed changes

test/SerialIOTest.cpp

s.close();

}

TEST_CASE("joined_dim", "[serial]")

Check notice

Code scanning / CodeQL

Unused static function Note test

Static function C_A_T_C_H_T_E_S_T_88 is unreachable (

autoRegistrar89

Loading
must be removed at the same time)

franzpoeschel force-pushed the joined-arrays branch from 4d68621 to 1f299b6 Compare March 1, 2023 13:21

franzpoeschel changed the title ~~[WIP] Joined arrays in ADIOS2~~ Joined arrays in ADIOS2 Mar 3, 2023

github-advanced-security bot found potential problems Mar 3, 2023

View reviewed changes

ax3l requested review from ax3l and guj March 7, 2023 18:41

guj approved these changes Mar 9, 2023

View reviewed changes

franzpoeschel commented Mar 9, 2023

View reviewed changes

franzpoeschel force-pushed the joined-arrays branch from 6a53c77 to 5bdbcc2 Compare March 9, 2023 15:06

franzpoeschel force-pushed the joined-arrays branch from 5bdbcc2 to 249ede8 Compare March 30, 2023 11:17

franzpoeschel force-pushed the joined-arrays branch 4 times, most recently from a941b31 to 92312a7 Compare April 14, 2023 10:38

franzpoeschel force-pushed the joined-arrays branch from 92312a7 to 36f81a2 Compare April 25, 2023 09:30

franzpoeschel commented Apr 25, 2023

View reviewed changes

franzpoeschel force-pushed the joined-arrays branch from 36f81a2 to 64b9a73 Compare April 25, 2023 11:41

franzpoeschel force-pushed the joined-arrays branch from 5cd4760 to 8a9968e Compare May 4, 2023 10:26

ax3l added this to the 0.16.0 milestone May 9, 2023

franzpoeschel force-pushed the joined-arrays branch from 8a9968e to 672985a Compare May 24, 2023 12:24

ax3l self-assigned this Jul 25, 2023

github-advanced-security bot found potential problems Dec 22, 2023

View reviewed changes

franzpoeschel force-pushed the joined-arrays branch from b70a5a1 to 4e4edea Compare December 22, 2023 18:31

franzpoeschel force-pushed the joined-arrays branch from 4e4edea to 200cc69 Compare January 9, 2024 14:28

github-advanced-security bot found potential problems Jan 9, 2024

View reviewed changes

include/openPMD/IO/ADIOS/ADIOS2File.hpp Outdated

void markActive(Writable *);

// bool isActive(std::string const & path);

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.

franzpoeschel force-pushed the joined-arrays branch 4 times, most recently from 23ff8e6 to c13c20e Compare January 23, 2024 16:58

ax3l reviewed Jan 30, 2024

View reviewed changes

src/IO/JSON/JSONIOHandlerImpl.cpp Show resolved Hide resolved

src/IO/HDF5/HDF5IOHandler.cpp Show resolved Hide resolved

franzpoeschel force-pushed the joined-arrays branch 2 times, most recently from 793576d to ce51f2a Compare February 6, 2024 14:33

github-advanced-security bot found potential problems Feb 6, 2024

View reviewed changes

src/binding/python/RecordComponent.cpp

extent.at(curAxis) =

undo_mocked_extent && slicelength == PYTHON_JOINED_DIMENSION

? Dataset::JOINED_DIMENSION

: slicelength; // stop - start;

Check notice

Code scanning / CodeQL

Commented-out code Note

This comment appears to contain commented-out code.

franzpoeschel force-pushed the joined-arrays branch from d9c545c to 342337e Compare February 13, 2024 19:10

franzpoeschel and others added 13 commits February 27, 2024 11:49

Extract verifyChunk

0df4704

Main implementation

af6eaad

Use magic number instead of API call (impl)

Python bindings

07b0331

Throw errors if unsupported

3d5dcf7

Testing

b79af4f

Only test if ADIOS2 version at least 2.9

Documentation

e55233e

[pre-commit.ci] auto fixes from pre-commit.com hooks

403c0a9

for more information, see https://pre-commit.ci

Fix ADIOS2 checks for multidimensional joined arrays

ea09b71

Python test

c45ccd2

Expose this to the sliced Python API

70d6f3f

Fix

5af59b1

Fix formatting

51eca2a

[pre-commit.ci] auto fixes from pre-commit.com hooks

32dbc60

for more information, see https://pre-commit.ci

franzpoeschel force-pushed the joined-arrays branch from 09eff2d to 32dbc60 Compare February 27, 2024 10:49

ax3l approved these changes Feb 28, 2024

View reviewed changes

ax3l merged commit 5fec415 into openPMD:dev Feb 28, 2024
31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Joined arrays in ADIOS2 #1382

Joined arrays in ADIOS2 #1382

franzpoeschel commented Feb 27, 2023 •

edited

Loading

franzpoeschel commented Mar 3, 2023

guj Mar 9, 2023

franzpoeschel Mar 9, 2023

guj Mar 9, 2023 •

edited

Loading

franzpoeschel Mar 9, 2023

franzpoeschel Jun 6, 2023

franzpoeschel left a comment

franzpoeschel Apr 25, 2023

ax3l commented Jan 30, 2024

ax3l left a comment

ax3l Feb 27, 2024

ax3l Feb 28, 2024


		void markActive(Writable *);

		// bool isActive(std::string const & path);


		(Currently) only supported in ADIOS2 no older than v2.9.0 under the conditions listed in the `ADIOS2 documentation on joined arrays <https://adios2.readthedocs.io/en/latest/components/components.html#shapes>`_.

		In some cases, the concrete chunk within a dataset does not matter and the computation of indexes is a needless computational and mental overhead.

Joined arrays in ADIOS2 #1382

Joined arrays in ADIOS2 #1382

Conversation

franzpoeschel commented Feb 27, 2023 • edited Loading

franzpoeschel commented Mar 3, 2023

guj Mar 9, 2023

Choose a reason for hiding this comment

franzpoeschel Mar 9, 2023

Choose a reason for hiding this comment

guj Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

franzpoeschel Mar 9, 2023

Choose a reason for hiding this comment

franzpoeschel Jun 6, 2023

Choose a reason for hiding this comment

franzpoeschel left a comment

Choose a reason for hiding this comment

franzpoeschel Apr 25, 2023

Choose a reason for hiding this comment

ax3l commented Jan 30, 2024

ax3l left a comment

Choose a reason for hiding this comment

ax3l Feb 27, 2024

Choose a reason for hiding this comment

ax3l Feb 28, 2024

Choose a reason for hiding this comment

franzpoeschel commented Feb 27, 2023 •

edited

Loading

guj Mar 9, 2023 •

edited

Loading