Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce more flexible storeChunk() syntax, use to add ADIOS2 memory selection #1620

Open
wants to merge 41 commits into
base: dev
Choose a base branch
from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented May 17, 2024

We currently don't support memory selections yet, e.g. when you have a 2D memory buffer but want to write only an arbitrary block from it. That block is then non-contiguous, but ADIOS2 has so-called memory selections to support this.

We currently don't have an API that would be able to expose this (except in Python where the native Python buffer protocol is powerful enough to express this; we currently throw an error when detecting this strides in selection are inefficient, not implemented!). Since the loadChunk()/storeChunk() API is somewhat convoluted by now anyway, I didn't want to add even further overloads.

Instead, this PR proposes a new syntax for setting up load/store calls:

RecordComponent E_y;

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueLoad<int>(); // -> std::shared_ptr<int>

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueLoadVariant(); // ->std::variant<std::shared_ptr<char>, ...>

E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .enqueueStore<int>(); // -> DynamicMemoryView<int>, i.e. Span API

std::shared_ptr<int> data;
E_y.prepareLoadStore()
    .offset({0, 5}) // optional
    .extent({1,5}) // optional
    .withSharedPtr(data) // or withUniquePtr(), withRawPtr(), withContiguousContainer()
    .memorySelection({{1,1},{5,5}}) // call only available after having called withSharedPtr()
    .enqueueStore(); // -> void, no template parameter needed since withSharedPtr(data) already gives the type

E_y.prepareLoadStore()
    .withSharedPtr(data) // or withUniquePtr(), withRawPtr(), withContiguousContainer()
    .enqueueLoad(); // load the whole dataset into the buffer

This API makes it easier to add functionality without going through every single overload and adapting it. Further, it makes the API more consistent, e.g. offset and extent are now always optional and not just for those calls that implement a default logic.

With this PR, the old API calls are now fully implemented via the new API, giving it a good test coverage.

TODO:

  • What about Python? Maybe stick with the buffer protocol there for now.
  • Experiment: Return futures instead of bare return types: Use the chance to introduce a safe async API. Introduce option .noop_future() to disable future return types.

@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch 2 times, most recently from 727cbbc to ddcdfc9 Compare May 22, 2024 11:44
@ax3l ax3l self-requested a review May 31, 2024 17:50
@ax3l ax3l self-assigned this May 31, 2024
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from c05d535 to ba7c582 Compare June 7, 2024 12:41
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from ba7c582 to d7eb891 Compare June 24, 2024 12:47
src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch 2 times, most recently from 332932f to efb0876 Compare June 25, 2024 13:56
src/LoadStoreChunk.cpp Fixed Show fixed Hide fixed
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from efb0876 to 39e3d71 Compare June 26, 2024 11:48
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch 4 times, most recently from 47d497b to b2bc355 Compare July 22, 2024 10:21
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from b699520 to 3430b6d Compare July 23, 2024 14:05
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from 9b5acff to bd7b378 Compare September 3, 2024 13:06
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from 208c381 to e51e635 Compare November 15, 2024 14:45
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from 92e8527 to 1963357 Compare December 10, 2024 17:36
@franzpoeschel franzpoeschel force-pushed the adios2-memory-selection branch from c8be94b to 8e455b4 Compare December 17, 2024 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants