-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unify Random-Access API and Streaming API into Series::snapshots() #1592
Conversation
0ee77a2
to
73336c9
Compare
1e862b7
to
ab0991d
Compare
src/snapshots/StatefulIterator.cpp
Outdated
switch (series.iterationEncoding()) | ||
{ | ||
using IE = IterationEncoding; | ||
case IE::fileBased: | ||
series.readFileBased(); | ||
break; | ||
case IE::groupBased: | ||
case IE::variableBased: { | ||
Parameter<Operation::OPEN_FILE> fOpen; | ||
fOpen.name = series.get().m_name; | ||
series.IOHandler()->enqueue(IOTask(&series, fOpen)); | ||
series.IOHandler()->flush(internal::defaultFlushParams); | ||
using PP = Parameter<Operation::OPEN_FILE>::ParsePreference; | ||
switch (*fOpen.out_parsePreference) | ||
{ | ||
case PP::PerStep: | ||
series.advance(AdvanceMode::BEGINSTEP); | ||
series.readGorVBased( | ||
/* do_always_throw_errors = */ false, /* init = */ true); | ||
break; | ||
case PP::UpFront: | ||
series.readGorVBased( | ||
/* do_always_throw_errors = */ false, /* init = */ true); | ||
/* | ||
* In linear read mode (where no parsing at all is done upon | ||
* constructing the Series), it might turn out after parsing | ||
* that what we expected to be a group-based Series was in fact | ||
* a single file of a file-based Series. | ||
* (E.g. when opening "data00000100.h5" directly instead of | ||
* "data%T.h5") | ||
* So we need to check the current value of | ||
* `iterationEncoding()` once more. | ||
*/ | ||
if (series.iterationEncoding() != IterationEncoding::fileBased) | ||
{ | ||
series.advance(AdvanceMode::BEGINSTEP); | ||
} | ||
break; | ||
} | ||
data.parsePreference = *fOpen.out_parsePreference; | ||
break; | ||
} | ||
} |
Check notice
Code scanning / CodeQL
Long switch case Note
variableBased (38 lines)
3845ed7
to
b5e4217
Compare
b5e4217
to
9987204
Compare
src/Series.cpp
Outdated
Access at, | ||
std::string const &options, | ||
// Either an MPI_Comm or none, the template works for both options | ||
MPI_Communicator &&...comm) |
Check notice
Code scanning / CodeQL
Unused local variable Note
667ee24
to
f5d8433
Compare
f5d8433
to
26e5a49
Compare
// increment/decrement | ||
// ChildClass &operator++(); | ||
// ChildClass &operator--(); | ||
// ChildClass &operator++(int); | ||
// ChildClass &operator--(int); |
Check notice
Code scanning / CodeQL
Commented-out code Note
// dereference | ||
// value_type const &operator*() const = 0; | ||
// value_type &operator*() = 0; |
Check notice
Code scanning / CodeQL
Commented-out code Note
@@ -185,6 +196,7 @@ | |||
} | |||
// ~Series intentionally not yet called | |||
|
|||
// std::cout << "READ " << filename << std::endl; |
Check notice
Code scanning / CodeQL
Commented-out code Note test
@@ -163,6 +173,7 @@ | |||
auxiliary::getEnvNum("OPENPMD_TEST_NFILES_MAX", 1030); | |||
std::string filename = | |||
"../samples/many_iterations/many_iterations_%T." + ext; | |||
// std::cout << "WRITE " << filename << std::endl; |
Check notice
Code scanning / CodeQL
Commented-out code Note test
19395b0
to
5b85307
Compare
for more information, see https://pre-commit.ci
Proper return() is supported beginning with CMake 3.25 only
for more information, see https://pre-commit.ci
a73c4c6
to
d278794
Compare
d278794
to
16c0f73
Compare
16c0f73
to
a6df22f
Compare
a6df22f
to
cad227a
Compare
The changes introduced with this PR are mostly (1) internal and (2) a new API. I will go ahead and merge this now, as many of the internal changes are the groundwork for new features laid out in the PR description. The new API should be treated as experimental until release. |
So far, there were two distinct APIs for accessing Iterations:
Series::iterations
Series::writeIterations()
andSeries::readIterations()
This PR introduces a new API
Series::snapshots()
that returns a unified container API. The unified container API can be implemented internally via random-access and via synchronous access, i.e. by using steps.This is part of an effort to bring both workflows and their functionalities closer together. While this redesigned container API does not yet go all the way along this road, it lays the fundament for such extensibility by (1) cleanly separating the API from its implementation and by (2) redesigning and cleaning up the implementation of the step-based workflow.
Preliminary/experimental support for reopening closed Iterations is now available, especially for file-based workflows.
Note that there are still caveats about reopening Iterations. Especially in ADIOS2, a step cannot be modified once closed, so writing new data is only possible into a new step. However, the Iteration handle currently does not delete old data definitions upon closing, meaning that there will be invalid data definitions in the handle.
In future, a closed Iteration will be removed from internal data structures (but users may keep it around for longer than that in their own data structures).
Future features to be built on top of this include:
SetStepSelection()
in ADIOS2writeIterations()
andreadIterations()
are now implemented as aliases/wrappers toSeries::snapshots()
. Due to the past somewhat unfortunate decision to introduceIndexedIteration
as iteration type forreadIterations()
, the method name cannot be reused without breaking compatibility.writeIterations()
however is just a wrapper forsnapshots()
with default arguments.TODO:
using
definitions, empty headerssnapshots(Snapshots::RandomAccess)
vssnapshots(Snapshots::CollectiveAccess)
or sosnapshots().begin()
-> reset to start, add sth likesnapshots().current()
, alternatively keep the current semantics and addsnapshots().reset()
?defer_iteration_parsing
in READ_LINEAR modeDiff: franzpoeschel/openPMD-api@shared-attributable-data...topic-iterator