Skip to content

Commit

Permalink
Merge pull request #2371 from NAThompson/docs_updates_2
Browse files Browse the repository at this point in the history
Update documentation.
  • Loading branch information
pnorbert authored Jul 10, 2020
2 parents 9b4f877 + 782e5c4 commit 54cca3e
Show file tree
Hide file tree
Showing 8 changed files with 86 additions and 67 deletions.
2 changes: 1 addition & 1 deletion docs/user_guide/source/api_high/cxx11.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ C++11 Write example
{
if(rank == 0 && step == 0) // global variable
{
oStream.write<int>("size", size);
oStream.write<int32_t>("size", size);
}

// physicalTime double, <double> is optional
Expand Down
7 changes: 5 additions & 2 deletions docs/user_guide/source/components/attribute.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
Attribute
*********

Attributes are extra information associated with a particular IO component. They can be thought of a very simplified version of a Variable, but with the goal of adding extra metadata. The most common use is the addition of human-readable information available when producing data (*e.g.* ``"experiment name"``, ``"date and time"``, ``"04,27,2017"``, or a schema).
Attributes are extra information associated with a particular IO component.
They can be thought of as a very simplified ``Variable``, but with the goal of adding extra metadata.
The most common use is the addition of human-readable metadata (*e.g.* ``"experiment name"``, ``"date and time"``, ``"04,27,2017"``, or a schema).

Currently, ADIOS2 supports single values and arrays of primitive types (excluding ``complex<T>``) for the template type in the ``IO::DefineAttribute<T>`` and ``IO::InquireAttribute<T>`` function (in C++).

Expand Down Expand Up @@ -30,4 +32,5 @@ The returned object (``DefineAttribute`` or ``InquireAttribute``) only serves th

.. note:
Attributes are not forcibly associated to a particular variable in ADIOS2. Developers are free to create associations through their own naming conventions.
Attributes are not forcibly associated to a particular variable in ADIOS2.
Developers are free to create associations through their own naming conventions.
54 changes: 35 additions & 19 deletions docs/user_guide/source/components/engine.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,35 @@
Engine
******

The Engine abstraction component serves as the base interface to the actual IO Systems executing the heavy-load tasks performed when Producing and Consuming data.
The Engine abstraction component serves as the base interface to the actual IO systems executing the heavy-load tasks performed when producing and consuming data.

Engine functionality works around two concepts from the application's point-of-view:
Engine functionality works around two concepts:

1. Self-describing variables are published and consumed in "steps" in either "File" random-access (all steps are available) or "Streaming" (steps are available as they are produced in a step-by-step fashion).
2. Self-describing variables are published (Put) and consumed (Get) using a "sync" or "deferred" (lazy evaluation) policy.
1. Variables are published (``Put``) and consumed (``Get``) in "steps" in either "File" random-access (all steps are available) or "Streaming" (steps are available as they are produced in a step-by-step fashion).
2. Variables are published (``Put``) and consumed (``Get``) using a "sync" or "deferred" (lazy evaluation) policy.

.. caution::

The ADIOS 2 "step" is a logical abstraction that means different things depending on the application context. Examples: "time step", "iteration step", "inner loop step", or "interpolation step", "variable section", etc. It only indicates how the variables were passed into ADIOS 2 (e.g. I/O steps) without the user having to index this information on their own.
The ADIOS2 "step" is a logical abstraction that means different things depending on the application context.
Examples: "time step", "iteration step", "inner loop step", or "interpolation step", "variable section", etc.
It only indicates how the variables were passed into ADIOS2 (e.g. I/O steps) without the user having to index this information on their own.

.. tip::

Publishing/Consuming data can be seen as a round-trip in ADIOS 2. Put and Get APIs for write/append and read modes aim to be "symmetric". Hence, reusing similar functions, objects, semantics as much as possible.
Publishing and consuming data can be seen as a round-trip in ADIOS2.
``Put`` and ``Get`` APIs for write/append and read modes aim to be "symmetric".
Hence, reusing similar functions, objects, semantics as much as possible.

The rest of the section explains the important concepts

BeginStep
---------

Begin logical step and return the status (via an enum) of the stream to be read/written. In streaming engines BeginStep in where the receiver tries to acquire a new step in the reading process. The full signature allows for a mode and timeout parameters. See :ref:`Supported Engines` for more information on what engine allows. A simplified signature allows each engine to pick reasonable defaults.
Begins a logical step and return the status (via an enum) of the stream to be read/written.
In streaming engines ``BeginStep`` is where the receiver tries to acquire a new step in the reading process.
The full signature allows for a mode and timeout parameters.
See :ref:`Supported Engines` for more information on what engine allows.
A simplified signature allows each engine to pick reasonable defaults.

.. code-block:: c++

Expand All @@ -41,17 +49,19 @@ EndStep

.. tip::

To write portable code for a step-by-step access across adios2 engines (file and streaming engines) use BeginStep and EndStep.
To write portable code for a step-by-step access across ADIOS2 engines (file and streaming engines) use ``BeginStep`` and ``EndStep``.

.. danger::

Accessing random steps in read mode (e.g. Variable<T>::SetStepSelection in file engines) will create a conflict with BeginStep and EndStep and will throw an exception. In file engines, data is either consumed in a random-access or step-by-step mode, but not both.
Accessing random steps in read mode (e.g. ``Variable<T>::SetStepSelection`` in file engines) will create a conflict with ``BeginStep`` and ``EndStep`` and will throw an exception.
In file engines, data is either consumed in a random-access or step-by-step mode, but not both.


Close
-----

Close current engine and underlying transports. Engine object can't be used after this call.
Close current engine and underlying transports.
An ``Engine`` object can't be used after this call.

.. tip::

Expand All @@ -61,30 +71,35 @@ Close
Put: modes and memory contracts
-------------------------------

``Put`` is the generalized abstract function for publishing data in adios2 when an Engine is created using Write, or Append, mode at ``IO::Open``.
``Put`` is the generalized abstract function for publishing data in adios2 when an Engine is created using ``Write`` or ``Append`` mode at ``IO::Open``.

The most common signature is the one that passes a ``Variable<T>`` object for the metadata, a ``const`` piece of contiguous memory for the data, and a mode for either Deferred (data is collected until EndStep/PerformPuts/Close) or Sync (data is reusable immediately). This is the most common use case in applications.
The most common signature is the one that passes a ``Variable<T>`` object for the metadata, a ``const`` piece of contiguous memory for the data, and a mode for either ``Deferred`` (data is collected until EndStep/PerformPuts/Close) or ``Sync`` (data is reusable immediately).
This is the most common use case in applications.

1. Deferred (default) or Sync mode, data is contiguous memory

.. code-block:: c++

void Put(Variable<T> variable, const T* data, const adios2::Mode = adios2::Mode::Deferred);

Optionally, adios2 Engines can provide direct access to its buffer memory using an overload that returns a piece of memory to a variable block, basically a zero-copy. Variable<T>::Span is based on a subset of the upcoming `C++20 std::span <https://en.cppreference.com/w/cpp/container/span>`_, which is non-owning and typed contiguous memory piece (it helps to review what std::span is, formerly known as array_view). Spans act as a 1D memory container meant to be filled out by the application. It is safely used as any other STL sequence container, with iterators ``begin()`` and ``end()``, ``operator[]`` and ``at()``, while also providing ``data()`` and ``size()`` functions to manipulate the internal pointer.
Optionally, ADIOS2 Engines can provide direct access to its buffer memory using an overload that returns a piece of memory to a variable block.
``Variable<T>::Span`` is based on a subset of the upcoming `C++20 std::span <https://en.cppreference.com/w/cpp/container/span>`_, which is a non-owning reference to a block of contiguous memory.
Spans act as a 1D container meant to be filled out by the application.
It is safely used like any other STL container, providing ``begin()`` and ``end()`` iterators, ``operator[]`` and ``at()``, while also providing ``data()`` and ``size()`` functions to manipulate the internal pointer.

Variable<T>::Span is helpful in situations in which temporaries are needed to create contiguous pieces of memory from non-contiguous pieces (``e.g.`` tables, arrays without ghost-cells), or just to save memory as the returned Variable<T>::Span can be used for computation, thus avoiding an extra copy from user memory into the adios buffer.
Variable<T>::Span combines a hybrid Sync and Deferred mode, in which the initial value and memory allocations are Sync, while data population and metadata collection are done at EndStep/PerformPuts/Close. Memory contracts are explained later in this chapter followed by examples.
``Variable<T>::Span`` is helpful in situations in which temporaries are needed to create contiguous pieces of memory from non-contiguous pieces (``e.g.`` tables, arrays without ghost-cells), or just to save memory as the returned ``Variable<T>::Span`` can be used for computation, thus avoiding an extra copy from user memory into the adios buffer.
``Variable<T>::Span`` combines a hybrid Sync and Deferred mode, in which the initial value and memory allocations are Sync, while data population and metadata collection are done at EndStep/PerformPuts/Close.
Memory contracts are explained later in this chapter followed by examples.

The following Variable<T>::Span signatures are available:
The following ``Variable<T>::Span`` signatures are available:

2. Return a span setting a default T() value into a default buffer
2. Return a span setting a default ``T()`` value into a default buffer

.. code-block:: c++

Variable<T>::Span Put(Variable<T> variable);

3. Return a span setting an initial fill value into a certain buffer. If span is not returned then the fillValue is fixed for that block.
3. Return a span setting an initial fill value into a certain buffer. If span is not returned then the ``fillValue`` is fixed for that block.

.. code-block:: c++

Expand All @@ -93,7 +108,8 @@ The following Variable<T>::Span signatures are available:

.. warning::

As of version 2.4.0 only the default BP3 engine using the C++11 bindings supports ``Variable<T>::Span`` Put signatures. We plan to support this feature and add this to streaming Engines.
As of version 2.4.0 only the default BP3 engine using the C++11 bindings supports ``Variable<T>::Span`` Put signatures.
We plan to support this feature and add this to streaming Engines.


In summary, the following are the current Put signatures for publishing data in ADIOS 2:
Expand Down
9 changes: 5 additions & 4 deletions docs/user_guide/source/components/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ If ``SetEngine`` is not called, then the ``BPFile`` engine is used.
void adios2::IO::SetEngine( const std::string engineType );
/** Example */
bpIO.SetEngine("BPFileWriter");
bpIO.SetEngine("BPFile");
Each ``Engine`` allows the user to fine tune execution of buffering and output tasks via parameters passed to the ``IO`` object.
These parameters are then propagated to the ``Engine``.
Expand Down Expand Up @@ -163,7 +163,8 @@ Keep in mind that Attributes apply to all Engines created by the ``IO`` object a
const size_t elements);
In situations in which a variable and attribute has been previously defined:
1) a variable/attribute reference goes out of scope, or 2) when reading from an incoming stream, IO can inquire the current variables and attributes and return a pointer acting as reference. If the inquired variable/attribute is not found, then ``nullptr`` is returned.
1) a variable/attribute reference goes out of scope, or 2) when reading from an incoming stream, IO can inquire the current variables and attributes and return a pointer acting as reference.
If the inquired variable/attribute is not found, then the overloaded ``bool()`` operator of returns ``false``.

.. code-block:: c++

Expand All @@ -172,15 +173,15 @@ In situations in which a variable and attribute has been previously defined:
adios2::Attribute<T> InquireAttribute<T>(const std::string &name) noexcept;
/** Example */
adios2::Variable<float> varPressure = io.InquireVariable<T>("pressure");
adios2::Variable<float> varPressure = io.InquireVariable<float>("pressure");
if( varPressure ) // it exists
{
...
}

.. note::
``InquireVariable`` returns a pointer so that ``nullptr`` can indicate invalid states (e.g. variables haven't arrived in a stream, weren't previously defined, or weren't written in a file).
``adios2::Variable`` overloads ``operator bool()`` so that we can check for invalid states (e.g. variables haven't arrived in a stream, weren't previously defined, or weren't written in a file).

.. caution::

Expand Down
6 changes: 4 additions & 2 deletions docs/user_guide/source/components/operator.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
Operator
********

The Operator abstraction allows ADIOS2 to act upon the user application data, either from a ``adios2::Variable`` or a set of Variables in an ``adios2::IO`` object. Current supported operations are:
The Operator abstraction allows ADIOS2 to act upon the user application data, either from a ``adios2::Variable`` or a set of Variables in an ``adios2::IO`` object.
Current supported operations are:

1. Data compression/decompression, lossy and lossless.
2. Callback functions (C++11 bindings only) supported by specific engines
Expand All @@ -11,4 +12,5 @@ ADIOS2 enables the use of third-party libraries to execute these tasks.

.. warning::

Make sure your ADIOS2 library installation used for writing and reading was linked with a compatible version of a third-party dependency when working with operators. ADIOS2 will issue an exception if an operator library dependency is missing.
Make sure your ADIOS2 library installation used for writing and reading was linked with a compatible version of a third-party dependency when working with operators.
ADIOS2 will issue an exception if an operator library dependency is missing.
3 changes: 2 additions & 1 deletion docs/user_guide/source/components/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ Components Overview

.. note::

If you are doing simple tasks where performance is a non-critical aspect please go to the :ref:`High-Level APIs` section for a quick start. If you are an HPC application developer or you want to use ADIOS2 functionality in full please read this chapter.
If you are doing simple tasks where performance is a non-critical aspect please go to the :ref:`High-Level APIs` section for a quick start.
If you are an HPC application developer or you want to use ADIOS2 functionality in full please read this chapter.


The simple way to understand the big picture for the ADIOS2 unified user interface components is to map each class to the actual definition of the ADIOS acronym.
Expand Down
9 changes: 6 additions & 3 deletions docs/user_guide/source/components/runtime.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ ADIOS2 supports passing an optional runtime configuration file to the :ref:`ADIO

This file contains key-value pairs equivalent to the compile time ``IO::SetParameters`` (``adios2_set_parameter`` in C, Fortran), and ``IO::AddTransport`` (``adios2_set_transport_parameter`` in C, Fortran).

Each Engine and Operator must provide a set of available parameters as described in the :ref:`Supported Engines` section. Up to version v2.6.0 only XML is supported, v2.6.0 and beyond support XML as well as YAML.
Each ``Engine`` and ``Operator`` must provide a set of available parameters as described in the :ref:`Supported Engines` section.
Up to version v2.6.0 only XML is supported, v2.6.0 and beyond support XML as well as YAML.

.. warning::

Expand Down Expand Up @@ -50,7 +51,10 @@ XML
YAML
----

Starting with v2.6.0, ADIOS supports YAML configuration files. The syntax follows strict use of the YAML node keywords mapping to the ADIOS2 components hierarchy. If a keyword is unknown ADIOS2 simply ignores it. For an example file refer to `adios2 config file example in our repo. <https://github.com/ornladios/ADIOS2/tree/master/testing/adios2/yaml/proto.yaml>`_
Starting with v2.6.0, ADIOS supports YAML configuration files.
The syntax follows strict use of the YAML node keywords mapping to the ADIOS2 components hierarchy.
If a keyword is unknown ADIOS2 simply ignores it.
For an example file refer to `adios2 config file example in our repo. <https://github.com/ornladios/ADIOS2/tree/master/testing/adios2/yaml/proto.yaml>`_

.. code-block:: yaml
Expand Down Expand Up @@ -100,4 +104,3 @@ Starting with v2.6.0, ADIOS supports YAML configuration files. The syntax follow
.. tip::

Run a YAML validator or use a YAML editor to make sure the provided file is YAML compatible.

Loading

0 comments on commit 54cca3e

Please sign in to comment.