-
Notifications
You must be signed in to change notification settings - Fork 128
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #2790 from caitlinross/plugin-engine
Plugin engine
- Loading branch information
Showing
24 changed files
with
1,497 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
************* | ||
Plugin Engine | ||
************* | ||
|
||
The ``Plugin`` engine enables the ability to load an engine located in a separate library. | ||
Your plugin class needs to inherit from the ``PluginEngineInterface`` class in the ``adios2/engine/plugin/PluginEngineInterface.h`` header. | ||
Depending on the type of engine you want to implement, you'll need to override a number of methods that are inherited from the ``adios2::core::Engine`` class. | ||
These are briefly described in the following table. | ||
More detailed documentation can be found in ``adios2/core/Engine.h``. | ||
|
||
========================= ===================== =========================================================== | ||
**Method** **Engine Type** **Description** | ||
========================= ===================== =========================================================== | ||
``BeginStep()`` Read/Write Indicates the beginning of a step | ||
``EndStep()`` Read/Write Indicates the end of a step | ||
``CurrentStep()`` Read/Write Returns current step info | ||
``DoClose()`` Read/Write Close a particular transport | ||
``Init()`` Read/Write Engine initialization | ||
``InitParameters()`` Read/Write Initialize parameters | ||
``InitTransports()`` Read/Write Initialize transports | ||
``PerformPuts()`` Write Execute all deferred mode ``Put`` | ||
``Flush()`` Write Flushes data and metadata to a transport | ||
``DoPut()`` Write Implementation for ``Put`` | ||
``DoPutSync()`` Write Implementation for ``Put`` (Sync mode) | ||
``DoPutDeferred()`` Write Implementation for ``Put`` (Deferred Mode) | ||
``PerformGets()`` Read Execute all deferred mode ``Get`` | ||
``DoGetSync()`` Read Implementation for ``Get`` (Sync mode) | ||
``DoGetDeferred()`` Read Implementation for ``Get`` (Deferred Mode) | ||
========================= ===================== =========================================================== | ||
|
||
Examples showing how to implement an engine plugin can be found in ``examples/plugins/engine``. | ||
An example write engine is ``ExampleWritePlugin.h``, while an example read engine is in ``ExampleReadPlugin.h``. | ||
The writer is a simple file writing engine that creates a directory (called ``ExamplePlugin`` by default) and writes variable information to vars.txt and actual data to data.txt. | ||
The reader example reads the files output by the writer example. | ||
|
||
In addition to implementing the methods above, you'll need to implement ``EngineCreate()`` and ``EngineDestroy`` functions so the Plugin Engine can create/destroy the engine object. | ||
Because of C++ name mangling, you'll need to use ``extern "C"``. | ||
Looking at ``ExampleWritePlugin.h``, this looks like: | ||
|
||
.. code-block:: c++ | ||
|
||
extern "C" { | ||
|
||
adios2::core::engine::ExampleWritePlugin * | ||
EngineCreate(adios2::core::IO &io, const std::string &name, | ||
const adios2::Mode mode, adios2::helper::Comm comm) | ||
{ | ||
return new adios2::core::engine::ExampleWritePlugin(io, name, mode, | ||
comm.Duplicate()); | ||
} | ||
|
||
void EngineDestroy(adios2::core::engine::ExampleWritePlugin * obj) | ||
{ | ||
delete obj; | ||
} | ||
|
||
} | ||
|
||
To build your plugin, your CMake should look something like: | ||
|
||
.. code-block:: cmake | ||
find_package(ADIOS2 REQUIRED) | ||
set(BUILD_SHARED_LIBS ON) | ||
add_library(PluginEngineWrite | ||
ExampleWritePlugin.cpp | ||
) | ||
target_link_libraries(PluginEngineWrite adios2::cxx11 adios2::core) | ||
When using the Plugin Engine, ADIOS will check for your plugin at the path specified in the ``ADIOS2_PLUGIN_PATH`` environment variable. | ||
If ``ADIOS2_PLUGIN_PATH`` is not set, and a path is not specified in the settings for the Plugin Engine (see below steps for using a plugin in your application), then the usual ``dlopen`` search is performed (see `dlopen man page <https://man7.org/linux/man-pages/man3/dlopen.3.html>`_). | ||
|
||
The following steps show how to use your engine plugin in your application. | ||
``examplePluginEngine_write.cpp`` and ``examplePluginEngine_read.cpp`` are an example of how to use the engine plugins described above. | ||
The key steps to use your plugin are: | ||
|
||
1. Set engine to ``Plugin``. i.e.: | ||
|
||
.. code-block:: c++ | ||
|
||
io.SetEngine("Plugin"); | ||
|
||
2. Set ``PluginName`` (optional) and ``PluginLibrary`` (required) parameters. | ||
If you don't set ``PluginName``, the Plugin Engine will give your plugin a default name of ``UserPlugin``. | ||
In the write example, this looks like | ||
|
||
.. code-block:: c++ | ||
|
||
io.SetParameters({{"PluginName", "WritePlugin"}}); | ||
io.SetParameters({{"PluginLibrary", "PluginEngineWrite"}}); | ||
// also possible to use the path here instead of setting ADIOS2_PLUGIN_PATH | ||
// io.SetParameters({{"PluginLibrary", "/path/to/libPluginEngineWrite.so"}}) | ||
|
||
.. note:: | ||
You don't need to add the ``lib`` prefix or the shared library ending (e.g., ``.so``, ``.dll``, etc.). | ||
ADIOS will add these when searching for your plugin library. | ||
If you do at the prefix/suffix, ADIOS should still be able to find your plugin. | ||
|
||
|
||
At this point you can open the engine and use it as you would any other ADIOS engine. | ||
You also shouldn't need to make any changes to your CMake files for your application. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
#------------------------------------------------------------------------------# | ||
# Distributed under the OSI-approved Apache License, Version 2.0. See | ||
# accompanying file Copyright.txt for details. | ||
#------------------------------------------------------------------------------# | ||
|
||
add_subdirectory(engine) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#------------------------------------------------------------------------------# | ||
# Distributed under the OSI-approved Apache License, Version 2.0. See | ||
# accompanying file Copyright.txt for details. | ||
#------------------------------------------------------------------------------# | ||
|
||
add_library(PluginEngineWriteExample | ||
ExampleWritePlugin.cpp | ||
) | ||
target_link_libraries(PluginEngineWriteExample adios2::cxx11 adios2_core) | ||
|
||
add_library(PluginEngineReadExample | ||
ExampleReadPlugin.cpp | ||
) | ||
target_link_libraries(PluginEngineReadExample adios2::cxx11 adios2_core) | ||
|
||
add_executable(examplePluginEngine_write | ||
examplePluginEngine_write.cpp | ||
) | ||
target_link_libraries(examplePluginEngine_write adios2::cxx11) | ||
|
||
add_executable(examplePluginEngine_read | ||
examplePluginEngine_read.cpp | ||
) | ||
target_link_libraries(examplePluginEngine_read adios2::cxx11) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,149 @@ | ||
/* | ||
* Distributed under the OSI-approved Apache License, Version 2.0. See | ||
* accompanying file Copyright.txt for details. | ||
* | ||
* ExampleReadPlugin.cpp | ||
* | ||
* Created on: Jul 5, 2021 | ||
* Author: Caitlin Ross <[email protected]> | ||
*/ | ||
|
||
#include "ExampleReadPlugin.h" | ||
#include "ExampleReadPlugin.tcc" | ||
|
||
#include "adios2/helper/adiosType.h" | ||
|
||
namespace adios2 | ||
{ | ||
namespace core | ||
{ | ||
namespace engine | ||
{ | ||
|
||
ExampleReadPlugin::ExampleReadPlugin(IO &io, const std::string &name, | ||
const Mode mode, helper::Comm comm) | ||
: PluginEngineInterface(io, name, mode, comm.Duplicate()) | ||
{ | ||
Init(); | ||
} | ||
|
||
ExampleReadPlugin::~ExampleReadPlugin() | ||
{ | ||
m_DataFile.close(); | ||
m_VarFile.close(); | ||
} | ||
|
||
Dims convertStrToDims(const std::string &str) | ||
{ | ||
Dims dims; | ||
if (str.size() > 2) | ||
{ | ||
auto vals = str.substr(1, str.size() - 2); | ||
std::stringstream ss(vals); | ||
while (ss.good()) | ||
{ | ||
std::string substr; | ||
std::getline(ss, substr, ','); | ||
dims.push_back(std::stoi(substr)); | ||
} | ||
} | ||
return dims; | ||
} | ||
|
||
void ExampleReadPlugin::Init() | ||
{ | ||
std::string dir = "ExamplePlugin"; | ||
auto paramFileNameIt = m_IO.m_Parameters.find("DirName"); | ||
if (paramFileNameIt != m_IO.m_Parameters.end()) | ||
{ | ||
dir = paramFileNameIt->second; | ||
} | ||
|
||
std::string fileName = dir + "/data.txt"; | ||
m_DataFile.open(fileName, std::ofstream::in); | ||
if (!m_DataFile) | ||
{ | ||
throw std::ios_base::failure("ExampleReadPlugin: Failed to open file " + | ||
fileName); | ||
} | ||
|
||
std::string varfName = dir + "/vars.txt"; | ||
m_VarFile.open(varfName, std::ofstream::in); | ||
if (!m_VarFile) | ||
{ | ||
throw std::ios_base::failure("ExampleReadPlugin: Failed to open file " + | ||
varfName + ".vars"); | ||
} | ||
|
||
// get var info | ||
while (m_VarFile.good()) | ||
{ | ||
std::string name, typeStr, shapeStr, startStr, countStr; | ||
std::getline(m_VarFile, name, ';'); | ||
std::getline(m_VarFile, typeStr, ';'); | ||
std::getline(m_VarFile, shapeStr, ';'); | ||
std::getline(m_VarFile, startStr, ';'); | ||
std::getline(m_VarFile, countStr); | ||
|
||
auto shape = convertStrToDims(shapeStr); | ||
auto start = convertStrToDims(startStr); | ||
auto count = convertStrToDims(countStr); | ||
|
||
const DataType type = helper::GetDataTypeFromString(typeStr); | ||
if (type == DataType::Compound) | ||
{ | ||
// not supported | ||
} | ||
#define declare_template_instantiation(T) \ | ||
else if (type == helper::GetDataType<T>()) \ | ||
{ \ | ||
AddVariable<T>(name, shape, start, count); \ | ||
} | ||
ADIOS2_FOREACH_STDTYPE_1ARG(declare_template_instantiation) | ||
#undef declare_template_instantiation | ||
} | ||
} | ||
|
||
#define declare(T) \ | ||
void ExampleReadPlugin::DoGetSync(Variable<T> &variable, T *values) \ | ||
{ \ | ||
ReadVariable(variable, values); \ | ||
} \ | ||
void ExampleReadPlugin::DoGetDeferred(Variable<T> &variable, T *values) \ | ||
{ \ | ||
ReadVariable(variable, values); \ | ||
} | ||
ADIOS2_FOREACH_STDTYPE_1ARG(declare) | ||
#undef declare | ||
|
||
StepStatus ExampleReadPlugin::BeginStep(StepMode mode, | ||
const float timeoutSeconds) | ||
{ | ||
return StepStatus::OK; | ||
} | ||
|
||
void ExampleReadPlugin::PerformGets() {} | ||
|
||
size_t ExampleReadPlugin::CurrentStep() const { return m_CurrentStep; } | ||
|
||
void ExampleReadPlugin::EndStep() { m_CurrentStep++; } | ||
|
||
void ExampleReadPlugin::DoClose(const int transportIndex) {} | ||
|
||
} // end namespace engine | ||
} // end namespace core | ||
} // end namespace adios2 | ||
|
||
extern "C" { | ||
|
||
adios2::core::engine::ExampleReadPlugin *EngineCreate(adios2::core::IO &io, | ||
const std::string &name, | ||
const adios2::Mode mode, | ||
adios2::helper::Comm comm) | ||
{ | ||
return new adios2::core::engine::ExampleReadPlugin(io, name, mode, | ||
comm.Duplicate()); | ||
} | ||
|
||
void EngineDestroy(adios2::core::engine::ExampleReadPlugin *obj) { delete obj; } | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
/* | ||
* Distributed under the OSI-approved Apache License, Version 2.0. See | ||
* accompanying file Copyright.txt for details. | ||
* | ||
* ExampleReadPlugin.h Simple file reading engine for reading files written by | ||
* the ExampleWritePlugin engine. Looks for directory | ||
* (called "ExamplePlugin" by default, but can be changed with engine parameter | ||
* "DirName") and reads variable info from vars.txt and actual data from | ||
* data.txt. | ||
* | ||
* Created on: Jul 5, 2021 | ||
* Author: Caitlin Ross <[email protected]> | ||
*/ | ||
|
||
#ifndef EXAMPLEREADPLUGIN_H_ | ||
#define EXAMPLEREADPLUGIN_H_ | ||
|
||
#include <fstream> | ||
#include <string> | ||
|
||
#include "adios2/common/ADIOSMacros.h" | ||
#include "adios2/common/ADIOSTypes.h" | ||
#include "adios2/core/IO.h" | ||
#include "adios2/engine/plugin/PluginEngineInterface.h" | ||
#include "adios2/helper/adiosComm.h" | ||
#include "adios2/helper/adiosString.h" | ||
|
||
namespace adios2 | ||
{ | ||
namespace core | ||
{ | ||
namespace engine | ||
{ | ||
|
||
/** An engine interface to be used by the plugin infrastructure */ | ||
class ExampleReadPlugin : public PluginEngineInterface | ||
{ | ||
public: | ||
ExampleReadPlugin(IO &io, const std::string &name, const Mode openMode, | ||
helper::Comm comm); | ||
virtual ~ExampleReadPlugin(); | ||
|
||
/** Indicates beginning of a step **/ | ||
StepStatus BeginStep(StepMode mode = StepMode::Read, | ||
const float timeoutSeconds = -1.0) override; | ||
|
||
/** Indicates end of a step **/ | ||
void EndStep() override; | ||
|
||
/** Return the current step **/ | ||
size_t CurrentStep() const override; | ||
|
||
/** Execute deferred mode Gets **/ | ||
void PerformGets() override; | ||
|
||
protected: | ||
void Init() override; | ||
|
||
#define declare(T) \ | ||
void DoGetSync(Variable<T> &variable, T *values) override; \ | ||
void DoGetDeferred(Variable<T> &variable, T *values) override; | ||
ADIOS2_FOREACH_STDTYPE_1ARG(declare) | ||
#undef declare | ||
|
||
void DoClose(const int transportIndex = -1) override; | ||
|
||
private: | ||
std::ifstream m_DataFile; | ||
std::ifstream m_VarFile; | ||
size_t m_CurrentStep = 0; | ||
|
||
template <typename T> | ||
void AddVariable(const std::string &name, Dims shape, Dims start, | ||
Dims count); | ||
|
||
template <class T> | ||
void ReadVariable(Variable<T> &variable, T *values); | ||
}; | ||
|
||
} // end namespace engine | ||
} // end namespace core | ||
} // end namespace adios2 | ||
|
||
extern "C" { | ||
|
||
adios2::core::engine::ExampleReadPlugin * | ||
EngineCreate(adios2::core::IO &io, const std::string &name, | ||
const adios2::Mode mode, adios2::helper::Comm comm); | ||
void EngineDestroy(adios2::core::engine::ExampleReadPlugin *obj); | ||
} | ||
|
||
#endif /* EXAMPLEREADPLUGIN_H_ */ |
Oops, something went wrong.