Explain use of data pack/unpack for shipping regex values

Signed-off-by: Ralph Castain <[email protected]>

Chap_API_Data_Mgmt.tex

@@ -6,16 +6,48 @@ \chapter{Data Packing and Unpacking}
 
 \ac{PMIx} intentionally does not include support for internode communications in the standard, instead relying on its host \ac{SMS} environment to transfer any needed data and/or requests between nodes. These operations frequently involve PMIx-defined public data structures that include binary data. Many \ac{HPC} clusters are homogeneous, and so transferring the structures can be done rather simply. However, greater effort is required in heterogeneous environments to ensure binary data is correctly transferred. \ac{PMIx} buffer manipulation functions are provided for this purpose via standardized interfaces to ease adoption.
 
+%%%%%%%%%%%
+\section{Data Buffer Type}
+\declarestruct{pmix_data_buffer_t}
+
+The \refstruct{pmix_data_buffer_t} structure describes a data buffer used for packing and unpacking.
+
+\versionMarker{2.0}
+\cspecificstart
+\begin{codepar}
+typedef struct pmix_data_buffer \{
+    /** Start of my memory */
+    char *base_ptr;
+    /** Where the next data will be packed to
+        (within the allocated memory starting
+        at base_ptr) */
+    char *pack_ptr;
+    /** Where the next data will be unpacked
+        from (within the allocated memory
+        starting as base_ptr) */
+    char *unpack_ptr;
+    /** Number of bytes allocated (starting
+        at base_ptr) */
+    size_t bytes_allocated;
+    /** Number of bytes used by the buffer
+        (i.e., amount of data -- including
+        overhead -- packed in the buffer) */
+    size_t bytes_used;
+\} pmix_data_buffer_t;
+\end{codepar}
+\cspecificend
+
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Support Macros}
-\label{chap:api_init:macros}
+\label{chap:datamgt:macros}
 
 \ac{PMIx} provides a set of convenience macros for creating, initiating, and releasing data buffers.
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_CREATE}}
-\declareapi{PMIX_DATA_BUFFER_CREATE}
+\declaremacro{PMIX_DATA_BUFFER_CREATE}
 
 %%%%
 \summary
@@ -44,7 +76,7 @@ \subsection{\code{PMIX_DATA_BUFFER_CREATE}}
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_RELEASE}}
-\declareapi{PMIX_DATA_BUFFER_RELEASE}
+\declaremacro{PMIX_DATA_BUFFER_RELEASE}
 
 %%%%
 \summary
@@ -73,7 +105,7 @@ \subsection{\code{PMIX_DATA_BUFFER_RELEASE}}
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_CONSTRUCT}}
-\declareapi{PMIX_DATA_BUFFER_CONSTRUCT}
+\declaremacro{PMIX_DATA_BUFFER_CONSTRUCT}
 
 %%%%
 \summary
@@ -102,7 +134,7 @@ \subsection{\code{PMIX_DATA_BUFFER_CONSTRUCT}}
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_DESTRUCT}}
-\declareapi{PMIX_DATA_BUFFER_DESTRUCT}
+\declaremacro{PMIX_DATA_BUFFER_DESTRUCT}
 
 %%%%
 \summary
@@ -131,7 +163,7 @@ \subsection{\code{PMIX_DATA_BUFFER_DESTRUCT}}
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_LOAD}}
-\declareapi{PMIX_DATA_BUFFER_LOAD}
+\declaremacro{PMIX_DATA_BUFFER_LOAD}
 
 %%%%
 \summary
@@ -162,7 +194,7 @@ \subsection{\code{PMIX_DATA_BUFFER_LOAD}}
 
 %%%%%%%%%%%
 \subsection{\code{PMIX_DATA_BUFFER_UNLOAD}}
-\declareapi{PMIX_DATA_BUFFER_UNLOAD}
+\declaremacro{PMIX_DATA_BUFFER_UNLOAD}
 
 %%%%
 \summary
@@ -243,7 +275,7 @@ \subsection{\code{PMIx_Data_pack}}
 \descr
 
 The pack function packs one or more values of a specified type into the specified buffer.  The buffer must have already been
-initialized via the \refapi{PMIX_DATA_BUFFER_CREATE} or \refapi{PMIX_DATA_BUFFER_CONSTRUCT}
+initialized via the \refmacro{PMIX_DATA_BUFFER_CREATE} or \refmacro{PMIX_DATA_BUFFER_CONSTRUCT}
 macros --- otherwise, \refapi{PMIx_Data_pack} will return an error.
 Providing an unsupported type flag will likewise be reported as an error.
 
@@ -311,7 +343,7 @@ \subsection{\code{PMIx_Data_unpack}}
 %%%%
 \descr
 
-The unpack function unpacks the next value (or values) of a specified type from the given buffer. The buffer must have already been initialized via an \refapi{PMIX_DATA_BUFFER_CREATE} or \refapi{PMIX_DATA_BUFFER_CONSTRUCT} call (and assumedly filled with some data) --- otherwise, the unpack_value function will return an error. Providing an unsupported type flag will likewise be reported as an error, as will specifying a data type that \textit{does not} match the type of the next item in the buffer. An attempt to read beyond the end of the stored data held in the buffer will also return an error.
+The unpack function unpacks the next value (or values) of a specified type from the given buffer. The buffer must have already been initialized via an \refmacro{PMIX_DATA_BUFFER_CREATE} or \refmacro{PMIX_DATA_BUFFER_CONSTRUCT} call (and assumedly filled with some data) --- otherwise, the unpack_value function will return an error. Providing an unsupported type flag will likewise be reported as an error, as will specifying a data type that \textit{does not} match the type of the next item in the buffer. An attempt to read beyond the end of the stored data held in the buffer will also return an error.
 
 NOTE: it is possible for the buffer to be corrupted and that \ac{PMIx} will \textit{think} there is a proper variable type at the beginning of an unpack region --- but that the value is bogus (e.g., just a byte field in a string array that so happens to have a value that matches the specified data type flag). Therefore, the data type error check is \textit{not} completely safe.
 

Chap_API_Server.tex

@@ -45,9 +45,11 @@ \subsection{\code{PMIx_generate_regex}}
 
 Given a comma-separated list of \refarg{input} values, generate a reduced size representation of the input that can be passed down to the \ac{PMIx} server library's \refapi{PMIx_server_register_nspace} \ac{API} for parsing. The order of the individual values in the \refarg{input} string is preserved across the operation. The caller is responsible for releasing the returned data.
 
-\adviceuserstart
-The returned representation may be an arbitrary array of bytes as opposed to a valid NULL-terminated string. However, the method used to generate the representation shall be identified with a colon-delimited string at the beginning of the output. For example, an output starting with \code{"pmix:"} indicates that the representation is a \ac{PMIx}-defined regular expression. In contrast, an output starting with \code{"blob:"} is a compressed binary array.
-\adviceuserend
+\advicermstart
+The returned representation may be an arbitrary array of bytes as opposed to a valid NULL-terminated string. However, the method used to generate the representation shall be identified with a colon-delimited string at the beginning of the output. For example, an output starting with \code{"pmix:"} indicates that the representation is a \ac{PMIx}-defined regular expression represented as a NULL-terminated string. In contrast, an output starting with \code{"blob:\textbackslash0size=1234:"} is a compressed binary array.
+
+Communicating the resulting output should be done by first packing the returned expression using the \refapi{PMIx_Data_pack}, declaring the input to be of type \refconst{PMIX_REGEX}, and then obtaining the blob to be communicated using the \refmacro{PMIX_DATA_BUFFER_UNLOAD} macro. The pack/unpack routines will ensure proper handling of the data based on the regex prefix.
+\advicermend
 
 
 %%%%%%%%%%%
@@ -81,9 +83,11 @@ \subsection{\code{PMIx_generate_ppn}}
 
 The input shall consist of a semicolon-separated list of ranges representing the ranks of processes on each node of the job - e.g.,  "1-4;2-5;8,10,11,12;6,7,9". Each field of the input must correspond to the node name provided at that position in the input to \refapi{PMIx_generate_regex}. Thus, in the example, ranks 1-4 would be located on the first node of the comma-separated list of names provided to \refapi{PMIx_generate_regex}, and ranks 2-5 would be on the second name in the list.
 
-\adviceuserstart
-The returned representation may be an arbitrary array of bytes as opposed to a valid NULL-terminated string. However, the method used to generate the representation shall be identified with a colon-delimited string at the beginning of the output. For example, an output starting with \code{"pmix:"} indicates that the representation is a \ac{PMIx}-defined regular expression. In contrast, an output starting with \code{"blob:"} is a compressed binary array.
-\adviceuserend
+\advicermstart
+The returned representation may be an arbitrary array of bytes as opposed to a valid NULL-terminated string. However, the method used to generate the representation shall be identified with a colon-delimited string at the beginning of the output. For example, an output starting with \code{"pmix:"} indicates that the representation is a \ac{PMIx}-defined regular expression represented as a NULL-terminated string. In contrast, an output starting with \code{"blob:\textbackslash0size=1234:"} is a compressed binary array.
+
+Communicating the resulting output should be done by first packing the returned expression using the \refapi{PMIx_Data_pack}, declaring the input to be of type \refconst{PMIX_REGEX}, and then obtaining the blob to be communicated using the \refmacro{PMIX_DATA_BUFFER_UNLOAD} macro. The pack/unpack routines will ensure proper handling of the data based on the regex prefix.
+\advicermend
 
 %%%%%%%%%%%
 \subsection{\code{PMIx_server_register_nspace}}

Chap_API_Struct.tex

@@ -2507,143 +2507,7 @@ \subsubsection{Load a\refstruct{pmix_byte_object_t} structure}
 \end{arglist}
 
 
-%%%%%%%%%%%
-\subsection{Data Buffer Type}
-\declarestruct{pmix_data_buffer_t}
-
-The \refstruct{pmix_data_buffer_t} structure describes a data buffer used for packing and unpacking.
-
-\versionMarker{2.0}
-\cspecificstart
-\begin{codepar}
-typedef struct pmix_data_buffer \{
-    /** Start of my memory */
-    char *base_ptr;
-    /** Where the next data will be packed to
-        (within the allocated memory starting
-        at base_ptr) */
-    char *pack_ptr;
-    /** Where the next data will be unpacked
-        from (within the allocated memory
-        starting as base_ptr) */
-    char *unpack_ptr;
-    /** Number of bytes allocated (starting
-        at base_ptr) */
-    size_t bytes_allocated;
-    /** Number of bytes used by the buffer
-        (i.e., amount of data -- including
-        overhead -- packed in the buffer) */
-    size_t bytes_used;
-\} pmix_data_buffer_t;
-\end{codepar}
-\cspecificend
-
-\subsection{Data buffer support macros}
-The following macros support the \refstruct{pmix_data_buffer_t} structure.
-
-\subsubsection{Initialize the \refstruct{pmix_data_buffer_t} structure}
-\declaremacro{PMIX_DATA_BUFFER_CONSTRUCT}
-
-Initialize the \refstruct{pmix_data_buffer_t} fields
-
-\versionMarker{2.0}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_CONSTRUCT(m)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\argin{m}{Pointer to the structure to be initialized (pointer to \refstruct{pmix_data_buffer_t})}
-\end{arglist}
-
-\subsubsection{Destruct the \refstruct{pmix_data_buffer_t} structure}
-\declaremacro{PMIX_DATA_BUFFER_DESTRUCT}
-
-Clear the \refstruct{pmix_data_buffer_t} fields
-
-\versionMarker{2.0}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_DESTRUCT(m)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\argin{m}{Pointer to the structure to be destructed (pointer to \refstruct{pmix_data_buffer_t})}
-\end{arglist}
-
-\subsubsection{Create a \refstruct{pmix_data_buffer_t} structure}
-\declaremacro{PMIX_DATA_BUFFER_CREATE}
-
-Allocate and intitialize a \refstruct{pmix_data_buffer_t} structure
-
-\versionMarker{2.0}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_CREATE(m)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\arginout{m}{Address where the pointer to the \refstruct{pmix_data_buffer_t} structure shall be stored (handle)}
-\end{arglist}
-
-\subsubsection{Free a \refstruct{pmix_data_buffer_t}}
-\declaremacro{PMIX_DATA_BUFFER_RELEASE}
-
-Release a \refstruct{pmix_data_buffer_t} structure
-
-\versionMarker{2.0}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_RELEASE(m)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\argin{m}{Pointer to the \refstruct{pmix_data_buffer_t} structure to be released (handle)}
-\end{arglist}
-
-\subsubsection{Load a \refstruct{pmix_data_buffer_t}}
-\declaremacro{PMIX_DATA_BUFFER_LOAD}
-
-Load data into a \refstruct{pmix_data_buffer_t} structure
-
-\versionMarker{2.2}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_LOAD(b, d, s)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\argin{b}{Pointer to the \refstruct{pmix_data_buffer_t} structure to be loaded (handle)}
-\argin{d}{Pointer to the data to be loaded into \refarg{b} (\code{void*})}
-\argin{s}{Number of bytes in \refarg{d} (\code{size_t})}
-\end{arglist}
-
-
-\subsubsection{Unload a \refstruct{pmix_data_buffer_t}}
-\declaremacro{PMIX_DATA_BUFFER_UNLOAD}
-
-Unload the data from a \refstruct{pmix_data_buffer_t} structure
-
-\versionMarker{2.2}
-\cspecificstart
-\begin{codepar}
-PMIX_DATA_BUFFER_UNLOAD(b, d, s)
-\end{codepar}
-\cspecificend
-
-\begin{arglist}
-\argin{b}{Pointer to the \refstruct{pmix_data_buffer_t} structure to be unloaded (handle)}
-\arginout{d}{Pointer to be set to the data region after unloading (\code{void*})}
-\arginout{s}{Variable to be set to the number of bytes in the returned data region (\code{size_t})}
-\end{arglist}
-
-
-%%%%%%%%%%%
+%%%%%%%%%%
 \subsection{Data Array Structure}
 \declarestruct{pmix_data_array_t}
 
@@ -2882,9 +2746,17 @@ \section{Generalized Data Types Used for Packing/Unpacking}
 \declareconstitem{PMIX_ENVAR}
 Environmental variable structure (\refstruct{pmix_envar_t})
 %
+\declareconstitemNEW{PMIX_COORD}
+Structure containing network coordinates (\refstruct{pmix_coord_t})
+%
 \declareconstitemNEW{PMIX_REGATTR}
 Structure supporting attribute registrations (\refstruct{pmix_regattr_t})
 %
+%
+\declareconstitemNEW{PMIX_REGEX}
+Regular expressions - can be a valid NULL-terminated string or an arbitrary array of bytes
+%
+
 \end{constantdesc}
 
 %%%%%%%%%%%