Merge pull request ESMCI#1465 from NCAR/ejh_doc_diagrams

adding some diagrams, fixing others

doc/images/Makefile.am

@@ -4,5 +4,6 @@
 
 # These are the images used in the documentation.
 EXTRA_DIST = block-cyclic.png block-cyclic-rearr.png dof.png	\
-dof-rearr.png PIO_Intercomm1.png PIO_Intracomm1.png		\
-PIO_Library_Architecture1.jpg PIO_Decomposition.png
+dof-rearr.png PIO_Intracomm1.png PIO_Library_Architecture1.jpg	\
+PIO_Decomposition.png I_O_on_Few.png I_O_on_Many_Intracomm.png	\
+I_O_on_Many_Async.png PIO_Async.png

doc/source/Introduction.txt

@@ -1,17 +1,54 @@
 @tableofcontents
 
-/** @page intro Introduction 
+/** @page intro Introduction
 
-PIO is a software interface layer designed to encapsolate the
-complexities of parallel IO and make it easier to replace the lower
-level software backend.  It currently supports <a
+Performing I/O is straightforward when a small number of processors
+are being used.
+
+@image html I_O_on_Few.png "I/O on One or a Few Processors"
+
+Parallel I/O does not scale to thousands of processors, because the
+parallel disk systems do not support the bandwitdh to allow thousands
+of processors to access the disk at the same time. As a result, most
+of the processors will have to wait to do I/O.
+
+An obvious solution is to designate a small number of processors to do
+I/O, and use the rest for computation.
+
+@image html I_O_on_Many_Intracomm.png "I/O on Many Processors Intracomm"
+
+PIO provides a netCDF-like API which provides this service. User code
+is written as if parallel I/O is being used from every processor, but,
+under the hood, PIO uses the I/O processors to do all data access.
+
+With Intracomm Mode, the I/O processors are a subset of the
+computational processors, and only one computational unit is
+supported. In Async Mode, the I/O processors are dedicated to I/O, and
+do not perform computation. Also, more than one computational unit may
+be designated.
+
+@image html I_O_on_Many_Async.png "I/O on Many Processors Async"
+
+The user initializes the PIO IO System, designating some processors
+for I/O, others for computation.
+
+PIO decompositions and distributed arrays allow the code to be written
+in terms of the local, distributed sub-array (see @ref decomp). PIO
+handles the stitching of all data into the correct global space in a
+netCDF variable.
+
+PIO also allows for the creation of multiple computational units. Each
+computational unit consists of many processors. I/O for all
+computational units is accomplished through one set of dedicated I/O
+processors (see @ref iosystem).
+
+PIO uses <a
 href=http://www.unidata.ucar.edu/software/netcdf/docs/html_guide/index.html#user_guide>
 netcdf </a> and <a
-href=http://trac.mcs.anl.gov/projects/parallel-netcdf> pnetcdf </a>.
-
-@image html PIO_Library_Architecture1.jpg
+href=http://trac.mcs.anl.gov/projects/parallel-netcdf> pnetcdf</a> to
+read and write the netCDF files (see @ref install).
 
-Basic description of how to optimize IO in a parallel environment:
+## Basic description of how to optimize IO in a parallel environment:
 
 PIO calls are collective.  A MPI communicator is set in a call to @ref
 PIO_init and all tasks associated with that communicator must

doc/source/faq.txt

@@ -1,16 +1,42 @@
 /*! \page faq Frequently Asked Questions 
 
-   Here is a list of frequently asked questions and their answers.
+Here is a list of frequently asked questions and their answers.
+
 <dl>
-<dt>How do I specify which tasks perform IO?  </dt>
-    <dd>This is done in the call to \ref PIO_init which has two interfaces: init_intracom and init_intercom.  
-  <ul><li> In the init_intracom interface, use the num_iotasks and stride variables to specify the total number of io tasks and the stride between them with respect to the mpi communicator, comp_comm, which is provided.  You can also use the optional base argument to shift the first IO task away from the first computational task, this is often desirable because the applications first computational task often has higher memory requirements than other tasks.   IO tasks are a subset of the tasks defined in comp_comm.
-<li> In the init_intercom interface, IO tasks are a disjoint set of tasks from those in the computational communicator.  <b>This interface is still experimental and not recommended for production use at this time. </b>
+
+<dt>How do I specify which tasks perform IO?</dt>
+
+<dd>This is done in the call to \ref PIO_init which has two
+interfaces: init_intracom and init_intercom.
+
+  <ul>
+
+    <li> In the init_intracom interface, use the num_iotasks and
+    stride variables to specify the total number of io tasks and the
+    stride between them with respect to the mpi communicator,
+    comp_comm, which is provided.  You can also use the optional base
+    argument to shift the first IO task away from the first
+    computational task, this is often desirable because the
+    applications first computational task often has higher memory
+    requirements than other tasks.  IO tasks are a subset of the tasks
+    defined in comp_comm.
+
+    <li> In the init_intercom interface, IO tasks are a disjoint set
+    of tasks from those in the computational communicator.
+
 </ul>
-Note that num_iotasks is the maximum number of IO tasks to use for an IO operation.   The size of the field being read or written along with the tunable blocksize parameter, \ref PIO_set_blocksize, determines the actual number of tasks used for a given IO operation.
+
+Note that num_iotasks is the maximum number of IO tasks to use for an
+IO operation.  The size of the field being read or written along with
+the tunable blocksize parameter, \ref PIO_set_blocksize, determines
+the actual number of tasks used for a given IO operation.
 </dd>
-<dt>How do I test if PIO is installed and working correctly? </dt> 
-    <dd>The PIO Library distribution contains tests for PIO. They are run my 'make check'. The tests use mpiexec to run tests on 4, 8, or 16 processors. </dd>
+
+<dt>How do I test if PIO is installed and working correctly?</dt> 
+
+<dd>The PIO Library distribution contains tests for PIO. They are run
+my 'make check'. The tests use mpiexec to run tests on 4, 8, or 16
+processors. </dd>
 
 </dl>
 */

doc/source/iosystem.txt

@@ -12,13 +12,13 @@ When the user program is complete, the IOSystem should be released by
 calling C function PIOc_finalize() or Fortran function piolib_mod::finalize()
 for each open IOSystem.
 
-@section intercomm_mode Intercomm Mode
+@section intracomm_mode Intracomm Mode
 
-@image html PIO_Intercomm1.png "PIO Intercomm Mode"
+@image html PIO_Intracomm1.png "PIO Intracomm Mode"
 
 @section async_mode Async Mode
 
-@image html PIO_Intracomm1.png "PIO Async Mode"
+@image html PIO_Async.png "PIO Async Mode"
 
 */
 

src/clib/pioc.c

@@ -1210,7 +1210,8 @@ PIOc_finalize(int iosysid)
     /* Free the MPI communicators. my_comm is just a copy (but not an
      * MPI copy), so does not have to have an MPI_Comm_free()
      * call. comp_comm and io_comm are MPI duplicates of the comms
-     * handed into init_intercomm. So they need to be freed by MPI. */
+     * handed into PIOc_init_async(). So they need to be freed by
+     * MPI. */
     if (ios->intercomm != MPI_COMM_NULL)
         MPI_Comm_free(&ios->intercomm);
     if (ios->union_comm != MPI_COMM_NULL)
@@ -1732,7 +1733,7 @@ PIOc_init_async(MPI_Comm world, int num_io_procs, int *io_proc_list,
     if ((ret = MPI_Group_free(&world_group)))
         return check_mpi(NULL, NULL, ret, __FILE__, __LINE__);
 
-    LOG((2, "successfully done with PIO_Init_Async"));
+    LOG((2, "successfully done with PIOc_init_async"));
     return PIO_NOERR;
 }