diff --git a/docs/source/api/attribute_api.rst b/docs/source/api/attribute_api.rst index de01492..7811c79 100644 --- a/docs/source/api/attribute_api.rst +++ b/docs/source/api/attribute_api.rst @@ -1,8 +1,24 @@ =========== -Attribute +Attributes =========== -In the library, netCDF attributes can be created, accessed, and manipulated -using python dictionary-like syntax. A Pythonic interface for metadata operations -is provided both in the ``File`` class (for global attributes) and the -``Variable`` class (for variable attributes). +In `object-oriented programming `_, +a class contains fields (state variables containing data) and methods +(subroutines or procedures defining the object's behavior in code). ``Fields`` +may also be known as members, attributes, or properties. To avoid confusion +with NetCDF's terminology of ``attribute``, this document uses `field` to refer +to a class's state variable. + +NetCDF attributes are small, supplementary metadata that annotates variables or +files. NetCDF attribute is not a Python class by itself. Instead, it is a +field of python dictionary in class :class:`pnetcdf.File` and class +:class:`pnetcdf.Variable`. Their data types can be any allowed by the classic +NetCDF file formats. The most common data type is `text` for annotation +purpose. NetCDF attributes can be created, accessed, and manipulated using +python dictionary-like syntax. An attribute can be associated to a file, +referred to as ``golbal attribute``, as well as to individual variables, +referred to as ``variable's attribute``. Pythonic interfaces for accessing +attributes are is provided both in class :class:`pnetcdf.File` (for global +attributes) and class :class:`pnetcdf.Variable` (for variable attributes). +Example programs are `examples/global_attribute.py` and `examples/put_var.py`. + diff --git a/docs/source/api/dimension_api.rst b/docs/source/api/dimension_api.rst index 65e5c29..b555aca 100644 --- a/docs/source/api/dimension_api.rst +++ b/docs/source/api/dimension_api.rst @@ -1,22 +1,23 @@ ============== -Dimension +Dimensions ============== -Dimension defines the shape and structure of variables and stores coordinate -data for multidimensional arrays. The ``Dimension`` object, which is also a key -component of ``File`` class, provides an interface to access dimensions. +Class ``Dimension`` is used to define the shape of NetCDF variables. In NetCDF, +a variable, an instance of :class:`pnetcdf.Variable`, is a multi-dimensional +array. Methods in :class:`pnetcdf.Dimension` provide an interface to access +dimensions objects stored in the file. .. autoclass:: pnetcdf::Dimension :members: getfile, isunlimited :exclude-members: name, size -Read-only Python Attributes of Dimension Class - The following class members are read-only and should not be modified by the +Read-only python fields of class :class:`pnetcdf.Dimension` + The following class fields are read-only and should not be modified by the user. .. attribute:: name - String name of Dimension instance. This class member is read-only and + String name of Dimension instance. This class field is read-only and should not be modified by the user. To rename a dimension, use :meth:`File.rename_dim` method. @@ -24,7 +25,8 @@ Read-only Python Attributes of Dimension Class .. attribute:: size - The current size of Dimension (calls ``len`` on Dimension instance). + The current size of Dimension (its value can be obtained by calling + python function ``len()`` on the Dimension instance). **Type:** `int` diff --git a/docs/source/api/file_api.rst b/docs/source/api/file_api.rst index 80f5a2d..3e71f66 100644 --- a/docs/source/api/file_api.rst +++ b/docs/source/api/file_api.rst @@ -1,12 +1,12 @@ ================ -File +Files ================ -``pnetcdf.File`` is a high-level object representing an netCDF file, -which provides a Pythonic interface to create, read and write within -an netCDF file. A File object serves as the root container for dimensions, -variables, and attributes. Together they describe the meaning of data and -relations among data fields stored in a netCDF file. +An instance of class ``pnetcdf.File`` is a high-level object representing a +netCDF file. The class methods provide a set of Pythonic interfaces to create, +read and write a netCDF file. A ``File`` instance serves as the root container +for dimensions, variables, and attributes. Together they describe the meaning +of data and relations among data objects stored in a netCDF file. .. autoclass:: pnetcdf::File :members: __init__, close, filepath, redef, enddef, begin_indep, end_indep, @@ -17,21 +17,21 @@ relations among data fields stored in a netCDF file. inq_header_size, inq_put_size, inq_header_extent, inq_nreqs :exclude-members: dimensions, variables, file_format, indep_mode, path -Read-only Python Attributes of File Class - The following class members are read-only and should not be modified by the +Read-only python fields of class :class:`pnetcdf.File` + The following class fields are read-only and should not be modified by the user. .. attribute:: dimensions The dimensions dictionary maps the names of dimensions defined in this - file as an instance of the ``pnetcdf.Dimension`` class. + file as an instance of the :class:`pnetcdf.Dimension`. **Type:** `dict` .. attribute:: variables The variables dictionary maps the names of variables defined in this file - as an instance of the ``pnetcdf.Variable`` class. + as an instance of the :class:`pnetcdf.Variable`. **Type:** `dict` diff --git a/docs/source/api/function_api.rst b/docs/source/api/function_api.rst index 07acfaa..4cb4827 100644 --- a/docs/source/api/function_api.rst +++ b/docs/source/api/function_api.rst @@ -1,7 +1,10 @@ ================ -Utility Functions +Other pnetcdf class methods ================ + PnetCDF class methods listed below are not associated with particular + instances of ``File`` or ``Variable``. + .. autofunction:: pnetcdf::libver .. autofunction:: pnetcdf::strerror .. autofunction:: pnetcdf::strerrno diff --git a/docs/source/api/variable_api.rst b/docs/source/api/variable_api.rst index 350906b..fcfdf71 100644 --- a/docs/source/api/variable_api.rst +++ b/docs/source/api/variable_api.rst @@ -1,14 +1,16 @@ ========= -Variable +Variables ========= -Variable is a core component of a netCDF file representing an array of data +``Variable`` is a core component of a netCDF file representing an array of data values organized along one or more dimensions, with associated metadata in the -form of attributes. The ``Variable`` object in the library provides operations -to read and write the data and metadata of a variable within a netCDF file. -Particularly, data mode operations have a flexible interface, where reads and -writes can be done through either explicit function-call style methods or -indexer-style (numpy-like) syntax. +form of attributes. An instance of class :class:`pnetcdf.Variable` represents a +NetCDF variable stored in the file. The class methods provide I/O operations to +read and write the data and metadata of a NetCDF variable. + +Reading and writing a subarray of a variable can be done through either +explicit function-call style methods or Python indexer-style (numpy-like) +syntax. .. autoclass:: pnetcdf::Variable :members: ncattrs, put_att, get_att, del_att, rename_att, get_dims, @@ -18,8 +20,8 @@ indexer-style (numpy-like) syntax. chartostring -Read-only Python Attributes of Variable Class - The following class members are read-only and should not be modified +Read-only python fields of class :class:`pnetcdf.Variable` + The following class fields are read-only and should not be modified directly by the user. .. attribute:: name diff --git a/docs/copyright.rst b/docs/source/copyright.rst similarity index 98% rename from docs/copyright.rst rename to docs/source/copyright.rst index ef12b1c..6ae10ac 100644 --- a/docs/copyright.rst +++ b/docs/source/copyright.rst @@ -2,10 +2,8 @@ Copyright Statement ================ -:: - - Copyright (c) 2024 Northwestern University and Argonne National - Laboratory All rights reserved. + Copyright (c) 2024 Northwestern University and Argonne National Laboratory + All rights reserved. Portions of this software were developed by the Unidata Program at the University Corporation for Atmospheric Research. diff --git a/docs/source/index.rst b/docs/source/index.rst index 920fcd5..6323fc6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,16 +3,17 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -PnetCDF-Python Documentation +PnetCDF-Python User Guide ============================ **Release:** |release| -.. toctree:: - :maxdepth: 2 - :caption: Introduction - - introduction/overview +PnetCDF-python is a Python interface to PnetCDF, a high-performance parallel +I/O library for accessing netCDF files. This integration with Python allows +for easy manipulation, analysis, and visualization of netCDF data using the +rich ecosystem of Python's scientific computing libraries, making it a valuable +tool for python-based applications that require high-performance access to +netCDF files. .. toctree:: :maxdepth: 2 @@ -32,16 +33,17 @@ PnetCDF-Python Documentation .. toctree:: :maxdepth: 3 - :caption: API Documentation + :caption: API Reference api/file_api api/dimension_api api/variable_api + api/attribute_api api/function_api .. toctree:: - :maxdepth: 2 - :caption: Copyright Statement + :maxdepth: 1 + :caption: Copyright copyright diff --git a/docs/source/tutorial/basic.rst b/docs/source/tutorial/basic.rst index b80c737..8430a7b 100644 --- a/docs/source/tutorial/basic.rst +++ b/docs/source/tutorial/basic.rst @@ -6,12 +6,12 @@ Basics Running Python scripts with MPI ------------------------------- - Python programs with PnetCDF-Python can be run with the command - :program:`mpiexec`. In practice, running Python programs looks like: + Python programs using PnetCDF-Python can be run with the command + :program:`mpiexec`. In practice, running a Python program looks like: - $ mpiexec -n 4 Python script.py + $ mpiexec -n 4 python script.py - to run the program with 4 processors. + to run the program with 4 MPI processes. Creating/Opening/Closing a netCDF file -------------------------------------- @@ -27,24 +27,24 @@ Creating/Opening/Closing a netCDF file Closing the netCDF file is accomplished via the :meth:`File.close` method of the ``File`` instance. - Here's an example: + Here is an example of creating a new file: .. code-block:: Python - from pnetcdf import File from mpi4py import MPI - comm = MPI.COMM_WORLD - f = File(filename="testfile.nc", mode='w', comm=comm, info=None) + import pnetcdf + + f = pnetcdf.File(filename="testfile.nc", mode='w', comm=MPI.COMM_WORLD, info=None) f.close() - Equivalent example codes in ``netCDF4-python``: + Equivalent example codes when using ``netCDF4-python``: .. code-block:: Python from mpi4py import MPI - from netCDF4 import Dataset - comm = MPI.COMM_WORLD - f = Dataset(filename="testfile.nc", mode="w", comm=comm, parallel=True) + import netCDF4 + + f = netCDF4.Dataset(filename="testfile.nc", mode="w", comm=MPI.COMM_WORLD, parallel=True) f.close() For the full example program, see ``examples/craete_open.py``. @@ -54,11 +54,11 @@ Dimensions NetCDF variables are multi-dimensional arrays. Before creating any variables, the dimensions they depend on must be established. To create a dimension, the - :meth:`File.def_dim` method is called on a File instance under define mode. + :meth:`File.def_dim` method is called on a ``File`` instance under define mode. The dimension's name is set using a Python string, while the size is defined using an integer value. To create an unlimited dimension (a dimension that can - be expanded), the size can be omitted or assigned as -1. A "Dimension" object - will be returned as a handler for this dimension. + be expanded), the parameter size can be omitted or assigned as -1. A + ``Dimension`` instance will be returned as a handler for this dimension. Here's an example (same if using netcdf4-python): @@ -70,8 +70,8 @@ Dimensions lat_dim = f.def_dim(LAT_NAME, LAT_LEN) time_dim = f.def_dim(TIME_NAME, -1) - All of the Dimension instances are stored in a dictionary as an Python - attribute of File. + All of the ``Dimension`` instances are stored in a Python dictionary as an + attribute of ``File``. .. code-block:: Python @@ -80,7 +80,8 @@ Dimensions To retrieve the previous defined dimension instance from the file, you can directly index the dictionary using variable name as the key. The dimension - information can be retrieved using following functions. + information can be retrieved using following python functions or ``Dimension`` + class methods. .. code-block:: Python @@ -94,13 +95,13 @@ Variables ------------ NetCDF variables are similar to multidimensional array objects in Python - provided by the numpy module. To define a netCDF variable, you can utilize the - :meth:`File.def_var` method within a File instance under define mode. The - mandatory arguments for this methods include the variable name (a string in - Python) and dimensions (either a tuple of dimension names or dimension + provided by the ``numpy`` module. To define a netCDF variable, you can utilize + the :meth:`File.def_var` method within a ``File`` instance under define mode. + The mandatory arguments for this methods include the variable name (a string + in Python) and dimensions (either a tuple of dimension names or dimension instances). In addition, the user need to specify the datatype of the variable - using module-level NC constants (e.g. pnetcdf.NC_INT). The supported - data types given each file format can be found :ref:`here`. + using module-level constants (e.g. ``pnetcdf.NC_INT``). The supported data + types given each file format can be found :ref:`here`. Here's an example (same if using netcdf4-python): @@ -132,7 +133,7 @@ Attributes In a netCDF file, there are two types of attributes: global attributes and variable attributes. Global attributes are usually related to the netCDF file as a whole and may be used for purposes such as providing a title or - processing history for a netCDF file. Variable's attributes are used to + processing history for a netCDF file. ``Variable``'s attributes are used to specify properties related to the variable, such as units, special values, maximum and minimum valid values, and annotation. @@ -144,11 +145,11 @@ Attributes .. code-block:: Python # set global attributes - f.floatatt = math.pi # Option1: Python attribute assignment - f.put_att("intatt", np.int32(1)) # Option2: method put_att() + f.floatatt = math.pi # Option 1: Python attribute assignment + f.put_att("intatt", np.int32(1)) # Option 2: method put_att() f.seqatt = np.int32(np.arange(10)) - # set variable attributes + # write variable attributes var = f.variables['var'] var.floatatt = math.pi var.put_att("int_att", np.int32(1)) @@ -159,8 +160,8 @@ Attributes .. code-block:: Python # set root group attributes - f.floatatt = math.pi # Option1: Python attribute assignment - f.setncattr("intatt", np.int32(1)) # Option2: method setncattr() + f.floatatt = math.pi # Option 1: Python attribute assignment + f.setncattr("intatt", np.int32(1)) # Option 2: method setncattr() f.seqatt = np.int32(np.arange(10)) # set variable attributes @@ -169,10 +170,10 @@ Attributes var.setncattr("int_att", np.int32(1)) var.seqatt = np.int32(np.arange(10)) - The :meth:`File.ncattrs` method of a File or Variable instance can be used to - retrieve the names of all the netCDF attributes. And the __dict__ attribute of - a File or Variable instance provides all the netCDF attribute name/value pairs - in a python dictionary: + The :meth:`File.ncattrs` method of a ``File`` or ``Variable`` instance can be + used to retrieve the names of all the netCDF attributes. And the __dict__ + attribute of a ``File`` or ``Variable`` instance provides all the netCDF + attribute name/value pairs in a python dictionary: .. code-block:: Python @@ -184,14 +185,14 @@ Attributes For the full example program, see ``examples/global_attributes.py``. -Writing to variable +Writing to a variable -------------------- Once a netCDF variable instance is created, writing the variable must be done while the file is in data mode. Then for writing, there are two options: -Option1 Indexer (or slicing) syntax - You can just treat the variable like an numpy array and assign data +Option 1 Indexer (or slicing) syntax + You can just treat the variable like an ``numpy`` array and assign data to a slice. Slices are specified as a `start:stop:step` triplet. .. code-block:: Python @@ -202,9 +203,9 @@ Option1 Indexer (or slicing) syntax The indexer syntax is the same as in ``netcdf4-python`` library for writing to netCDF variable. -Option2 Method calls of put_var()/get_var() - Alternatively you can also leverage Variable.put/get_var() method of a - Variable instance to perform I/O according to specific access pattern needs. +Option 2 Method calls of put_var()/get_var() + Alternatively you can also leverage ``Variable.put/get_var()`` method of a + ``Variable`` instance to perform I/O according to specific access pattern needs. Here is the example below to write an array to the netCDF variable. The part of the netCDF variable to write is specified by giving a corner (`start`) and @@ -218,7 +219,7 @@ Option2 Method calls of put_var()/get_var() # The above line is equivalent to var[10:20, 0:50] = buff -Reading from variable +Reading from a variable ---------------------- Symmetrically, users can use two options with different syntaxes to retrieve @@ -228,10 +229,10 @@ Reading from variable .. code-block:: Python var = f.variables['var'] - # Option1 Indexer: read the topleft 10*10 corner from variable var + # Option 1 Indexer: read the top-left 10*10 corner from variable var buf = var[:10, :10] - # Option2 Method Call: equivalent to var[10:20, 0:50] + # Option 2 Method Call: equivalent to var[10:20, 0:50] buf = var.get_var_all(start = [10, 0], count = [10, 50]) Similarly, :meth:`Variable.get_var` takes the same set of optional arguments diff --git a/docs/source/tutorial/compare_netcdf4.rst b/docs/source/tutorial/compare_netcdf4.rst index 2e30402..cf88605 100644 --- a/docs/source/tutorial/compare_netcdf4.rst +++ b/docs/source/tutorial/compare_netcdf4.rst @@ -20,22 +20,21 @@ Difference in Programming Model Data/Define Mode NetCDF4-python library automatically switches between data and define mode - for the user by calling ``redef`` and ``enddef`` internally within the - define-mode operation functions. For performance reason, this is **not** - adopted in Pnetcdf-python. A manual call to :meth:`File.redef` is compulsory - to re-enter the define mode, following the C library convention. Similarly, - :meth:`File.enddef` is required before switching to data mode operations. - This design is based on considerations of the following aspects: + by calling ``redef`` and ``enddef`` internally. For performance reason, this + is **not** adopted in Pnetcdf-python. A manual call to :meth:`File.redef` is + compulsory to re-enter the define mode, following the C library convention. + Similarly, :meth:`File.enddef` is required before switching to data mode + operations. This design is based on considerations of the following aspects: - Minimize overheads during consecutive define operations: Automatically - wrapping all define functions with :meth:`File.redef` and + wrapping all define methods with :meth:`File.redef` and :meth:`File.enddef` could introduce significant overhead between consecutive define operations. The netCDF4-python approach results in unnecessary data/define mode switches, impacting performance. - Avoid potential hanging when performing independent I/O: if - :meth:`File.enddef` is automatically embedded in all data mode operation - functions, the program will hang when partial processes are performing + :meth:`File.enddef` is automatically embedded in all data mode + methods, the program will hang when partial processes are performing independent I/O (while others don't) because :meth:`File.enddef` is a collective call which requires all processes to participate. diff --git a/docs/source/tutorial/datatypes.rst b/docs/source/tutorial/datatypes.rst index 205c857..e96ae67 100644 --- a/docs/source/tutorial/datatypes.rst +++ b/docs/source/tutorial/datatypes.rst @@ -26,7 +26,7 @@ NetCDF Variable Data Types +-------+----------------+-------+----------------------------------------+---------------------+ - New data types supported in CDF-5 format: + Additional data types supported in CDF-5 format: +---------------------+----------------+-------+----------------------------------------+---------------------+ | Type | C #define | Bits | Intent of use | Numpy Equivalent | diff --git a/docs/source/tutorial/non_blocking.rst b/docs/source/tutorial/non_blocking.rst index 2db1b63..1b36709 100644 --- a/docs/source/tutorial/non_blocking.rst +++ b/docs/source/tutorial/non_blocking.rst @@ -14,14 +14,14 @@ Nonblocking Write Write requests can be posted by the method call of :meth:`Variable.iput_var`. Same as :meth:`Variable.put_var`, the behavior of :meth:`Variable.iput_var` - varies depending on the pattern of provided optional arguments - `index`, - `start`, `count`, `stride`, and `imap` as shown below. Note that the method - only posts the request, which is not committed until :meth:`File.wait`. The - method call returns a request id that can be optionally passed to - :meth:`File.wait` to select this request. + varies depending on the pattern of provided optional arguments - `start`, + `count`, `stride`, and `imap` as shown below. Note that the method only posts + the request, which is not committed until :meth:`File.wait`. The method call + returns a request id that can be optionally passed to :meth:`File.wait` to + select this request. - `data` - Request to write an entire variable - - `data`, `index` - Request to write a single data value + - `data`, `start` - Request to write a single data value - `data`, `start`, `count` - Request to write an array of values - `data`, `start`, `count`, `stride` - Request to write a subarray of values - `data`, `start`, `count`, `imap` - Request to write a mapped array of values @@ -58,10 +58,10 @@ Nonblocking Read the method call returns a request id that can be optionally passed to :meth:`File.wait` to select this request. Similar to :meth:`Variable.get_var`, the behavior of :meth:`Variable.iget_var` varies depending on the pattern of - provided optional arguments - `index`, `start`, `count`, `stride`, and `imap`. + provided optional arguments - `start`, `count`, `stride`, and `imap`. - `buff` - Request to read an entire variable - - `buff`, `index` - Request to read a single data value + - `buff`, `start` - Request to read a single data value - `buff`, `start`, `count` - Request to read an array of values - `buff`, `start`, `count`, `stride` - Request to read a subarray of values - `buff`, `start`, `count`, `imap` - Request to read a mapped array of values diff --git a/docs/source/tutorial/read_write.rst b/docs/source/tutorial/read_write.rst index 34b1e8f..df0daf2 100644 --- a/docs/source/tutorial/read_write.rst +++ b/docs/source/tutorial/read_write.rst @@ -6,10 +6,10 @@ Parallel Read and Write NumPy Slicing Syntax -------------------------------------- - PnetCDF-python datasets re-use the numpy slicing syntax to read and write to - the file. Slice specifications are translated directly to PnetCDF “start, - count, stride” selections, and are a fast and efficient way to access data in - the file. The following slicing arguments are recognized: + PnetCDF-python datasets re-use the ``numpy`` slicing syntax to read and write + to the file. Slice specifications are translated directly to PnetCDF-C style + of subarray selection, i.e. using index arrays of “start, count, stride”. + The following slicing arguments are recognized: - Indices (var[1,5]) - Slices (i.e. [:] or [0:10]) @@ -29,7 +29,7 @@ NumPy Slicing Syntax # put values to the entire variable var[:] = buff - # read the topleft 10*10 corner from variable var + # read the top-left 10*10 corner from variable var print(var[:10, :10]) @@ -42,8 +42,8 @@ Method Call of put_var()/get_var() written. :meth:`Variable.get_var` requires `buff` as a mandatory argument, which serves as a read buffer that stores values to be read. The behavior of :meth:`Variable.put_var` and :meth:`Variable.get_var` varies depending on the - pattern of provided optional arguments - `index`, `start`, `count`, `stride`, - and `imap`. The suffix `_all` indicates this is collective I/O in contrast to + pattern of provided optional arguments - `start`, `count`, `stride`, and + `imap`. The suffix `_all` indicates this is collective I/O in contrast to independent I/O (without `_all`). Read from netCDF variables @@ -51,7 +51,7 @@ Read from netCDF variables provided input parameter pattern: - `buff` - Read an entire variable - - `buff`, `index` - Read a single data value + - `buff`, `start` - Read a single data value - `buff`, `start`, `count` - Read an array of values - `buff`, `start`, `count`, `stride` - Read a subarray of values - `buff`, `start`, `count`, `imap`, `buff` - Read a mapped array of values @@ -84,7 +84,7 @@ Write to netCDF variables provided input parameter pattern: - `data` - Write an entire variable - - `data`, `index` - Write a single data value (a single element) + - `data`, `start` - Write a single data value (a single element) - `data`, `start`, `count` - Write an array of values - `data`, `start`, `count`, `stride` - Write a subarray of values - `data`, `start`, `count`, `imap` - Write a mapped array of values diff --git a/examples/fill_mode.py b/examples/fill_mode.py index b724663..31a4b52 100644 --- a/examples/fill_mode.py +++ b/examples/fill_mode.py @@ -77,18 +77,22 @@ def pnetcdf_io(filename): old_fillmode = f.set_fill(pnetcdf.NC_FILL) if verbose: if old_fillmode == pnetcdf.NC_FILL: - print("The old fill mode is NC_FILL\n") + print("The old fill mode is NC_FILL") else: - print("The old fill mode is NC_NOFILL\n") + print("The old fill mode is NC_NOFILL") - # set the fill mode to back to NC_NOFILL for the entire file + # set the fill mode back to NC_NOFILL for the entire file f.set_fill(pnetcdf.NC_NOFILL) - # set the variable's fill mode to NC_FILL with default fill value + # set the variable's fill mode to NC_FILL with PnetCDF default fill value fix_var.def_fill(no_fill = 0) - # set a customized fill value -1 + # enable the variable's fill mode and use a customized value, -1 fill_value = np.int32(-1) + rec_var.def_fill(no_fill = 0, fill_value = fill_value) + + # Equivalently this can be done by setting the PnetCDF pre-defined + # attribute "_FillValue" rec_var._FillValue = fill_value # exit define mode diff --git a/examples/nonblocking/nonblocking_write.py b/examples/nonblocking/nonblocking_write.py index a5020c0..b44506c 100644 --- a/examples/nonblocking/nonblocking_write.py +++ b/examples/nonblocking/nonblocking_write.py @@ -112,7 +112,7 @@ def pnetcdf_io(filename, length): if pnetcdf.strerrno(req_errs[i]) != "NC_NOERR": print(f"Error on request {i}:", pnetcdf.strerror(req_errs[i])) - # detach the temporary buffer + # detach the buffer f.detach_buff() # Close the file diff --git a/examples/put_var.py b/examples/put_var.py index c620f5b..f820b95 100644 --- a/examples/put_var.py +++ b/examples/put_var.py @@ -92,7 +92,7 @@ def pnetcdf_io(filename, file_format): # Define dimensions dim_y = f.def_dim("Y", global_ny) - dim_x = f.def_dim("X",global_nx) + dim_x = f.def_dim("X", global_nx) # Define a 2D variable of integer type var = f.def_var("var", pnetcdf.NC_INT, (dim_y, dim_x)) diff --git a/src/pnetcdf/_Dimension.pyx b/src/pnetcdf/_Dimension.pyx index b0c12a6..333df00 100644 --- a/src/pnetcdf/_Dimension.pyx +++ b/src/pnetcdf/_Dimension.pyx @@ -37,6 +37,16 @@ cdef class Dimension: .. note:: ``Dimension`` instances should be created using the :meth:`File.def_dim` method of a ``File`` instance, not using :meth:`Dimension.__init__` directly. + + :Example: A example is available in ``examples/put_var.py`` + + :: + + # Define dimensions + dim_t = f.def_dim('time', size = -1) + dim_y = f.def_dim("Y", size = 100) + dim_x = f.def_dim("X", size = 200) + """ cdef int ierr cdef char *dimname @@ -89,7 +99,7 @@ cdef class Dimension: def __str__(self): if not dir(self._file): - return 'Dimension object no longer valid' + return 'Dimension is not valid' if self.isunlimited(): return "%r (unlimited): name = '%s', size = %s" %\ (type(self), self._name, len(self)) @@ -110,8 +120,7 @@ cdef class Dimension: """ getfile(self) - :return: the ``pnetcdf.File`` instance that this ``Dimension`` is a - member of. + :return: the ``pnetcdf.File`` instance that this ``Dimension`` belongs to. :rtype: :class:`pnetcdf.File` """ diff --git a/src/pnetcdf/_File.pyx b/src/pnetcdf/_File.pyx index 940e5e5..70a2025 100644 --- a/src/pnetcdf/_File.pyx +++ b/src/pnetcdf/_File.pyx @@ -63,12 +63,23 @@ cdef class File: :type comm: mpi4py.MPI.Comm or None :param info: [Optional] - MPI info object to use for file access. `None` defaults to + MPI info instance to use for file access. `None` defaults to ``MPI_INFO_NULL``. :type info: mpi4py.MPI.Info or None :return: The created file instance. :rtype: :class:`pnetcdf.File` + + :Example: A example is available in ``examples/create_open.py`` + + :: + + # create a new file using file clobber mode, i.e. flag "-w" + f = pnetcdf.File(filename = "foo.nc", mode = 'w', comm = MPI.COMM_WORLD, info = None) + + # open an existing file for read only + f = pnetcdf.File(filename = "foo.nc", mode = 'r', comm = MPI.COMM_WORLD, info = None) + """ cdef int ncid encoding = sys.getfilesystemencoding() @@ -129,6 +140,16 @@ cdef class File: close(self) Close the opened netCDF file + + :Example: A example is available in ``examples/create_open.py`` + + :: + + # create a new file using file clobber mode, i.e. flag "-w" + f = pnetcdf.File(filename = "foo.nc", mode = 'w', comm = MPI.COMM_WORLD, info = None) + + f.close() + """ self._close(True) @@ -178,7 +199,7 @@ cdef class File: def __dealloc__(self): - # close file when there are no references to object left + # close file when there are no references to it left if self._isopen: self._close(False) @@ -296,13 +317,24 @@ cdef class File: must be a positive integer or `-1`, which stands for "unlimited" (default is `-1`). The return value is the `Dimension` class instance describing the new dimension. To determine the current maximum size of - the dimension, use the `len` function on the `Dimension` instance. To - determine if a dimension is 'unlimited', use the + the dimension, use the python function `len()` on the `Dimension` + instance. To determine if a dimension is 'unlimited', use the :meth:`Dimension.isunlimited` method of the `Dimension` instance. :param str dimname: Name of the new dimension. :param int size: [Optional] Size of the new dimension. + :Example: A example is available in ``examples/put_var.py`` + + :: + + dim_t = f.def_dim('time', size = -1) + dim_y = f.def_dim("Y", size = 100) + dim_x = f.def_dim("X", size = 200) + + # Define a 2D variable of integer type + var = f.def_var("foo", pnetcdf.NC_INT, (dim_y, dim_x)) + """ self.dimensions[dimname] = Dimension(self, dimname, size=size) return self.dimensions[dimname] @@ -446,6 +478,17 @@ cdef class File: :return: The created variable :rtype: :class:`pnetcdf.Variable` + + :Example: A example is available in ``examples/put_var.py`` + + :: + + dim_y = f.def_dim("Y", global_ny) + dim_x = f.def_dim("X", global_nx) + + # Define a 2D variable of integer type + var = f.def_var("foo", pnetcdf.NC_INT, (dim_y, dim_x)) + """ # the following should be added to explanation of variable class. @@ -454,7 +497,7 @@ cdef class File: # # containing all the netCDF attribute name/value pairs is provided by # # the `__dict__` attribute of a `Variable` instance. - # # `Variable` instances behave much like array objects. Data can be + # # `Variable` instances behave much like arrays. Data can be # # assigned to or retrieved from a variable with indexing and slicing # # operations on the `Variable` instance. A `Variable` instance has six # # Dataset standard attributes: `dimensions, dtype, shape, ndim, name`. @@ -521,6 +564,17 @@ cdef class File: :Operational mode: This method must be called while the file is in define mode. + + :Example: A example is available in ``examples/put_var.py`` + + :: + + str_att = "example attribute of type text." + var.foo_attr = str_att + + # Equivalently, below uses function call + var.put_att("foo_attr", str_att) + """ cdef nc_type xtype xtype=-99 @@ -545,6 +599,17 @@ cdef class File: :Operational mode: This method can be called while the file is in either define or data mode (collective or independent). + + :Example: A example is available in ``examples/get_var.py`` + + :: + + # Get global attribute named "foo_attr" + str_att = f.get_att("foo_attr") + + # Get the variable's attribute named "foo_attr" + str_att = v.foo_attr + """ return _get_att(self, NC_GLOBAL, name, encoding=encoding) @@ -681,13 +746,14 @@ cdef class File: _check_err(ierr) return None - def wait(self, num=None, requests=None, status=None): + def wait_all(self, num=None, requests=None, status=None): """ - wait(self, num=None, requests=None, status=None) + wait_all(self, num=None, requests=None, status=None) This method is a blocking call that wait for the completion of - nonblocking I/O requests made by :meth:`Variable.iput_var`, - :meth:`Variable.iget_var` and :meth:`Variable.bput_var` + nonblocking I/O requests made by one of more method calls to + :meth:`Variable.iput_var`, :meth:`Variable.iget_var` and + :meth:`Variable.bput_var` :param int num: [Optional] number of requests. It is also the array size of the next two @@ -709,21 +775,36 @@ cdef class File: the error messages. :type status: list - :Operational mode: it is an independent subroutine and must be called - while the file is in independent data mode. + :Operational mode: it is an collective subroutine and must be called + while the file is in collective data mode. + + :Example: A example is available in ``examples/nonblocking/nonblocking_write.py`` + + :: + + # Write one variable at a time, using iput APIs + reqs = [] + for i in range(NUM_VARS): + req_id = vars[i].iput_var(buf[i], start = start, count = count) + reqs.append(req_id) + + # commit posted nonblocking requests + req_errs = [None] * NUM_VARS + f.wait_all(NUM_VARS, reqs, req_errs) + """ - return self._wait(num, requests, status, collective=False) + return self._wait(num, requests, status, collective=True) - def wait_all(self, num=None, requests=None, status=None): + def wait(self, num=None, requests=None, status=None): """ - wait_all(self, num=None, requests=None, status=None) + wait(self, num=None, requests=None, status=None) - Same as :meth:`File.wait` but in collective data mode + Same as :meth:`File.wait_all` but called in independent data mode - :Operational mode: it is an collective subroutine and must be called - while the file is in collective data mode. + :Operational mode: it is an independent subroutine and must be called + while the file is in independent data mode. """ - return self._wait(num, requests, status, collective=True) + return self._wait(num, requests, status, collective=False) def cancel(self, num=None, requests=None, status=None): """ @@ -814,6 +895,15 @@ cdef class File: ``numpy.ndarray.nbytes`` :type bufsize: int + :Example: A example is available in ``examples/nonblocking/nonblocking_write.py`` + + :: + + # Before calling bput APIs, calculate allocate space needed + bufsize = length * NUM_VARS * np.dtype(np.int32).itemsize + + f.attach_buff(bbufsize) + """ cdef MPI_Offset buffsize cdef int _file_id @@ -829,6 +919,18 @@ cdef class File: Detach the write buffer previously attached for buffered non-blocking write + + :Example: A example is available in ``examples/nonblocking/nonblocking_write.py`` + + :: + + # Before calling bput APIs, calculate allocate space needed + bufsize = length * NUM_VARS * np.dtype(np.int32).itemsize + f.attach_buff(bbufsize) + + # detach the buffer + f.detach_buff() + """ cdef int _file_id = self._ncid with nogil: @@ -912,6 +1014,22 @@ cdef class File: :Operational mode: This method is a collective subroutine and must be called in define mode + + :Example: A example is available in ``examples/fill_mode.py`` + + :: + + # set the fill mode to NC_FILL for the entire file + old_fillmode = f.set_fill(pnetcdf.NC_FILL) + + if old_fillmode == pnetcdf.NC_FILL: + print("The old fill mode is NC_FILL") + else: + print("The old fill mode is NC_NOFILL") + + # set the fill mode to back to NC_NOFILL for the entire file + f.set_fill(pnetcdf.NC_NOFILL) + """ cdef int _file_id, _fillmode, _old_fillmode _file_id = self._ncid @@ -925,10 +1043,10 @@ cdef class File: """ set_auto_chartostring(self, value) - Call :meth:`Variable.set_auto_chartostring` for all variables - contained in this `File`. Calling this function only affects existing - variables. Variables defined after calling this function will follow - the default behaviour. + Call :meth:`Variable.set_auto_chartostring` for all variables contained + in this `File`. Calling this method only affects existing variables. + Variables defined after calling this method will follow the default + behaviour. :param value: True or False :type value: bool @@ -1026,7 +1144,7 @@ cdef class File: """ inq_info(self) - Returns an MPI info object containing all the file hints used by + Returns an MPI info instance containing all the file hints used by PnetCDF library. :rtype: mpi4py.MPI.Info @@ -1106,7 +1224,7 @@ cdef class File: return extent cdef _get_dims(file): - # Private function to create `Dimension` instances for all the + # Private method to create `Dimension` instances for all the # dimensions in a `File` cdef int ierr, numdims, n, _file_id cdef int *dimids @@ -1132,7 +1250,7 @@ cdef _get_dims(file): return dimensions cdef _get_variables(file): - # Private function to create `Variable` instances for all the + # Private method to create `Variable` instances for all the # variables in a `File` cdef int ierr, numvars, n, nn, numdims, varid, classp, iendian, _file_id cdef int *varids diff --git a/src/pnetcdf/_Variable.pyx b/src/pnetcdf/_Variable.pyx index 25be239..6acdc9b 100644 --- a/src/pnetcdf/_Variable.pyx +++ b/src/pnetcdf/_Variable.pyx @@ -34,8 +34,7 @@ ctypedef MPI.Datatype Datatype cdef class Variable: """ A PnetCDF variable is used to read and write netCDF data. They are - analogous to numpy array objects. See :meth:`Variable.__init__` for more - details. + analogous to numpy arrays. See :meth:`Variable.__init__` for more details. .. note:: ``Variable`` instances should be created using the :meth:`File.def_var` method of a :meth:`File` instance, not using this @@ -87,6 +86,20 @@ cdef class Variable: :return: The created variable :rtype: :class:`pnetcdf.Variable` + :Example: A example is available in ``examples/put_var.py`` + + :: + + # Define dimensions + dim_y = f.def_dim("Y", global_ny) + dim_x = f.def_dim("X", global_nx) + + # Define a 2D variable of integer type, using :meth:`File.def_var`. + var = f.def_var("var", pnetcdf.NC_INT, (dim_y, dim_x)) + + # Or equivalently, using :meth:`File.createVariable`. + var = f.createVariable("var", pnetcdf.NC_INT, (dim_y, dim_x)) + """ cdef int ierr, ndims, icontiguous, icomplevel, numdims, _file_id, nsd, @@ -102,7 +115,7 @@ cdef class Variable: self._file = file _file_id = self._file_id #TODO: decide whether we need to check xtype at python-level - if isinstance(datatype, str): # convert to numpy datatype object + if isinstance(datatype, str): # convert to numpy data type object datatype = np.dtype(datatype) if isinstance(datatype, np.dtype): if datatype.str[1:] in _supportedtypes: @@ -161,7 +174,8 @@ cdef class Variable: def __array__(self): # numpy special method that returns a numpy array. - # allows numpy ufuncs to work faster on Variable objects + # This allows numpy Universal functions ufuncs to work faster on + # Variable instances. return self[...] def __repr__(self): @@ -312,6 +326,17 @@ cdef class Variable: :Operational mode: This method must be called while the associated netCDF file is in define mode. + + :Example: A example is available in ``examples/put_var.py`` + + :: + + str_att = "example attribute of type text." + var.put_att("foo_attr", str_att) + + # Equivalently, uses python dictionary way + var.foo_attr = str_att + """ cdef nc_type xtype xtype=-99 @@ -335,6 +360,17 @@ cdef class Variable: :Operational mode: This method can be called while the file is in either define or data mode (collective or independent). + + :Example: A example is available in ``examples/get_var.py`` + + :: + + # Get attribute named "foo_attr" + str_att = v.get_att("foo_attr") + + # Equivalently, uses python dictionary way + str_att = v.foo_attr + """ return _get_att(self._file, self._varid, name, encoding=encoding) @@ -447,6 +483,20 @@ cdef class Variable: ignored and the default fill value is used. :type fill_value: any + :Example: A example is available in ``examples/fill_mode.py`` + + :: + + # set the variable's fill mode to NC_FILL with PnetCDF default fill value + var.def_fill(no_fill = 0) + + # enable the variable's fill mode and use a customized value + fill_value = np.int32(-1) + var.def_fill(no_fill = 0, fill_value = fill_value) + + # Equivalently this can be done by setting the PnetCDF pre-defined attribute "_FillValue" + var._FillValue = fill_value + """ cdef ndarray data cdef int ierr, _no_fill @@ -526,11 +576,10 @@ cdef class Variable: self.chartostring = bool(chartostring) def __getitem__(self, elem): - # This special method is used to index the netCDF variable - # using the "extended slice syntax". The extended slice syntax - # is a perfect match for the "start", "count" and "stride" - # arguments to the ncmpi_get_var() function, and is much more easy - # to use. + # This special method is used to index the netCDF variable using the + # "extended slice syntax". The extended slice syntax is a perfect match + # for the "start", "count" and "stride" arguments to the C function + # ncmpi_get_var(), and is much more easy to use. start, count, stride, put_ind =\ _StartCountStride(elem,self.shape,dimensions=self.dimensions,file=self._file) datashape = _out_array_shape(count) @@ -593,11 +642,10 @@ cdef class Variable: return data def __setitem__(self, elem, data): - # This special method is used to assign to the netCDF variable - # using "extended slice syntax". The extended slice syntax - # is a perfect match for the "start", "count" and "stride" - # arguments to the ncmpi_put_var() function, and is much more easy - # to use. + # This special method is used to assign to the netCDF variable using + # "extended slice syntax". The extended slice syntax is a perfect match + # for the "start", "count" and "stride" arguments to the C function + # ncmpi_put_var(), and is much more easy to use. # if _Encoding is specified for a character variable, convert # numpy array of strings to a numpy array of characters with one more @@ -833,13 +881,13 @@ cdef class Variable: bufftype) _check_err(ierr) - def put_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None): + def put_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None): """ - put_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) + put_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None) Method write multiple subarrays of a netCDF variable to the file. This - an independent I/O call and can only be called when the file is in the - independent I/O mode. This method is equivalent to making multiple + an collective I/O call and can only be called when the file is in the + collective I/O mode. This method is equivalent to making multiple calls to :meth:`Variable.put_var`. Note, combining multiple `put_var` calls into one can achieve a better performance. @@ -919,21 +967,36 @@ cdef class Variable: An MPI derived data type that describes the memory layout of the write buffer. :type buftype: mpi4py.MPI.Datatype + + :Example: A example is available in ``examples/put_varn_int.py`` + + :: + + num_reqs = 4 + starts = np.zeros((num_reqs, NDIMS), dtype=np.int64) + counts = np.zeros((num_reqs, NDIMS), dtype=np.int64) + starts[0][0] = 0; starts[0][1] = 5; counts[0][0] = 1; counts[0][1] = 2 + starts[1][0] = 1; starts[1][1] = 0; counts[1][0] = 1; counts[1][1] = 1 + starts[2][0] = 2; starts[2][1] = 6; counts[2][0] = 1; counts[2][1] = 2 + starts[3][0] = 3; starts[3][1] = 0; counts[3][0] = 1; counts[3][1] = 3 + + v.put_varn_all(w_buf, num = num_reqs, starts = starts, counts = counts) + """ self._put_varn(data, num, starts, counts, bufcount = bufcount, - buftype = buftype, collective = False) + buftype = buftype, collective = True) - def put_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None): + def put_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None): """ - put_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None) + put_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) - This method call is the same as method :meth:`Variable.put_varn`, - except it is collective and can only be called in the collective I/O - mode. Please refer to :meth:`Variable.put_varn` for its argument - usage. + This method call is the same as method :meth:`Variable.put_varn_all`, + except it is an independent call and can only be called in the + independent I/O mode. Please refer to :meth:`Variable.put_varn_all` for + its argument usage. """ self._put_varn(data, num, starts, counts, bufcount = bufcount, - buftype = buftype, collective = True) + buftype = buftype, collective = False) def iput_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None): """ @@ -942,22 +1005,22 @@ cdef class Variable: This method call is the nonblocking counterpart of :meth:`Variable.put_varn`. The syntax is the same as :meth:`Variable.put_varn`. For the argument usage, please refer to - method :meth:`Variable.put_varn`. This method returns a request ID - that can be used in :meth:`File.wait` or :meth:`File.wait_all`. The - posted write request may not be committed until :meth:`File.wait` or - :meth:`File.wait_all` is called. + method :meth:`Variable.put_varn_all`. This method returns a request ID + that can be used in :meth:`File.wait_all` or :meth:`File.wait`. The + posted write request may not be committed until :meth:`File.wait_all` or + :meth:`File.wait` is called. .. note:: Unlike :meth:`Variable.put_varn`, the posted nonblocking write requests may not be committed to the file until the time of calling - :meth:`File.wait` or :meth:`File.wait_all`. Users should not + :meth:`File.wait_all` or :meth:`File.wait`. Users should not alter the contents of the write buffer once the request is posted - until the :meth:`File.wait` or :meth:`File.wait_all` is + until the :meth:`File.wait_all` or :meth:`File.wait` is returned. Any change to the buffer contents in between will result in unexpected error. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int """ @@ -970,22 +1033,22 @@ cdef class Variable: This method call is the nonblocking, buffered counterpart of :meth:`Variable.put_varn`. For the argument usage, please refer to - method :meth:`Variable.put_varn`. This method returns a request ID - that can be used in :meth:`File.wait` or :meth:`File.wait_all`. The - posted write request may not be committed until :meth:`File.wait` or - :meth:`File.wait_all` is called. + method :meth:`Variable.put_varn_all`. This method returns a request ID + that can be used in :meth:`File.wait_all` or :meth:`File.wait`. The + posted write request may not be committed until :meth:`File.wait_all` + or :meth:`File.wait` is called. .. note:: Unlike :meth:`Variable.iput_varn`, the write data is buffered (cached) internally by PnetCDF and will be flushed to the file at - the time of calling :meth:`File.wait` or :meth:`File.wait_all`. + the time of calling :meth:`File.wait_all` or :meth:`File.wait`. Once the call to this method returns, the caller is free to change the contents of write buffer. Prior to calling this method, make sure :meth:`File.attach_buff` is called to allocate an internal buffer for accommodating the write requests. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int """ @@ -1087,12 +1150,12 @@ cdef class Variable: - def put_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): + def put_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): """ - put_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) + put_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method to write in parallel to the netCDF variable in independent I/O - mode. The behavior of the method varies depends on the pattern of + Method to write in parallel to the netCDF variable in the collective + I/O mode. The behavior of the method varies depends on the pattern of provided optional arguments - `start`, `count`, `stride`, `bufcount` and `buftype`. @@ -1235,47 +1298,57 @@ cdef class Variable: :type buftype: mpi4py.MPI.Datatype :Operational mode: This method must be called while the file is in - independent data mode.""" + collective data mode. + + :Example: A example is available in ``examples/put_var.py`` + :: + + var.put_var_all(buf, start = start, count = count) + + # Equivalently, below uses python index style + end = [start[i] + count[i] for i in range(2)] + var[start[0]:end[0], start[1]:end[1]] = buf + + """ if data is not None and all(arg is None for arg in [start, count, stride, imap]): - self._put_var(data, collective = False, bufcount = bufcount, buftype = buftype) + self._put_var(data, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [data, start]) and all(arg is None for arg in [count, stride, imap]): - self._put_var1(data, start, collective = False, bufcount = bufcount, buftype = buftype) + self._put_var1(data, start, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [data, start, count]) and all(arg is None for arg in [stride, imap]): - self._put_vara(start, count, data, collective = False, bufcount = bufcount, buftype = buftype) + self._put_vara(start, count, data, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [data, start, count, stride]) and all(arg is None for arg in [imap]): - self._put_vars(start, count, stride, data, collective = False, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start, count, stride, imap]): - self._put_varm(data, start, count, stride, imap, collective = False, bufcount = bufcount, buftype = buftype) + self._put_vars(start, count, stride, data, collective = True, bufcount = bufcount, buftype = buftype) + elif all(arg is not None for arg in [data, start, count, imap]): + self._put_varm(data, start, count, stride, imap, collective = True, bufcount = bufcount, buftype = buftype) else: - raise ValueError("Invalid input arguments for put_var") + raise ValueError("Invalid input arguments for put_var_all") + - def put_var_all(self, data, start=None, count=None, stride=None, num=None, imap=None, bufcount=None, buftype=None): + def put_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): """ - put_var_all(self, data, start=None, count=None, stride=None, num=None, imap=None, bufcount=None, buftype=None) + put_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method to write in parallel to the netCDF variable in the collective + Method to write in parallel to the netCDF variable in the independent I/O mode. For the argument usage, please refer to method - :meth:`Variable.put_var`. The only difference is this method is a - collective operation. + :meth:`Variable.put_var_all`. The only difference is this method is a + independent operation. :Operational mode: This method must be called while the file is in - collective data mode. + independent data mode. """ - if data is not None and all(arg is None for arg in [start, count, stride, num, imap]): - self._put_var(data, collective = True, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start]) and all(arg is None for arg in [count, stride, num, imap]): - self._put_var1(data, start, collective = True, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start, count]) and all(arg is None for arg in [stride, num, imap]): - self._put_vara(start, count, data, collective = True, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start, count, stride]) and all(arg is None for arg in [num, imap]): - self._put_vars(start, count, stride, data, collective = True, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start, count, num]) and all(arg is None for arg in [stride, imap]): - self._put_varn(start, count, num, data, collective = True, bufcount = bufcount, buftype = buftype) - elif all(arg is not None for arg in [data, start, count, imap]) and all(arg is None for arg in [num]): - self._put_varm(data, start, count, stride, imap, collective = True, bufcount = bufcount, buftype = buftype) + if data is not None and all(arg is None for arg in [start, count, stride, imap]): + self._put_var(data, collective = False, bufcount = bufcount, buftype = buftype) + elif all(arg is not None for arg in [data, start]) and all(arg is None for arg in [count, stride, imap]): + self._put_var1(data, start, collective = False, bufcount = bufcount, buftype = buftype) + elif all(arg is not None for arg in [data, start, count]) and all(arg is None for arg in [stride, imap]): + self._put_vara(start, count, data, collective = False, bufcount = bufcount, buftype = buftype) + elif all(arg is not None for arg in [data, start, count, stride]) and all(arg is None for arg in [imap]): + self._put_vars(start, count, stride, data, collective = False, bufcount = bufcount, buftype = buftype) + elif all(arg is not None for arg in [data, start, count, stride, imap]): + self._put_varm(data, start, count, stride, imap, collective = False, bufcount = bufcount, buftype = buftype) else: - raise ValueError("Invalid input arguments for put_var_all") + raise ValueError("Invalid input arguments for put_var") def _put(self, ndarray data, start, count, stride): """Private method to put data into a netCDF variable""" @@ -1593,11 +1666,11 @@ cdef class Variable: imapp, PyArray_DATA(buff), buffcount, bufftype) _check_err(ierr) - def get_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): + def get_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): """ - get_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) + get_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method to read in parallel from the netCDF variable in the independent + Method to read in parallel from the netCDF variable in the collective I/O mode. The behavior of the method varies depends on the pattern of provided optional arguments - `start`, `count`, `stride`, and `imap`. The method requires a empty array (`data`) as a read buffer from caller @@ -1618,7 +1691,7 @@ cdef class Variable: - `data`, `start` - Read a single data value (a single element). Put a single array element specified by `start` from a variable of - an opened netCDF file that is in data mode. For example, index = + an opened netCDF file that is in data mode. For example, start = [0,5] would specify the following position in a 4 * 10 two-dimensional variable ("-" means skip). @@ -1736,7 +1809,22 @@ cdef class Variable: :type buftype: mpi4py.MPI.Datatype :Operational mode: This method must be called while the file is in - independent data mode. + collective data mode. + + :Example: A example is available in ``examples/get_var.py`` + + :: + + # allocate read buffer + r_buf = np.empty(tuple(count), v.dtype) + + # Read a subarray in collective mode + v.get_var_all(r_buf, start = start, count = count) + + # Equivalently, below uses python index style + end = [start[i] + count[i] for i in range(2)] + r_bufs = v[start[0]:end[0], start[1]:end[1]] + """ # Note that get_var requires a empty array as a buffer arg from caller # to store returned array values. We understand this is against python @@ -1746,50 +1834,50 @@ cdef class Variable: # 2. Other i/o methods (iget/put/iput) all require buffer array as mandatory argument if all(arg is None for arg in [start, count, stride, imap]): - self._get_var(data, collective = False, bufcount = bufcount, buftype = buftype) + self._get_var(data, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start]) and all(arg is None for arg in [count, stride, imap]): - self._get_var1(data, start, collective = False, bufcount = bufcount, buftype = buftype) + self._get_var1(data, start, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count]) and all(arg is None for arg in [stride, imap]): - self._get_vara(data, start, count, collective = False, bufcount = bufcount, buftype = buftype) + self._get_vara(data, start, count, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count, stride]) and all(arg is None for arg in [imap]): - self._get_vars(data, start, count, stride, collective = False, bufcount = bufcount, buftype = buftype) + self._get_vars(data, start, count, stride, collective = True, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count, imap]): - self._get_varm(data, start, count, stride, imap, collective = False, bufcount = bufcount, buftype = buftype) + self._get_varm(data, start, count, stride, imap, collective = True, bufcount = bufcount, buftype = buftype) else: - raise ValueError("Invalid input arguments for get_var") + raise ValueError("Invalid input arguments for get_var_all") - def get_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): + def get_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None): """ - get_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) + get_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method to read in parallel from the netCDF variable in the collective + Method to read in parallel from the netCDF variable in the independent I/O mode. For the argument usage, please refer to method - :meth:`Variable.get_var`. The only difference is this method is a - collective operation. + :meth:`Variable.get_var_all`. The only difference is this method is a + independent operation. :Operational mode: This method must be called while the file is in - collective data mode. + independent data mode. """ if all(arg is None for arg in [start, count, stride, imap]): - self._get_var(data, collective = True, bufcount = bufcount, buftype = buftype) + self._get_var(data, collective = False, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start]) and all(arg is None for arg in [count, stride, imap]): - self._get_var1(data, start, collective = True, bufcount = bufcount, buftype = buftype) + self._get_var1(data, start, collective = False, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count]) and all(arg is None for arg in [stride, imap]): - self._get_vara(data, start, count, collective = True, bufcount = bufcount, buftype = buftype) + self._get_vara(data, start, count, collective = False, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count, stride]) and all(arg is None for arg in [imap]): - self._get_vars(data, start, count, stride, collective = True, bufcount = bufcount, buftype = buftype) + self._get_vars(data, start, count, stride, collective = False, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [start, count, imap]): - self._get_varm(data, start, count, stride, imap, collective = True, bufcount = bufcount, buftype = buftype) + self._get_varm(data, start, count, stride, imap, collective = False, bufcount = bufcount, buftype = buftype) else: - raise ValueError("Invalid input arguments for get_var_all") + raise ValueError("Invalid input arguments for get_var") - def get_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None): + def get_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None): """ - get_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) + get_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None) Method to read multiple subarrays of a netCDF variables from the file. - This an independent I/O call and can only be called when the file is in - the independent I/O mode. This method is equivalent to making multiple + This a collective I/O call and can only be called when the file is in + the collective I/O mode. This method is equivalent to making multiple calls to :meth:`Variable.get_var`. Note, combining multiple `get_var` calls into one can achieve a better performance. @@ -1869,21 +1957,36 @@ cdef class Variable: An MPI derived data type that describes the memory layout of the write buffer. :type buftype: mpi4py.MPI.Datatype + + :Example: an example code fragment is given below. + + :: + + num_reqs = 4 + starts = np.zeros((num_reqs, NDIMS), dtype=np.int64) + counts = np.zeros((num_reqs, NDIMS), dtype=np.int64) + starts[0][0] = 0; starts[0][1] = 5; counts[0][0] = 1; counts[0][1] = 2 + starts[1][0] = 1; starts[1][1] = 0; counts[1][0] = 1; counts[1][1] = 1 + starts[2][0] = 2; starts[2][1] = 6; counts[2][0] = 1; counts[2][1] = 2 + starts[3][0] = 3; starts[3][1] = 0; counts[3][0] = 1; counts[3][1] = 3 + + v.get_varn_all(r_buf, num = num_reqs, starts = starts, counts = counts) + """ return self._get_varn(data, num, starts, counts, bufcount = bufcount, - buftype = buftype, collective = False) + buftype = buftype, collective = True) - def get_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None): + def get_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None): """ - get_varn_all(self, data, num, starts, counts=None, bufcount=None, buftype=None) + get_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) - This method call is the same as method :meth:`Variable.get_varn`, - except it is collective and can only be called while the file in the - collective I/O mode. Please refer to :meth:`Variable.get_varn` for - its argument usage. + This method call is the same as method :meth:`Variable.get_varn_all`, + except it is an independent call and can only be called while the file + in the independent I/O mode. Please refer to + :meth:`Variable.get_varn_all` for its argument usage. """ return self._get_varn(data, num, starts, counts, bufcount = bufcount, - buftype = buftype, collective = True) + buftype = buftype, collective = False) def _get(self,start,count,stride): """Private method to retrieve data from a netCDF variable""" @@ -2241,24 +2344,25 @@ cdef class Variable: Method to post a nonblocking, buffered write request to write to the netCDF variable. The syntax is the same as :meth:`Variable.put_var`. - For the argument usage, please refer to :meth:`Variable.put_var`. This - method returns a request ID that can be used in :meth:`File.wait` or - :meth:`File.wait_all`. The posted write request may not be committed - until :meth:`File.wait` or :meth:`File.wait_all` is called. + For the argument usage, please refer to :meth:`Variable.put_var_all`. + This method returns a request ID that can be used in + :meth:`File.wait_all` or :meth:`File.wait`. The posted write request + may not be committed until :meth:`File.wait_all` or :meth:`File.wait` + is called. .. note:: Note that this method requires a numpy array (`data`) as a write buffer from caller prepared for writing returned array values - when :meth:`File.wait` or :meth:`File.wait_all` is called. + when :meth:`File.wait_all` or :meth:`File.wait` is called. Unlike :meth:`Variable.iput_var`, the write data is buffered (cached) internally by PnetCDF and will be flushed to the file at - the time of calling :meth:`File.wait` or :meth:`File.wait_all`. + the time of calling :meth:`File.wait_all` or :meth:`File.wait`. Once the call to this method returns, the caller is free to change the contents of write buffer. Prior to calling this method, make sure :meth:`File.attach_buff` is called to allocate an internal buffer for accommodating the write requests. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int @@ -2285,22 +2389,22 @@ cdef class Variable: Method to post a nonblocking request to write to the netCDF variable. The syntax is the same as :meth:`Variable.put_var`. For the argument - usage, please refer to :meth:`Variable.put_var`. This method returns a - request ID that can This method returns a request ID that can be used - in :meth:`File.wait` or :meth:`File.wait_all`. The posted write request - may not be committed until :meth:`File.wait` or :meth:`File.wait_all` - is called. + usage, please refer to :meth:`Variable.put_var_all`. This method + returns a request ID that can This method returns a request ID that can + be used in :meth:`File.wait_all` or :meth:`File.wait`. The posted write + request may not be committed until :meth:`File.wait_all` or + :meth:`File.wait` is called. .. note:: Note that this method requires a numpy array (`data`) as a write buffer from caller prepared for writing returned array values - when :meth:`File.wait` or :meth:`File.wait_all` is called. - Users should not alter the contents of the write buffer once the - request is posted until the :meth:`File.wait` or - :meth:`File.wait_all` is returned. Any change to the buffer - contents in between will result in unexpected error. + when :meth:`File.wait_all` or :meth:`File.wait` is called. Users + should not alter the contents of the write buffer once the request + is posted until the :meth:`File.wait_all` or :meth:`File.wait` is + returned. Any change to the buffer contents in between will result + in unexpected error. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int @@ -2435,22 +2539,22 @@ cdef class Variable: This method call is the nonblocking counterpart of :meth:`Variable.get_varn`. The syntax is the same as :meth:`Variable.get_varn`. For the argument usage, please refer to - method :meth:`Variable.get_varn`. This method returns a request ID - that can be used in :meth:`File.wait` or :meth:`File.wait_all`. The - posted write request may not be committed until :meth:`File.wait` or - :meth:`File.wait_all` is called. + method :meth:`Variable.get_varn_all`. This method returns a request ID + that can be used in :meth:`File.wait_all` or :meth:`File.wait`. The + posted write request may not be committed until :meth:`File.wait_all` + or :meth:`File.wait` is called. .. note:: Unlike :meth:`Variable.get_varn`, the posted nonblocking read requests may not be committed until the time of calling - :meth:`File.wait` or :meth:`File.wait_all`. Users should not + :meth:`File.wait_all` or :meth:`File.wait`. Users should not alter the contents of the read buffer once the request is posted - until the :meth:`File.wait` or :meth:`File.wait_all` is + until the :meth:`File.wait_all` or :meth:`File.wait` is returned. Any change to the buffer contents in between will result in unexpected error. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int """ @@ -2543,20 +2647,20 @@ cdef class Variable: Method to post a nonblocking request to read from the netCDF variable. The syntax is the same as :meth:`Variable.get_var`. For the argument - usage, please refer to :meth:`Variable.get_var`. This method returns a - request ID that can be used in :meth:`File.wait` or - :meth:`File.wait_all`. The posted read request may not be committed - until :meth:`File.wait` or :meth:`File.wait_all` is called. + usage, please refer to :meth:`Variable.get_var_all`. This method + returns a request ID that can be used in :meth:`File.wait_all` or + :meth:`File.wait`. The posted read request may not be committed until + :meth:`File.wait_all` or :meth:`File.wait` is called. .. note:: Note that this method requires a empty array (`data`) as a read buffer from caller prepared for storing returned array values - when :meth:`File.wait` or :meth:`File.wait_all` is called. User + when :meth:`File.wait_all` or :meth:`File.wait` is called. User is expected to retain this buffer array handler (the numpy variable) until the read buffer is committed and the transaction is completed. :return: The request ID, which can be used in a successive call to - :meth:`File.wait` or :meth:`File.wait_all` for the completion + :meth:`File.wait_all` or :meth:`File.wait` for the completion of the nonblocking operation. :rtype: int diff --git a/src/pnetcdf/_utils.pyx b/src/pnetcdf/_utils.pyx index 7b1ff75..651635c 100644 --- a/src/pnetcdf/_utils.pyx +++ b/src/pnetcdf/_utils.pyx @@ -168,7 +168,7 @@ NC_BP = NC_BP_C _private_atts = \ ['_ncid','_varid','dimensions','variables', 'file_format', '_nunlimdim','path', 'name', '__orthogonal_indexing__', '_buffer'] -# internal C functions. +# internal methods that call PnetCDF-C functions. cdef _strencode(pystr,encoding=""): # encode a string into bytes. If already bytes, do nothing. # uses 'utf-8' for default encoding. @@ -194,7 +194,7 @@ cdef _check_err(ierr, err_cls=RuntimeError, filename=""): cdef _set_att(file, int varid, name, value,\ nc_type xtype=-99): - # Private function to set an attribute name/value pair + # Private method to set an attribute name/value pair cdef int ierr, N, file_id cdef char *attname cdef char *datstring @@ -245,7 +245,7 @@ cdef _set_att(file, int varid, name, value,\ cdef _get_att(file, int varid, name, encoding='utf-8'): - # Private function to get an attribute value given its name + # Private method to get an attribute value given its name cdef int ierr, n, file_id, var_id cdef MPI_Offset att_len cdef char *attname @@ -296,7 +296,7 @@ cdef _get_att(file, int varid, name, encoding='utf-8'): cdef _get_att_names(int file_id, int varid): - # Private function to get all the attribute names of a variable + # Private method to get all the attribute names of a variable cdef int ierr, numatts, n cdef char namstring[NC_MAX_NAME+1] if varid == NC_GLOBAL: @@ -405,14 +405,12 @@ cpdef stringtochar(src, encoding='utf-8'): out_array.shape = src.shape + (src.itemsize,) return out_array -cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None,\ - - put=False): +cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None, put=False): """Return start, count, stride and indices needed to store/extract data into/from a netCDF variable. - This function is used to convert a slicing expression into a form that is - compatible with the nc_get_vars function. Specifically, it needs + This method is used to convert a slicing expression into a form that is + compatible with the C function ncmpi_get_vars. Specifically, it needs to interpret integers, slices, Ellipses, and 1-d sequences of integers and booleans. @@ -431,7 +429,7 @@ cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None,\ index arrays be 1-d boolean or integer. If integer arrays are used, the index values must be sorted and contain no duplicates. - In summary, slicing netcdf variable objects with 1-d integer or boolean arrays + In summary, slicing netcdf variable instances with 1-d integer or boolean arrays is allowed, but may give a different result than slicing a numpy array. Numpy also supports slicing an array with a boolean array of the same @@ -444,7 +442,7 @@ cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None,\ Allow for this sort of simple variable subsetting is the reason we decided to deviate from numpy's slicing rules. - This function is used both by the __setitem__ and __getitem__ method of + This method is used both by the __setitem__ and __getitem__ method of the Variable class. Parameters @@ -477,8 +475,8 @@ cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None,\ Notes: - netCDF data is accessed via the function: - nc_get_vars(fileid, varid, start, count, stride, data) + netCDF data is accessed via the C function: + ncmpi_get_vars(fileid, varid, start, count, stride, data) Assume that the variable has dimension n, then @@ -780,7 +778,7 @@ cdef _is_int(a): return False cdef _get_format(int ncid): - # Private function to get the netCDF file format + # Private method to get the netCDF file format cdef int ierr, formatp with nogil: ierr = ncmpi_inq_format(ncid, &formatp) @@ -790,14 +788,14 @@ cdef _get_format(int ncid): return _reverse_format_dict[formatp] -# external C functions. +# methods calling external POnetCDF-C functions. cpdef inq_clibvers(): """ inq_clibvers() - This function returns a string describing the version of the PnetCDF-C - library used to build this PnetCDF-Python module, and when the PnetCDF-C - library was built. + Method to return a string describing the version of the PnetCDF-C library + used to build this PnetCDF-Python module, and when the PnetCDF-C library + was built. :return: A string about PnetCDF-C library, for example, "1.13.0 of March 29, 2024". @@ -810,11 +808,11 @@ cpdef strerror(err_code): """ strerror(err_code) - This function returns an error message string corresponding to an integer - netCDF error code or to a system error number, presumably returned by a - call to a PnetCDF function. + Method to return an error message string corresponding to an integer netCDF + error code or to a system error number, presumably returned by a call to a + PnetCDF method. - :param err_code: An error code returned from a call to a PnetCDF function. + :param err_code: An error code returned from a call to a PnetCDF method. :type err_code: int :return: error message @@ -829,11 +827,11 @@ cpdef strerrno(err_code): """ strerrno(err_code) - This function returns a character string containing the name of the NC + Method to return a character string containing the name of the NC error code. For instance, ncmpi_strerrno(NC_EBADID) returns string "NC_EBADID". - :param err_code: An error code returned from a call to a PnetCDF function. + :param err_code: An error code returned from a call to a PnetCDF method. :type err_code: int :return: name of the NC error code. @@ -848,9 +846,9 @@ cpdef set_default_format(int new_format): """ set_default_format(int new_format) - Thise function change the default format of the netCDF file to be created - in the successive file creations when no file format is explicitly passed - as a parameter. + Method to change the default format of the netCDF file to be created in the + successive file creations when no file format is explicitly passed as a + parameter. :param new_format: @@ -862,7 +860,7 @@ cpdef set_default_format(int new_format): :return: The current default format before this call of setting a new default format. - :Operational mode: This function can be called in either collective or + :Operational mode: This method can be called in either collective or independent I/O mode, but is expected to be called by all processes. """ cdef int ierr, newformat, oldformat @@ -885,7 +883,7 @@ cpdef inq_default_format(): :rtype: int - :Operational mode: This function is an independent subroutine. + :Operational mode: This method is an independent subroutine. """ cdef int ierr, curformat with nogil: @@ -906,7 +904,7 @@ cpdef inq_file_format(str file_name): :rtype: int - :Operational mode: This function is an independent subroutine. + :Operational mode: This method is an independent subroutine. """ cdef char *filename cdef int ierr, curformat