README.large_files

==== BACKGROUND

The "traditonal" netcdf file format (CDF-1) uses a 4-byte value to hold
the offset into the file where one can find a variable's data

Recently (October 2003), Greg Sjaardema <gdsjaar@sandia.gov> proposed a
new file format (CDF-2) which uses an 8-byte value for the offset.
We use his approach in parallel netcdf, though we have modified his
patch against netcdf-3.5-beta1 to apply to our codebase.  

I couldn't find a URL to Greg's original message, but here's Russ Rew's 
followup: 
http://www.unidata.ucar.edu/projects/coohl/mhonarc/MailArchives/netcdf/msg04811.html

This means there are two different but compatable implementations of the
CDF-2 file format.  We (the parallel netcdf developers) will make our
best effort to keep our implementation compatable with the serial netcdf
implementation.  Please report any incompatabilities to the developers
at parallel-netcdf@mcs.anl.gov. 

==== PRELIMINARIES

First, it is important that your MPI-IO implementation uses an 8 byte
type to represent the 'MPI_Offset'

Second, your platform should use an 8 byte type to represent the 'off_t'
type.  On linux, solaris, IRIX64 (and quite possibly others), parallel-netcdf
will automatically add the right options to the compiler to make this
happen.   

Run configure as you normally would.  Let the developers know if
configure says your 'off_t' is 4 bytes.  Proceed to compile and install
the library.

==== USAGE

By default, parallel-netcdf will create CDF-1 formatted files.  This
will ensure that datasets created by our library will be compatable with
the large body of applications which expect NetCDF files to be CDF-1
formatted.

To write a CDF-2 formatted file, add the flag 'NC_64BIT_OFFSET' to the
ncmpi_create() function call ( or nfmpi_create() if using the Fortran
interface)

The parallel-netcdf library will detect the format of the dataset.
There are no special options needed to read back files created with the
NC_64BIT_OFFSET flag set.