Large File Tests Fail on some 32-bit platforms

[WardF]

I've discovered that at some point large file tests have started failing on some platforms. My evaluation is not yet complete so I do not want to enumerate it here yet in this issue; instead I'll provide a link to my working document. Once I've finalized those tests I will embed the information here.

• Working Document on hosted on MindNode

[t-b]

The large file tests fail on debian jessie/macosx sierra when building the 32bit libraries.

On debian jessie I tried to bisect the error using the following script run.sh

#!/bin/sh

set -e

rm -rf build
mkdir build
cd build 
cmake -DCMAKE_C_FLAGS=-m32 -DCMAKE_BUILD_TYPE=Debug -DBUILD_UTILITIES=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_NETCDF_4=OFF -DENABLE_DAP=OFF -DENABLE_LARGE_FILE_TESTS=ON -G "Unix Makefiles" ..
# cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_UTILITIES=OFF -DBUILD_SHARED_LIBS=OFF -DENABLE_NETCDF_4=OFF -DENABLE_DAP=OFF -DENABLE_LARGE_FILE_TESTS=ON -G "Unix Makefiles" ..
make
./nc_test/nc_test_large_files

and

git checkout master
git bisect start
git bisect bad HEAD
git bisect good netcdf-4.3.0
git bisect run ./run.sh

and this resulted in

$git bisect log
git bisect start
# bad: [77047588bc2eb113cfbdacf0e14a650c0fe3dd70] Merge branch 'dap4.dmh' into ddap4
git bisect bad 77047588bc2eb113cfbdacf0e14a650c0fe3dd70
# good: [4eaf7c2759270f7fafbee18f807c3e9d5112c587] Deleting extraneous white space in CMakeLists.txt.
git bisect good 4eaf7c2759270f7fafbee18f807c3e9d5112c587
# good: [62b9141af1b58ca9b051c5e34fd31ace9cf4125a] Reverting the delete from the tm2.pl script.
git bisect good 62b9141af1b58ca9b051c5e34fd31ace9cf4125a
# bad: [75c6670416da7eb7de4a3c0e6cb988d6a7adaf9e] Resource Leak, Coverity 1264408
git bisect bad 75c6670416da7eb7de4a3c0e6cb988d6a7adaf9e
# bad: [3ef7eeb30a820ffb31cb78424769f2033cb7a38c] Merge remote-tracking branch 'upstream/master'
git bisect bad 3ef7eeb30a820ffb31cb78424769f2033cb7a38c
# bad: [eef6e4679bf3d900e57ce3cb84c9c8f9345d0cc7] Merged latest from trunk.
git bisect bad eef6e4679bf3d900e57ce3cb84c9c8f9345d0cc7
# good: [3ea6cc1b39b5a16aa88e26616647093bab0abfe2] Updated to reflect latest version, 4.3.0rc2
git bisect good 3ea6cc1b39b5a16aa88e26616647093bab0abfe2
# good: [57560487c7f2d6f5ccb40cc863c702f76d1c5938] Re-added Doxyfile, CTest script file to the branch.
git bisect good 57560487c7f2d6f5ccb40cc863c702f76d1c5938
# good: [d4e6e04d291290f153f18e16669223b90b1e171c] Merged latest from trunk.
git bisect good d4e6e04d291290f153f18e16669223b90b1e171c
# bad: [2ce8f6b5c94f4ed81af2ddcedc7d1009ad948bf5] Merged latest from trunk, plus a minor fix while waiting on Dennis's fix.
git bisect bad 2ce8f6b5c94f4ed81af2ddcedc7d1009ad948bf5
# bad: [4ffd3054b5f96bfb464087ffceffa3d2c5ad2539] 
git bisect bad 4ffd3054b5f96bfb464087ffceffa3d2c5ad2539
# bad: [85e527550eef71391be8fbde54e7f08dfe77fde9] Merged latest from Trunk.
git bisect bad 85e527550eef71391be8fbde54e7f08dfe77fde9
# good: [181aaf28c1dd3d26fce4212e424c1e5412634ace] Added initial 'make dist', 'make distcheck' support.
git bisect good 181aaf28c1dd3d26fce4212e424c1e5412634ace
# bad: [33d3d06971b1565997ef9717ff6eee56edf3c058] Added initial 'make dist', 'make distcheck' support to CMake-based builds.
git bisect bad 33d3d06971b1565997ef9717ff6eee56edf3c058
# bad: [946a7ab8b5c49388dbee1bc2a92ed0ed48920520] Fix contributed by Nath Gopalaswamy to large file problem reading netCDF classic or 64-bit offset files that have a UINT32_MAX flag for large last record size of a variable that has values larger than 1 byte.  This problem had previously been fixed for *writing* such data, but was only tested with an ncbyte variable.  Fixed test to demonstrate problem and the fix.
git bisect bad 946a7ab8b5c49388dbee1bc2a92ed0ed48920520
# first bad commit: [946a7ab8b5c49388dbee1bc2a92ed0ed48920520] Fix contributed by Nath Gopalaswamy to large file problem reading netCDF classic or 64-bit offset files that have a UINT32_MAX flag for large last record size of a variable that has values larger than 1 byte.  This problem had previously been fixed for *writing* such data, but was only tested with an ncbyte variable.  Fixed test to demonstrate problem and the fix.

So the first bad commit seems to be 946a7ab.

There were quite some compilation problems older versions so I'm not sure the result is valid.

[WardF]

The issue appears to be related, at least with one of the tests, to an error in the cmake config.h template. Diagnosing that now.

[WardF]

Fixed.

[t-b]

@WardF Is the fix already commited?

I've retried here with 68f5e3a (No idea how this was possibly working, previously., 2017-05-23) and now
I get

$./nc_test/nc_test_large_files 

*** Testing large files, slowly.
line 118 of /home/firma/devel/netcdf-c/nc_test/large_files.c: NetCDF: One or more variable sizes violate format constraints
*** Creating large file ./large_files.nc..

I've used the same cmake invocation as in #385 (comment).

[WardF]

Ok, thanks again for pointing this out. I didn't test with the specific compilation flag, although the fix was tested as working on a 32-bit ubuntu 16.04 system. I will reopen this but will not have a chance to investigate until next week (I'm out of the office for the rest of the week).

[WardF]

Working on master (with all the fixes merged) I see the following (it was built using the command you outlined above).

$ file nc_test_large_files 
nc_test_large_files: Mach-O executable i386

$ ./nc_test_large_files 

*** Testing large files, slowly.
*** Creating large file ./large_files.nc...ok
*** Reading large file ./large_files.nc...ok
*** Tests successful!

[WardF]

Can you try with master and let me know if the issue persists on your end @t-b ?

[WardF]

Closing for the time being, can re-open if confirmed it is still occurring.

[t-b]

@WardF Thanks for the update. I've tried again using 89ebc2b7 (Fixed an issue on visual studio which can result in a race state (more or less) when dap4 AND hdf4 support is enabled., 2017-06-01).

It still fails here with a 32bit executable

 $./nc_test/nc_test_quick_large_files 

*** Testing large files, quickly.
*** Creating ./quick_large_files.nc for 64-bit offset large file test...ok
*** Now a 64-bit offset, large file, fixed var test...ok
*** Now writing some values...ok
*** Now a classic file which will fail...ok
*** Now a 64-bit offset, simple fixed var create test...ok
*** Now a 64-bit offset, over-sized file that will fail...ok
*** Now a 64-bit offset, record var file...ok
*** Now a classic file that's too big and will fail...ok
*** Now a classic file with recs and one fixed will fail...ok
*** Now a classic file with one large rec var...ok
*** Now a classic file with extra-large last record var...Sorry! Unexpected result, /home/firma/devel/netcdf-c/nc_test/quick_large_files.c, line: 436
firma@thomas-debian-x64:~/devel/netcdf-c/build (master)
$./nc_test/nc_test_large_files 

*** Testing large files, slowly.
line 118 of /home/firma/devel/netcdf-c/nc_test/large_files.c: NetCDF: One or more variable sizes violate format constraints
*** Creating large file ./large_files.nc...firma@thomas-debian-x64:~/devel/netcdf-c/build (master)
$

 $file ./nc_test/nc_test_quick_large_files
./nc_test/nc_test_quick_large_files: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=13b325f2f214172f644ef02654b583c37d903aff, not stripped

I've tried also 64bit and that works here.

I can help debug it here, if you can give me some pointers where to start.

I'm on debian stretch.

[t-b]

In both test cases nc_enddef returns an error.

I'm suspecting, wildly without knowing anything, that maybe the large file support is not working here.

The cmake output

-- The C compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Performing Test LIBTOOL_HAS_NO_UNDEFINED
-- Performing Test LIBTOOL_HAS_NO_UNDEFINED - Success
-- Performing Test CC_HAS_WCONVERSION
-- Performing Test CC_HAS_WCONVERSION - Success
-- Performing Test CC_HAS_SHORTEN_64_32
-- Performing Test CC_HAS_SHORTEN_64_32 - Failed
-- Found Math library: /usr/lib/i386-linux-gnu/libm.so
-- Performing Test __CHAR_UNSIGNED__
-- Performing Test __CHAR_UNSIGNED__ - Failed
-- Looking for math.h
-- Looking for math.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for alloca.h
-- Looking for alloca.h - found
-- Looking for malloc.h
-- Looking for malloc.h - found
-- Looking for ctype.h
-- Looking for ctype.h - found
-- Looking for dirent.h
-- Looking for dirent.h - found
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for errno.h
-- Looking for errno.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for getopt.h
-- Looking for getopt.h - found
-- Looking for stdbool.h
-- Looking for stdbool.h - found
-- Looking for locale.h
-- Looking for locale.h - found
-- Looking for stdio.h
-- Looking for stdio.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for stdarg.h
-- Looking for stdarg.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for sys/dir.h
-- Looking for sys/dir.h - found
-- Looking for sys/ndir.h
-- Looking for sys/ndir.h - not found
-- Looking for sys/param.h
-- Looking for sys/param.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/wait.h
-- Looking for sys/wait.h - found
-- Looking for sys/mman.h
-- Looking for sys/mman.h - found
-- Looking for sys/resource.h
-- Looking for sys/resource.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for pstdint.h
-- Looking for pstdint.h - found
-- Looking for endian.h
-- Looking for endian.h - found
-- Looking for BaseTsd.h
-- Looking for BaseTsd.h - not found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for include files time.h, sys/time.h
-- Looking for include files time.h, sys/time.h - found
-- Looking for isfinite
-- Looking for isfinite - found
-- Looking for isnan
-- Looking for isnan - found
-- Looking for isinf
-- Looking for isinf - found
-- Looking for signbit
-- Looking for signbit - found
-- Looking for st_blksize
-- Looking for st_blksize - not found
-- Looking for alloca
-- Looking for alloca - found
-- Looking for snprintf
-- Looking for snprintf - found
-- Looking for vprintf
-- Looking for vprintf - found
-- Check size of char
-- Check size of char - done
-- Check size of double
-- Check size of double - done
-- Check size of float
-- Check size of float - done
-- Check size of int
-- Check size of int - done
-- Check size of uint
-- Check size of uint - done
-- Check size of long
-- Check size of long - done
-- Check size of long long
-- Check size of long long - done
-- Check size of unsigned long long
-- Check size of unsigned long long - done
-- Check size of off_t
-- Check size of off_t - done
-- Check size of off64_t
-- Check size of off64_t - failed
-- Check size of short
-- Check size of short - done
-- Check size of ushort
-- Check size of ushort - done
-- Check size of _Bool
-- Check size of _Bool - done
-- Check size of size_t
-- Check size of size_t - done
-- Check size of ssize_t
-- Check size of ssize_t - done
-- Check size of ptrdiff_t
-- Check size of ptrdiff_t - done
-- Check size of __int64
-- Check size of __int64 - failed
-- Check size of int64_t
-- Check size of int64_t - done
-- Check size of uint64_t
-- Check size of uint64_t - done
-- Check size of unsigned char
-- Check size of unsigned char - done
-- Check size of unsigned short int
-- Check size of unsigned short int - done
-- Check size of unsigned int
-- Check size of unsigned int - done
-- Check size of long long
-- Check size of long long - done
-- Check size of unsigned long long
-- Check size of unsigned long long - done
-- Looking for fsync
-- Looking for fsync - found
-- Looking for strlcat
-- Looking for strlcat - not found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for strchr
-- Looking for strchr - found
-- Looking for strrchr
-- Looking for strrchr - found
-- Looking for strcat
-- Looking for strcat - found
-- Looking for strcpy
-- Looking for strcpy - found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strndup
-- Looking for strndup - found
-- Looking for strcasecmp
-- Looking for strcasecmp - found
-- Looking for strtod
-- Looking for strtod - found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for strtoull
-- Looking for strtoull - found
-- Looking for strstr
-- Looking for strstr - found
-- Looking for mkstemp
-- Looking for mkstemp - found
-- Looking for rand
-- Looking for rand - found
-- Looking for random
-- Looking for random - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for MPI_Comm_f2c
-- Looking for MPI_Comm_f2c - not found
-- Looking for memmove
-- Looking for memmove - found
-- Looking for memcmp
-- Looking for memcmp - found
-- Looking for getpagesize
-- Looking for getpagesize - found
-- Looking for sysconf
-- Looking for sysconf - found
-- Looking for getrlimit
-- Looking for getrlimit - found
-- Looking for _filelengthi64
-- Looking for _filelengthi64 - not found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for mremap
-- Looking for mremap - found
-- Found bash: /bin/bash
-- CMAKE_SOURCE_DIR=/home/firma/devel/netcdf-c
-- CMAKE_BINARY_DIR=/home/firma/devel/netcdf-c/build
-- CMAKE_CURRENT_SOURCE_DIR=/home/firma/devel/netcdf-c/nc_test
-- CMAKE_CURRENT_BINARY_DIR=/home/firma/devel/netcdf-c/build/nc_test


Configuration Summary:

-- Building Shared Libraries:     OFF
-- Building netCDF-4:             OFF
-- Building DAP2 Support:         OFF
-- Building DAP4 Support:         OFF
-- Building Utilities:            OFF

Tests Enabled:              ON
-- DAP Remote Tests:  ON
-- Extra Tests:       OFF
-- Coverage Tests:    OFF
-- Parallel Tests:    OFF
-- Large File Tests:  ON
-- Extreme Numbers:   ON

Compiler:

-- Build Type:           DEBUG
-- CMAKE_C_COMPILER:     /usr/bin/cc
-- CMAKE_C_FLAGS:        -m32
-- CMAKE_C_FLAGS_DEBUG:  -g -Wall -Wconversion
-- Linking against:      /usr/lib/i386-linux-gnu/libm.so

# NetCDF C Configuration Summary
==============================

# General
-------
NetCDF Version:		4.5.1-development
Configured On:		Sa 3. Jun 12:38:39 CEST 2017
Host System:		x86_64-Linux-4.9.0-2-amd64
Build Directory: 	/home/firma/devel/netcdf-c/build
Install Prefix:         /usr/local

# Compiling Options
-----------------
C Compiler:		/usr/bin/cc
CFLAGS:			-m32 -g -Wall -Wconversion
CPPFLAGS:		 
LDFLAGS:		  -Wl,--no-undefined
AM_CFLAGS:		
AM_CPPFLAGS:		
AM_LDFLAGS:		
Shared Library:		no
Static Library:		yes
Extra libraries:	-lm

# Features
--------
NetCDF-2 API:		yes
HDF4 Support:		no
NetCDF-4 API:		no
NC-4 Parallel Support:	no
PNetCDF Support:	no
DAP2 Support:		no
DAP4 Support:		no
Diskless Support:	yes
MMap Support:		yes
JNA Support:		no

-- Configuring done
-- Generating done
-- Build files have been written to: /home/firma/devel/netcdf-c/build

config.h

#ifndef CONFIG_H
#define CONFIG_H

#ifdef _MSC_VER

/* Prevent an issue where there is a circular inclusion
   of winsock.h/windows.h.  This weird state occurs with
   libdap4 and hdf4 support. The solution comes from the
   following URL, found after a bit of research.

   Added in support of the 4.5.0-rc1.  Hello, future generations.

   * https://stackoverflow.com/questions/1372480/c-redefinition-header-files-winsock2-h

   */
   #define _WINSOCKAPI_

   #if _MSC_VER>=1900
     #define STDC99
   #endif
/* Define O_BINARY so that the appropriate flags
are set when opening a binary file on Windows. */

/* Disable a few warnings under Visual Studio, for the
   time being. */
   #include <io.h>
   #pragma warning( disable: 4018 4996 4244 4305 )
   #define unlink _unlink
   #define open _open
   #define close _close
   #define read _read
   #define lseek _lseeki64

   #define fstat _fstat64

   #define off_t __int64
   #define _off_t __int64

   #ifndef _OFF_T_DEFINED
   #define _OFF_T_DEFINED
   #endif

   #ifdef _WIN32
    #ifndef strcasecmp
      #define strcasecmp _stricmp
    #endif

    #ifndef snprintf
     #if _MSC_VER<1900
       #define snprintf _snprintf
     #endif
    #endif
   #endif


   #define strdup _strdup
   #define fdopen _fdopen
   #define write _write
   #define strtoll _strtoi64
#endif /*_MSC_VER */

/* #undef const */

#ifndef _FILE_OFFSET_BITS
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE64_SOURCE
#define _LARGEFILE_SOURCE
#endif

/* Define if building universal (internal helper macro) */
/* #undef AC_APPLE_UNIVERSAL_BUILD */

/* If true, will attempt to download and build netcdf-fortran. */
/* #undef BUILD_FORTRAN */

/* default file chunk cache nelems. */
#define CHUNK_CACHE_NELEMS 1009

/* default file chunk cache preemption policy. */
#define CHUNK_CACHE_PREEMPTION 0.75

/* default file chunk cache size in bytes. */
#define CHUNK_CACHE_SIZE 4194304

/* Define to one of `_getb67', `GETB67', `getb67' for Cray-2 and Cray-YMP
   systems. This function is required for `alloca.c' support on those systems.
   */
/* #undef CRAY_STACKSEG_END */

/* Define to 1 if using `alloca.c'. */
/* #undef C_ALLOCA */

/* num chunks in default per-var chunk cache. */
#define DEFAULT_CHUNKS_IN_CACHE 10

/* default chunk size in bytes */
#define DEFAULT_CHUNK_SIZE 4194304

/* set this only when building a DLL under MinGW */
/* #undef DLL_EXPORT */

/* set this only when building a DLL under MinGW */
/* #undef DLL_NETCDF */

/* if true, build DAP2 and DAP4 Client */
/* #undef ENABLE_DAP */

/* if true, build DAP4 Client */
/* #undef ENABLE_DAP4 */

/* if true, build DAP4 Client */
/* #undef ENABLE_DAP4 */

/* if true, enable DAP group names */
/* #undef ENABLE_DAP_GROUPS */

/* if true, do remote tests */
#define ENABLE_DAP_REMOTE_TESTS 1

/* define the possible sources for remote test servers */
#define REMOTETESTSERVERS	"remotetest.unidata.ucar.edu,jetstream.unidata.ucar.edu"

/* if true, run extra tests which may not work yet */
/* #undef EXTRA_TESTS */

/* use HDF5 1.6 API */
/* #undef H5_USE_16_API */

/* Define to 1 if you have `alloca', as a function or macro. */
#define HAVE_ALLOCA 1

/* Define to 1 if you have <alloca.h> and it should be used (not on Ultrix).
   */
#define HAVE_ALLOCA_H 1

/* Define to 1 if you have the <ctype.h> header file. */
#define HAVE_CTYPE_H 1

/* Define to 1 if you have hdf5_coll_metadata_ops */
/* #undef HDF5_HAS_COLL_METADATA_OPS */

/* Is CURLINFO_RESPONSE_CODE defined */
/* #undef HAVE_CURLINFO_RESPONSE_CODE */

/* Is CURLOPT_CHUNK_BGN_FUNCTION defined */
/* #undef HAVE_CURLOPT_CHUNK_BGN_FUNCTION */

/* Is CURLOPT_KEYPASSWD defined */
/* #undef HAVE_CURLOPT_KEYPASSWD */

/* Is CURLOPT_PASSWORD defined */
/* #undef HAVE_CURLOPT_PASSWORD */

/* Is CURLOPT_USERNAME defined */
/* #undef HAVE_CURLOPT_USERNAME */

/* Define to 1 if you have the declaration of `isfinite', and to 0 if you
   don't. */
#define HAVE_DECL_ISFINITE 1

/* Define to 1 if you have the declaration of `isinf', and to 0 if you don't.
   */
#define HAVE_DECL_ISINF 1

/* Define to 1 if you have the declaration of `isnan', and to 0 if you don't.
   */
#define HAVE_DECL_ISNAN 1

/* Define to 1 if you have the declaration of `signbit', and to 0 if you
   don't. */
#define HAVE_DECL_SIGNBIT 1

/* Define to 1 if you have the <dirent.h> header file, and it defines `DIR'.
   */
#define HAVE_DIRENT_H 1

/* Define to 1 if you have the <dlfcn.h> header file. */
#define HAVE_DLFCN_H 1

/* Define to 1 if you don't have `vprintf' but do have `_doprnt.' */
/* #undef HAVE_DOPRNT */

/* Define to 1 if you have the <errno.h> header file. */
#define HAVE_ERRNO_H 1

/* Define to 1 if you have the <fcntl.h> header file. */
#define HAVE_FCNTL_H 1

/* Define to 1 if you have the BaseTsd.h header file. */
/* #undef HAVE_BASETSD_H */

/* Define if we have filelengthi64. */
/* #undef HAVE_FILE_LENGTH_I64 */


/* Define to 1 if you have the `fsync' function. */
#define HAVE_FSYNC 1

/* Define to 1 if you have the <getopt.h> header file. */
#define HAVE_GETOPT_H 1

/* Define to 1 if you have the `getpagesize' function. */
#define HAVE_GETPAGESIZE 1

/* Define to 1 if you have the `getrlimit' function. */
#define HAVE_GETRLIMIT 1

/* Define to 1 if you have the `gettimeofday' function. */
#define HAVE_GETTIMEOFDAY 1

/* Define to 1 if you have the `H5free_memory' function. */
/* #undef HAVE_H5FREE_MEMORY */

/* Define to 1 if you have the `H5Pget_fapl_mpio' function. */
/* #undef HAVE_H5PGET_FAPL_MPIO */

/* Define to 1 if you have the `H5Pget_fapl_mpiposix' function. */
/* #undef HAVE_H5PGET_FAPL_MPIPOSIX */

/* Define to 1 if you have the `H5Pset_deflate' function. */
/* #undef HAVE_H5PSET_DEFLATE */

/* if true, netcdf4 file properties will be set using H5Pset_libver_bounds */
/* #undef HDF5_HAS_LIBVER_BOUNDS */

/* Define to 1 if you have the `H5Z_SZIP' function. */
/* #undef HAVE_H5Z_SZIP */

/* Define to 1 if you have the <hdf5.h> header file. */
/* #undef HAVE_HDF5_H */

/* Define to 1 if you have the <hdf5.h> header file. */
/* #undef HAVE_HDF5_HL_H */

/* Define to 1 if the system has the type `int64'. */
/* #undef HAVE_INT64 */

/* Define to 1 if you have the <inttypes.h> header file. */
#define HAVE_INTTYPES_H 1

/* Define to 1 if you have the `df' library (-ldf). */
/* #undef HAVE_LIBDF */

/* Define to 1 if you have the `dl' library (-ldl). */
/* #undef HAVE_LIBDL */

/* Define to 1 if you have the `jpeg' library (-ljpeg). */
/* #undef HAVE_LIBJPEG */

/* Define to 1 if you have the `m' library (-lm). */
#define HAVE_LIBM 1

/* Define to 1 if you have the `mfhdf' library (-lmfhdf). */
/* #undef HAVE_LIBMFHDF */

/* Define to 1 if you have the `pnetcdf' library (-lpnetcdf). */
/* #undef HAVE_LIBPNETCDF */

/* Define to 1 if you have the <locale.h> header file. */
#define HAVE_LOCALE_H 1

/* Define to 1 if the system has the type `longlong'. */
/* #undef HAVE_LONGLONG */

/* Define to 1 if the system has the type 'long long int'. */
#define HAVE_LONG_LONG_INT 1

/* Define to 1 if you have the <malloc.h> header file. */
#define HAVE_MALLOC_H 1

/* Define to 1 if you have the `memcmp' function. */
#define HAVE_MEMCMP 1

/* Define to 1 if you have the `memmove' function. */
#define HAVE_MEMMOVE 1

/* Define to 1 if you have the <memory.h> header file. */
#define HAVE_MEMORY_H 1

/* Define to 1 if you have the <mfhdf.h> header file. */
/* #undef HAVE_MFHDF_H */

/* Define to 1 if you have the `mkstemp' function. */
#define HAVE_MKSTEMP 1

/* Define to 1 if you have a working `mmap' system call. */
#define HAVE_MMAP 1

/* Define to 1 if you have the `MPI_Comm_f2c' function. */
/* #undef HAVE_MPI_COMM_F2C */

/* Define to 1 if you have the `mremap' function. */
#define HAVE_MREMAP 1

/* Define to 1 if you have the <ndir.h> header file, and it defines `DIR'. */
/* #undef HAVE_NDIR_H */

/* Define to 1 if the system has the type `ptrdiff_t'. */
#define HAVE_PTRDIFF_T 1

/* Define to 1 if you have the `rand' function. */
#define HAVE_RAND 1

/* Define to 1 if you have the `random' function. */
#define HAVE_RANDOM 1

/* Define to 1 if the system has the type `size_t'. */
#define HAVE_SIZE_T 1

/* Define to 1 if you have the `snprintf' function. */
#define HAVE_SNPRINTF 1

/* Define to 1 if the system has the type `ssize_t'. */
#define HAVE_SSIZE_T 1

/* Define to 1 if you have the <stdarg.h> header file. */
#define HAVE_STDARG_H 1

/* Define to 1 if stdbool.h conforms to C99. */
#define HAVE_STDBOOL_H 1

/* Define to 1 if you have the <stdint.h> header file. */
#define HAVE_STDINT_H 1

/* Define to 1 if you have the <stdio.h> header file. */
#define HAVE_STDIO_H 1

/* Define to 1 if you have the <stdlib.h> header file. */
#define HAVE_STDLIB_H 1

/* Define to 1 if you have the `strcasecmp' function. */
#define HAVE_STRCASECMP 1

/* Define to 1 if you have the `strcat' function. */
#define HAVE_STRCAT 1

/* Define to 1 if you have the `strchr' function. */
#define HAVE_STRCHR 1

/* Define to 1 if you have the `strcpy' function. */
#define HAVE_STRCPY 1

/* Define to 1 if you have the `strdup' function. */
#define HAVE_STRDUP 1

/* Define to 1 if you have the `strndup` function. */
#define HAVE_STRNDUP

/* Define to 1 if you have the `strerror' function. */
#define HAVE_STRERROR 1

/* Define to 1 if you have the <strings.h> header file. */
#define HAVE_STRINGS_H 1

/* Define to 1 if you have the <signal.h> header file. */
#define HAVE_SIGNAL_H 1

/* Define to 1 if you have the <string.h> header file. */
#define HAVE_STRING_H 1

/* Define to 1 if you have the `strlcat' function. */
/* #undef HAVE_STRLCAT */

/* Define to 1 if you have the `strrchr' function. */
#define HAVE_STRRCHR 1

/* Define to 1 if you have the `strstr' function. */
#define HAVE_STRSTR 1

/* Define to 1 if you have the `strtod' function. */
#define HAVE_STRTOD 1

/* Define to 1 if you have the `strtoll' function. */
#define HAVE_STRTOLL 1

/* Define to 1 if you have the `strtoull' function. */
#define HAVE_STRTOULL 1

/* Define to 1 if you have the `stroull' function. */
/* #undef HAVE_STROULL */

/* Define to 1 if `st_blksize' is a member of `struct stat'. */
/* #undef HAVE_STRUCT_STAT_ST_BLKSIZE */

/* Define to 1 if your `struct stat' has `st_blksize'. Deprecated, use
   `HAVE_STRUCT_STAT_ST_BLKSIZE' instead. */
/* #undef HAVE_ST_BLKSIZE */

/* Define to 1 if you have the `sysconf' function. */
#define HAVE_SYSCONF 1

/* Define to 1 if you have the <sys/dir.h> header file, and it defines `DIR'.
   */
#define HAVE_SYS_DIR_H 1

/* Define to 1 if you have the <sys/ndir.h> header file, and it defines `DIR'.
   */
/* #undef HAVE_SYS_NDIR_H */

/* Define to 1 if you have the <sys/param.h> header file. */
#define HAVE_SYS_PARAM_H 1

/* Define to 1 if you have the <sys/resource.h> header file. */
#define HAVE_SYS_RESOURCE_H 1

/* Define to 1 if you have the <sys/stat.h> header file. */
#define HAVE_SYS_STAT_H 1

/* Define to 1 if you have the <sys/time.h> header file. */
#define HAVE_SYS_TIME_H 1

/* Define to 1 if you have the <sys/types.h> header file. */
#define HAVE_SYS_TYPES_H 1

/* Define to 1 if you have <sys/wait.h> that is POSIX.1 compatible. */
#define HAVE_SYS_WAIT_H 1

/* Define to 1 if the system has the type `uchar'. */
/* #undef HAVE_UCHAR */

/* Define to 1 if the system has the type `uint'. */
#define HAVE_UINT 1

/* Define to 1 if the system has the type `uint64'. */
/* #undef HAVE_UINT64 */

/* Define to 1 if you have the <unistd.h> header file. */
#define HAVE_UNISTD_H 1
/* #undef YY_NO_UNISTD_H */

/* Define to 1 if the system has the type 'unsigned long long int'. */
#define HAVE_UNSIGNED_LONG_LONG_INT 1

/* Define to 1 if the system has the type `ushort'. */
#define HAVE_USHORT 1

/* Define to 1 if you have the `vprintf' function. */
#define HAVE_VPRINTF 1

/* Define to 1 if the system has the type `_Bool'. */
#define HAVE__BOOL 1

/* if true, H5free_memory() will be used to free hdf5-allocated memory in
   nc4file. */
/* #undef HDF5_HAS_H5FREE */

/* if true, hdf5 has parallelism enabled */
/* #undef HDF5_PARALLEL */

/* if true, include JNA bug fix */
/* #undef JNA */

/* do large file tests */
#define LARGE_FILE_TESTS 1

/* If true, turn on logging. */
/* #undef LOGGING */

/* max size of the default per-var chunk cache. */
#define MAX_DEFAULT_CACHE_SIZE 67108864

/* min blocksize for posixio. */
#define NCIO_MINBLOCKSIZE 256

/* no IEEE float on this platform */
/* #undef NO_IEEE_FLOAT */

#define BUILD_V2 1
/* #undef ENABLE_DOXYGEN */
/* #undef ENABLE_INTERNAL_DOCS */
/* #undef VALGRIND_TESTS */
/* #undef ENABLE_CDMREMOTE */
/* #undef USE_HDF5 */
/* #undef ENABLE_FILEINFO */
/* #undef TEST_PARALLEL */
/* #undef BUILD_RPC */
/* #undef USE_X_GETOPT */
#define ENABLE_EXTREME_NUMBERS 1

/* do not build the netCDF version 2 API */
/* #undef NO_NETCDF_2 */

/* no stdlib.h */
/* #undef NO_STDLIB_H */

/* no sys_types.h */
/* #undef NO_SYS_TYPES_H */

/* Name of package */
#define PACKAGE "netcdf"

/* Define to the address where bug reports for this package should be sent. */
#define PACKAGE_BUGREPORT "[email protected]"

/* Define to the full name of this package. */
#define PACKAGE_NAME "netCDF"

/* Define to the full name and version of this package. */
#define PACKAGE_STRING "netCDF 4.5.1-development"

/* Define to the one symbol short name of this package. */
#define PACKAGE_TARNAME "netcdf"

/* Define to the home page for this package. */
#define PACKAGE_URL ""

/* Define to the version of this package. */
#define PACKAGE_VERSION "4.5.1-development"

/* The size of `ulonglong` as computed by sizeof. */
#define SIZEOF_ULONGLONG 8

/* The size of `longlong` as computed by sizeof. */
#define SIZEOF_LONGLONG 8

/* The size of `char` as computed by sizeof. */
#define SIZEOF_CHAR 1

/* The size of `uchar` as computed by sizeof. */
#define SIZEOF_UCHAR 1

/* The size of `ssize_t` as computed by sizeof. */
#define SIZEOF_SSIZE_T 4

/* The size of `__int64` found on Windows systems. */
/* #undef SIZEOF___INT64 */

/* The size of `void*` as computed by sizeof. */
#define SIZEOF_VOIDSTAR 4

/* The size of `short` as computed by sizeof. */
/* #undef SIZEOF_OFF64_T */

/* The size of `double', as computed by sizeof. */
#define SIZEOF_DOUBLE 8

/* The size of `float', as computed by sizeof. */
#define SIZEOF_FLOAT 4

/* The size of `int', as computed by sizeof. */
#define SIZEOF_INT 4

/* The size of `long', as computed by sizeof. */
#define SIZEOF_LONG 4

/* The size of `long long', as computed by sizeof. */
#define SIZEOF_LONG_LONG 8

/* The size of `off_t', as computed by sizeof. */
#define SIZEOF_OFF_T 4

/* The size of `ptrdiff_t', as computed by sizeof. */
#define SIZEOF_PTRDIFF_T 4

/* The size of `short', as computed by sizeof. */
#define SIZEOF_SHORT 2

/* The size of `size_t', as computed by sizeof. */
#define SIZEOF_SIZE_T 4

/* The size of `uint', as computed by sizeof. */
#define SIZEOF_UINT 4

/* The size of `unsigned int', as computed by sizeof. */
#define SIZEOF_UNSIGNED_INT 4

/* The size of `unsigned long long', as computed by sizeof. */
#define SIZEOF_UNSIGNED_LONG_LONG 8

/* The size of `unsigned short int', as computed by sizeof. */
#define SIZEOF_UNSIGNED_SHORT_INT 2

/* The size of `ushort', as computed by sizeof. */
#define SIZEOF_USHORT 2

/* The size of `void*', as computed by sizeof. */
#define SIZEOF_VOIDP 4

/* If using the C implementation of alloca, define if you know the
   direction of stack growth for your system; otherwise it will be
   automatically deduced at runtime.
	STACK_DIRECTION > 0 => grows toward higher addresses
	STACK_DIRECTION < 0 => grows toward lower addresses
	STACK_DIRECTION = 0 => direction of growth unknown */
/* #undef STACK_DIRECTION */

/* Define to 1 if you have the ANSI C header files. */
/* #undef STDC_HEADERS */

/* Place to put very large netCDF test files. */
#define TEMP_LARGE "."

/* Define to 1 if you can safely include both <sys/time.h> and <time.h>. */
#define TIME_WITH_SYS_TIME 1

/* if true, build DAP Client */
/* #undef USE_DAP */

/* if true, include NC_DISKLESS code */
#define USE_DISKLESS 1

/* set this to use extreme numbers in tests */
#define USE_EXTREME_NUMBERS 1

/* if true, use ffio instead of posixio */
/* #undef USE_FFIO */

/* if true, include experimental fsync code */
/* #undef USE_FSYNC */

/* if true, use HDF4 too */
/* #undef USE_HDF4 */

/* If true, use use wget to fetch some sample HDF4 data, and then test against
   it. */
/* #undef USE_HDF4_FILE_TESTS */

/* if true, enable dynamic loading support */
/* #undef USE_LIBDL */

/* if true, use mmap for in-memory files */
#define USE_MMAP 1

/* if true, build netCDF-4 */
/* #undef USE_NETCDF4 */

/* build the netCDF version 2 API */
#define USE_NETCDF_2 1

/* if true, pnetcdf or parallel netcdf-4 is in use */
/* #undef USE_PARALLEL */

/* if true, parallel netcdf-4 is in use */
/* #undef USE_PARALLEL4 */

/* if true, compile in parallel netCDF-4 based on MPI/IO */
/* #undef USE_PARALLEL_MPIO */

/* if true, compile in parallel netCDF-4 based on MPI/POSIX */
/* #undef USE_PARALLEL_POSIX */

/* if true, parallel netCDF is used */
/* #undef USE_PNETCDF */

/* if true, use stdio instead of posixio */
/* #undef USE_STDIO */

/* if true, compile in szip compression in netCDF-4 variables */
/* #undef USE_SZIP */

/* if true, compile in zlib compression in netCDF-4 variables */
/* #undef USE_ZLIB */

/* Version number of package */
#define VERSION "4.5.1-development"

/* Define WORDS_BIGENDIAN to 1 if your processor stores words with the most
   significant byte first (like Motorola and SPARC, unlike Intel). */
#if defined AC_APPLE_UNIVERSAL_BUILD
# if defined __BIG_ENDIAN__
#  define WORDS_BIGENDIAN 1
# endif
#else
# ifndef WORDS_BIGENDIAN
/* #undef WORDS_BIGENDIAN */
# endif
#endif

/* Enable large inode numbers on Mac OS X 10.5.  */
#ifndef _DARWIN_USE_64_BIT_INODE
# define _DARWIN_USE_64_BIT_INODE 1
#endif

/* Define for large files, on AIX-style hosts. */
/* #undef _LARGE_FILES */

/* Define to 1 if type `char' is unsigned and you are not using gcc.  */
#ifndef __CHAR_UNSIGNED__
/* #undef __CHAR_UNSIGNED__ */
#endif

/* Define to `long int' if <sys/types.h> does not define. */
/* #undef off_t */

/* Define to `unsigned int' if <sys/types.h> does not define. */
/* #undef size_t */

/* Define strcasecmp, snprintf on Win32 systems. */
#ifdef _WIN32
    #ifndef HAVE_STRCASECMP
	 #define strcasecmp _stricmp
    #endif

    #ifndef HAVE_SNPRINTF
	 #define snprintf _snprintf
    #endif
#endif

/* #undef WORDS_BIGENDIAN */

#include "ncconfigure.h"

#endif

[t-b]

The same tests pass on my MacOSX box.

[WardF]

I will continue to investigate. How much memory is in the 32-bit machine you're working with?

[t-b]

@WardF 64GB.

[t-b]

@WardF Just to clarify. I'm compiling a 32bit binary on a 64bit OS.

[t-b]

I've looked into the normal autotools based build system.

I've compile netcdf again with ./configure --disable-netcdf4 --enable-large-file-tests --disable-dap --disable-utilities CFLAGS=-m32 and then compared the output of config.h of that and the cmake one.

And there I get for example

$grep SIZEOF_OFF netcdf-c/config.h netcdf-c-configure/config.h
netcdf-c/config.h:/* #undef SIZEOF_OFF64_T */
netcdf-c/config.h:#define SIZEOF_OFF_T 4
netcdf-c-configure/config.h:#define SIZEOF_OFF_T 8

netcdf-c: built with cmake
netcdf-c-configure: built with configure

so these two don't agree on the size here.

With the configure build only large_files fails, quick_large_files passes.

[WardF]

Thank you @t-b that pointed me in the right direction. I've got this fixed and in the branch gh385, and it will make its way back into master. This was the primary roadblock to the next release candidate, so I will try to have that out tomorrow or Monday.

[t-b]

@WardF Thanks, I've just verified that the fix in v4.5.0-release-branch works. Now all tests pass with cmake.