Ncdump fails under visual studio for certain dap urls

[DennisHeimbigner]

re: #365
Following command fails under visual studio.
ncdump -h http://thredds.ucar.edu/thredds/dodsC/nexrad/composite/gini/n0r/1km/20170216/Level3_Composite_n0r_1km_20170216_1635.gini

The problem is that sscanf for windows does not appear to support
scanning 8bit integers: it appears to only allow scanning of characters.

Solution:
Scan the input as an integer (for type Byte) or unsigned int (for type UByte)
and then recast the result as a char or unsigned char.

Primary code fix is in libdap2/dapcvt.c#dapcvtattrval

[WardF]

Encountering build failures reported in ncgen.l when -DENABLE_HDF4=TRUE. Details can be seen here:

• http://my.cdash.org/index.php?project=netcdf-c

Failures not occuring when HDF4 support is disabled. Debugging now, if anything about these errors leaps out, let me know.

[WardF]

Looks like I found the compilation issue at least; ncgen.y was defining a variable of name constant, which is reserved with Visual Studio. renamed, rebuilding to run tests now.

[WardF]

Ok, compilation issue is not resolved, and not certain where it was introduced. Will update here when ready to merge. It looks like it may pre-date this branch.

[DennisHeimbigner]

Looking at a number of the failed tests here,
http://my.cdash.org/viewTest.php?onlyfailed&buildid=1139214
Some are because a previous call to e.g. ncdump failed and did not create a file
and some (tst_inmemory_nc3) are because a call to rm -fr result failed
to remove all the files in the directory -- permissions problem?
might be useful to temporarily add an ls -lR command to see what is in results
and what its permissions are.

[WardF]

It isn't a permissions problem, as far as I can tell. The failure only occurs when HDF4 support is turned on, with failures in the ncgen generated files. Still working to figure out what the issue is.

[WardF]

@DennisHeimbigner To rule out something environmental, can you see if the same occurs on your system?

[DennisHeimbigner]

I don't have the ability at the moment to test HDF4. It will take some time. =Dennis

…

On 2/21/2017 4:29 PM, Ward Fisher wrote: @DennisHeimbigner <https://github.com/DennisHeimbigner> To rule out something environmental, can you see if the same occurs on your system? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#366 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P2xC3JagIaP7iVJfTV-v3-n4Yg4P2ks5re3NGgaJpZM4MFxnz>.

[WardF]

I've confirmed that it still works on the system under 4.4.1.1, so that tentatively rules out environmental. I'll start walking through and looking for where this broke

[WardF]

Ok, I've solved the issue and will be pulling all the threads together to get this and the other pulls merged.

@DennisHeimbigner maybe you can shed some light on what's going on. The issue was in the bison-generated file ncgen/ncgeny.h. When generated on Windows, it is missing a number of defines. When generated on OSX, it contains the required defines.

The odd thing about this is that, surprisingly, the version of bison on my Windows VM is newer than the OSX version. Version 2.3 on OSX, version 2.4.2 on Windows. The file is generated with the following command:

$ bison -pncg -t -d ncgen.y

There are other commands to move the generated file into place; you can see all of these near the bottom of ncgen/CMakeLists.txt. I've attached a zip file containing the two generated .h files, with version number in the filename. I don't know bison, can you provide any insight into what is going on?

bison_files.zip

[DennisHeimbigner]

Under what circumstances are ncgeny.* and ncgenl.* being recreated by ctest or travis?
I was not aware that bison could compile under cmake/Visual-Studio.
The short term answer is to make it clear that

1. the lex and parse files should not be generated on the fly except by Unidata.
2. the generated files ncgeny.* and ncgenl.* are a fixed part of the distribution
   and the repository and are never deleted.

[WardF]

They are not automatically recreated, and are tracked in github, although they can be generated manually by running 'make makeparser'. Occasionally there have been bugs which require editing something in the source files which then require the corresponding files be re-generated, which is what happened at some point in this case. Windows/Visual studio arent involved in generating them on Windows; if cmake can find bison on the path, it will use it l,otherwise it will throw an error or warning.

It appears that we need to regenerate these files either on OS X/Linux, on the rare occasion they must be regenerated, or possibly with an older version of bison. If it's the latter, I am concerned that our build process will break in the future. I need to install a newer version under Linux and see if the results look like what we see on windows (broken) or OS X (works)

[WardF]

So it does look like the issue might be the newer bison. The files you generated and committed with the newer version lack the various #define statements, as does the ncgeny.h file generated under linux with bison 3.0.4. I'll play around with it, but looking at the bison man page, I don't see any obvious changes to how bison is being invoked.

I'm going to close out this issue once the tests pass and I merge it in; this probably merits its own issue.

[DennisHeimbigner]

Let me look at the release notes for bison. It may be that they just moved the statements. =Dennis

…

On 2/23/2017 12:47 PM, Ward Fisher wrote: So it does look like the issue might be the newer bison. The files you generated and committed with the newer version lack the various #define statements, as does the ncgeny.h file generated under linux with bison |3.0.4|. I'll play around with it, but looking at the bison man page, I don't see any obvious changes to how bison is being invoked. I'm going to close out this issue once the tests pass and I merge it in; this probably merits its own issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#366 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P2wh-xc_KLNE5Jd6wOwUIGAIPwuSYks5rfeJWgaJpZM4MFxnz>.

[DennisHeimbigner]

I was trying to install hdf4 and noticed that it has its own old version of ncgen. It is possible
that the wrong ncgen is being invoked?

[DennisHeimbigner]

Also, I note that the #defines for tokens is now an enum. Could that be causing problems?

[WardF]

Looking at previous version it was both an enum and a block of defined. The problem is that without the block of defines, we end up with a lot of undefined values when compiling but only with hdf4 turned on for some reason.

[WardF]

Also the issue isn't the invocation; Ncgen just fails to compile. I wonder if it's related to the hdf4 versions although that doesn't explain why it used to work until a bison update

[DennisHeimbigner]

BTW I note that hdf4 has a --disable-netcdf configure flag. Was that set when you build hdf4?

[WardF]

Yes.

[DennisHeimbigner]

Another question. Does it fail only when using hdf4 and visual studio? =Dennis

…

On 2/23/2017 1:40 PM, Ward Fisher wrote: Yes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#366 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3P2x30ZjzTzXr3HjHJfjE7FGXueWdJks5rfe7IgaJpZM4MFxnz>.

[WardF]

correct. It doesn't fail when building with hdf4 on other platforms, or when building with visual studio without hdf4 support. It was something about the combination of hdf4 + visual studio, which seems super odd.

[DennisHeimbigner]

Ok. Finally able to reproduce. Hopefully I can figure out what is happening.