small range on netcdf floats?

[rgknox]

I've noticed that the CTSM parameter files have changed over to using double precision as a default for storing floating point numbers. In FATES we are still using "float" or single precision floating point variables, and I noticed an unexpected problem.

*Note that in fates, our parameter interface only allows for floating point values (double and single prec.)

We have a field called fates_logging_event_code, a scalar float. This should be able to hold a date in the form YYYYMMDD (maximum of 8 digits). I was running a test (on both cheyenne and my work-station), converting from text (cdl) to binary (nc), and then dumping the result (e.g. ncgen and then ncdump immediately afterwards. The number 20030101 was returning as 20030100. I tried more numbers, 20030102, 20030103 ... About half of them came back as the correct number, the others were 1 digit off. Seems like a rounding issue of some kind..

The simple solution was to just re-define fates_logging_event_code as a double precision variable. But I was mystified as to why the float was having trouble holding and converting this value. It should be large enough to accommodate 8 digits of precision.

I was curious if anyone else has seen this type of strange behavior. Its also a warning to others that the actual valid range of unsigned "floats" in netcdf might not actually be what they think (at least without having a good grasp of how to specify/convert their variables, like me).

https://www.unidata.ucar.edu/software/netcdf/netcdf/CDL-Data-Types.html

[ekluzek]

@rgknox, this seems about right to me. Rounding for single precision is about 1.e-8 and that's the level you are at. You can represent individual numbers to a higher degree of precision, but at about a relative error of 1.e-8 you'll start to see it rounding. So I would say this is happening as expected. But, yes the solution is to use double precision (or for these specific numbers you could represent them as integer).

[ekluzek]

Here's more information on machine epsilon than you likely want to know...

https://en.wikipedia.org/wiki/Machine_epsilon

[rgknox]

Thanks for the info @ekluzek, feel free to close if you like

[billsacks]

This seems resolved, so closing.