Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Float32 converted Float64 during reference creation #532

Open
yuanjianz opened this issue Dec 3, 2024 · 3 comments
Open

Float32 converted Float64 during reference creation #532

yuanjianz opened this issue Dec 3, 2024 · 3 comments

Comments

@yuanjianz
Copy link

yuanjianz commented Dec 3, 2024

Kerchunk Verison: 0.2.7
I am using kerchunk to convert my netcdf files on s3 to json references. The original netcdf files are composed of variables of mixed data type of float32 and float64. After converted to json and read through xarray, all float data are converted to float64. I wonder is it an expected behaviour?

@martindurant
Copy link
Member

The output type must be consistent everywhere within a variable, sorry.

@yuanjianz
Copy link
Author

Hi @martindurant, thanks for the reply. Sorry for the confusion. The netcdf files contain several variables and they are int, float and double. In the reference created, variables with float precision in the original netcdf are converted to double somehow.

Input netcdf:

	float PM25(time, lev, nf, Ydim, Xdim) ;
		PM25:_FillValue = NaNf ;
		PM25:fmissing_value = 1.e+15f ;
		PM25:grid_mapping = "cubed_sphere" ;
		PM25:long_name = "Particulate matter with radii < 2.5 um" ;
		PM25:regrid_method = "bilinear" ;
		PM25:standard_name = "Particulate matter with radii < 2.5 um" ;
		PM25:units = "ug m-3" ;
		PM25:valid_range = -1.e+15f, 1.e+15f ;
		PM25:vmax = 1.e+15f ;
		PM25:vmin = -1.e+15f ;
		PM25:coordinates = "lats lons" ;
	double Xdim(Xdim) ;
		Xdim:_FillValue = NaN ;
		Xdim:long_name = "Fake Longitude for GrADS Compatibility" ;
		Xdim:units = "degrees_east" ;
	double Ydim(Ydim) ;
		Ydim:_FillValue = NaN ;
		Ydim:long_name = "Fake Latitude for GrADS Compatibility" ;
		Ydim:units = "degrees_north" ;
	int nf(nf) ;
		nf:axis = "e" ;
		nf:grads_dim = "e" ;
		nf:long_name = "cubed-sphere face" ;
	double lons(nf, Ydim, Xdim) ;
		lons:_FillValue = NaN ;
		lons:long_name = "longitude" ;
		lons:units = "degrees_east" ;
	double lats(nf, Ydim, Xdim) ;
		lats:_FillValue = NaN ;
		lats:long_name = "latitude" ;
		lats:units = "degrees_north" ;
	double lev(lev) ;
		lev:long_name = "vertical level" ;
		lev:_FillValue = NaN ;
		lev:standard_name = "model_layers" ;
		lev:units = "layer" ;
		lev:positive = "down" ;
		lev:coordinate = "eta" ;
	int64 time(time) ;
		time:units = "days since 2019-07-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
...

output json and read through xarray:

	double PM25(time, lev, nf, Ydim, Xdim) ;
		PM25:_FillValue = NaN ;
		PM25:fmissing_value = 999999986991104. ;
		PM25:grid_mapping = "cubed_sphere" ;
		PM25:long_name = "Particulate matter with radii < 2.5 um" ;
		PM25:regrid_method = "bilinear" ;
		PM25:standard_name = "Particulate matter with radii < 2.5 um" ;
		PM25:units = "ug m-3" ;
		PM25:valid_range = -999999986991104., 999999986991104. ;
		PM25:vmax = 999999986991104. ;
		PM25:vmin = -999999986991104. ;
		PM25:coordinates = "lats lons" ;
	double Xdim(Xdim) ;
		Xdim:_FillValue = NaN ;
		Xdim:long_name = "Fake Longitude for GrADS Compatibility" ;
		Xdim:units = "degrees_east" ;
	double Ydim(Ydim) ;
		Ydim:_FillValue = NaN ;
		Ydim:long_name = "Fake Latitude for GrADS Compatibility" ;
		Ydim:units = "degrees_north" ;
	double lats(nf, Ydim, Xdim) ;
		lats:_FillValue = NaN ;
		lats:long_name = "latitude" ;
		lats:units = "degrees_north" ;
	double lev(lev) ;
		lev:long_name = "vertical level" ;
		lev:_FillValue = NaN ;
		lev:standard_name = "model_layers" ;
		lev:units = "layer" ;
		lev:positive = "down" ;
		lev:coordinate = "eta" ;
	double lons(nf, Ydim, Xdim) ;
		lons:_FillValue = NaN ;
		lons:long_name = "longitude" ;
		lons:units = "degrees_east" ;
	int nf(nf) ;
		nf:axis = "e" ;
		nf:grads_dim = "e" ;
		nf:long_name = "cubed-sphere face" ;
	int64 time(time) ;
		time:units = "days since 2019-07-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
...

@martindurant
Copy link
Member

OK, that's probably not supposed to happen. It would be worthwhile to see what h5py thinks the data type is when you get to here. I'm not sure if there is any configuration we can pass to h5py or if the upcasting is happening elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants