Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nccopy -d 0 -c / a.nc b.nc not work #1936

Closed
an-ttt opened this issue Jan 26, 2021 · 5 comments · Fixed by #1944
Closed

nccopy -d 0 -c / a.nc b.nc not work #1936

an-ttt opened this issue Jan 26, 2021 · 5 comments · Fixed by #1944
Assignees
Milestone

Comments

@an-ttt
Copy link

an-ttt commented Jan 26, 2021

  • netcdf 4.7.4
  • win10 netcdf-c

nccopy -d 0 -c / a.nc b.nc not work;
My File a.nc has deflation level 9 and chunks, I am going to make b.nc no compression and no chunks, but failed.

After I changed the codes in three place, then it worked, but I am not sure it's a good fix

//if(option_deflate_level >= 0 || hasfilterspecforvar(ofqn)) //ocontig = NC_CHUNKED; if (option_deflate_level > 0 || hasfilterspecforvar(ofqn)) ocontig = NC_CHUNKED; //

} else if(option_deflate_level == 0) { /* special case; force off */ shuffle_out = 0; //deflate_out = 0; deflate_out = 1; // deflate_level_out = 0; }

//if(shuffle_out != 0 || deflate_out != 0) // NC_CHECK(nc_def_var_deflate(ogrp, o_varid, shuffle_out, deflate_out, deflate_level_out)); if (shuffle_out != 0 || (deflate_out != 0 && deflate_level_out != 0)) NC_CHECK(nc_def_var_deflate(ogrp, o_varid, shuffle_out, deflate_out, deflate_level_out)); //

@WardF WardF self-assigned this Jan 26, 2021
@WardF WardF added this to the 4.8.0 milestone Jan 26, 2021
@WardF
Copy link
Member

WardF commented Jan 26, 2021

Would it be possible for you to attach the file a.nc, so that I might duplicate your test as closely as possible? If that is not practical, just let me know and I'll create a file on my end.

@WardF
Copy link
Member

WardF commented Jan 26, 2021

The issue appears to be in nccopy.c:1133, where shuffle_out == 0 and deflate_out == 0. Diving in to nc_def_var_deflate(), I'm finding text that specifies it is unclear whether these values should specify 'disable' or 'leave as is'. It seems to me that there should be a way to turn off the compression. @edwardhartnett, @DennisHeimbigner, what am I missing here? It looks like work was put in to handle 'no shuffle, no compression' such that they don't disable compression, but I'm not finding an alternative way to disable compression.

@WardF
Copy link
Member

WardF commented Jan 26, 2021

@an-ttt A work-around would be to use ncdump a.nc > a.cdl and then ncgen -k 4 -o b.nc a.cdl; this introduces an intermediary file, but the end result is an uncompressed binary NetCDF file.

@DennisHeimbigner
Copy link
Collaborator

Sigh! I wish that the intertwining of filters and chunking was less implicit that it is.
Anyway, part of the problem is that -d 0 specifies a deflate level of zero and nccopy
is interpreting it as turning on deflation but at level 0, so chunking is implicit.
However elsewhere deflatelevel ==0 is interpreted as no-deflation.
I would have said that you should use the -F flag, but I see that it does not work either.
The code I created is such a frankenstein; it really needs revision AGAIN!

@an-ttt
Copy link
Author

an-ttt commented Jan 28, 2021

Good to know It's in fixing, thanks.
As to the a.nc, you can try any nc file that use the "nccopy -d9 ..." to get the deflation level 9 file to test.

DennisHeimbigner added a commit to DennisHeimbigner/netcdf-c that referenced this issue Jan 31, 2021
re: Issue Unidata#1936

The algorithm controlling interaction of -d 0 -F and -c / options
is incorrect.

The fix:
1. make -d 0 => no deflation
2. make -F properly use the filter specs to decide.

Also added a test case to ncdump/tst_nccopy4.sh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants