-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing zlib (deflate) compression level from dmrpp files #954
Comments
You might check the version of the
It may be the case that these dmr++ files were created prior to the |
After reviewing the examples I think we'll need to appeal to @kyang2014 for his insights on this. |
I can point to some examples of NASA datasets/dmrpp that have this issue but I'm not sure how to find all of them. The file I linked earlier with metadata:
metadata:
This file below does have metadata:
|
Kent: There is a question about DMR++ files and the deflateLevel information. Can you answer it?
On Jul 8, 2024, at 4:02 PM, Ayush Nag ***@***.***> wrote:
I can point to some examples of NASA datasets/dmrpp that have this issue but I'm not sure how to find all of them. The file I linked earlier with deflateLevel was build_dmrpp version 3.20.13. Some of the files below are above that version number and still missing deflateLevel. Could it be a DAAC/collection level issue?
I don’t know the exact version where the deflateLevel starts appearing, but we added that when we implemented a specific feature that required the information. To simply decompress those chunk, you don’t need this information, but we have a feature where chunks from an HDF5 file can be directly passed through to a netCDF4 file so that the HDF5 file can be subset with the result stored in a netCDF4 file without the expensive decompression and compression operations being performed. For this to work, we need to set the deflate level of the chunks as they are written to the netCDF4 file.
James
PS. Nathan is right, we should ask Kent about which versions of the DMR++ builder have this feature. They will be recent versions.
… https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20240101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc.dmrpp
metadata:
<Attribute name="build_dmrpp" type="String">
<Value>3.20.13-664</Value>
</Attribute>
<Attribute name="bes" type="String">
<Value>3.20.13-664</Value>
</Attribute>
<Attribute name="libdap" type="String">
<Value>libdap-3.20.11-198</Value>
</Attribute>
https://archive.swot.podaac.earthdata.nasa.gov/podaac-swot-ops-cumulus-protected/SWOT_L2_LR_SSH_2.0/SWOT_L2_LR_SSH_Expert_017_463_20240705T134333_20240705T143458_PIC0_01.nc.dmrpp
metadata:
<Attribute name="build_dmrpp" type="String">
<Value>3.21.0-272</Value>
</Attribute>
<Attribute name="bes" type="String">
<Value>3.21.0-272</Value>
</Attribute>
<Attribute name="libdap" type="String">
<Value>libdap-3.21.0-70</Value>
</Attribute>
This file below does have deflateLevel
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/VNP14A1.002/VNP14A1.A2024189.h00v08.002.2024190091116/VNP14A1.A2024189.h00v08.002.2024190091116.h5.dmrpp
metadata:
<Attribute name="build_dmrpp" type="String">
<Value>3.21.0-46</Value>
</Attribute>
<Attribute name="bes" type="String">
<Value>3.21.0-46</Value>
</Attribute>
<Attribute name="libdap" type="String">
<Value>libdap-3.21.0-27</Value>
</Attribute>
—
Reply to this email directly, view it on GitHub <#954 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7Q4KV24TVGQEKWUOMK7ELZLMEAHAVCNFSM6AAAAABKRHRIHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVGQZDGNBXGQ>.
You are receiving this because you are subscribed to this thread.
|
I double checked the date I added the deflateLevel to the dmrpp file. It is Sep. 5th, 2023 with #809 And the tag for this pull request is https://github.com/OPENDAP/bes/releases/tag/3.20.13-845. which is bes-3.20.13-845. |
Thanks Kent.
… On Jul 10, 2024, at 9:18 AM, kyang2014 ***@***.***> wrote:
I double checked the date I added the deflateLevel to the dmrpp file. It is Sep. 5th, 2023 with #809 <#809>
And the tag for this pull request is https://github.com/OPENDAP/bes/releases/tag/3.20.13-845. which is bes-3.20.13-845.
So BES 3.20.13-664, the one listed as without having the deflateLevel, is before the time I added this feature. The other two examples that have deflateLevels are after the time I added this feature.
—
Reply to this email directly, view it on GitHub <#954 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7Q4KWZQE4JA7G6SL3BL33ZLUYB5AVCNFSM6AAAAABKRHRIHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGQ4TANBWHA>.
You are receiving this because you commented.
|
I am working on parsing dmrpp metadata files to match the format of zarr metadata files (parser PR here). The only piece of missing information I have encountered in dmrpp files is the zlib (deflate) compression level. It seems like the opendap dmrpp generator has this logic to retrieve and store this attribute as
deflateLevel
(see example below) but it is not implemented across many files. Many NASA datasets includingSWOT L2 SSH
,MUR L4 SST
, andIce-SAT
are missing this field.Example dmrpp file WITH zlib level
Example dmrpp file WITHOUT zlib level
The text was updated successfully, but these errors were encountered: