Support for MOD14A1 MODIS/Terra Thermal Anomalies/Fire Daily L3 Global 1km SIN Grid V061

[abarciauskas-bgse]

👋🏽 thank you for adding HDF4 support! I am hoping to use it for one of NASA projects but ran into an error for this MOD14A1 MODIS/Terra Thermal Anomalies/Fire Daily L3 Global 1km SIN Grid V061 dataset.

The full reproducible example is here: https://nbviewer.org/gist/abarciauskas-bgse/1fa66a708a9b17472f063daec756d751. Note: you will need Earthdata LPDAAC S3 credentials configured in your environment for this to work, which you can get using https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials or https://earthaccess.readthedocs.io/.

The error is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 r = kerchunk.hdf4.HDF4ToZarr(MCD12Q1).translate()

File [/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py:135](https://hub.openveda.cloud/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py#line=134), in HDF4ToZarr.translate(self, filename, storage_options)
    132             roots.add((tag, ref))
    134 # hierarchical output
--> 135 output = self._descend_vg(*list(roots)[0])
    136 prot = fo.fs.protocol
    137 prot = prot[0] if isinstance(prot, tuple) else prot

File [/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py:187](https://hub.openveda.cloud/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py#line=186), in HDF4ToZarr._descend_vg(self, tag, ref)
    185 inf2 = self.tags[(t, r)]
    186 if t == "VG":
--> 187     tmp = self._descend_vg(t, r)
    188     if list(tmp)[0] == inf2["name"]:
    189         tmp = tmp[inf2["name"]]

File [/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py:206](https://hub.openveda.cloud/srv/conda/envs/notebook/lib/python3.11/site-packages/kerchunk/hdf4.py#line=205), in HDF4ToZarr._descend_vg(self, tag, ref)
    204 elif t == "SD":
    205     out["refs"] = inf2["data"][:-1]
--> 206     out["chunks"] = [_["chunk_length"] for _ in inf2["data"][-1]]
    207 elif t == "SDD":
    208     out["dims"] = inf2["dims"]

TypeError: 'int' object is not iterable

I started to investigate this by learning a bit more about HDF4 format and it's implementation in this dataset, but haven't yet reached a conclusion. I will continue to investigate this week but thought I should open an issue in case there are any clues @martindurant may have.

[martindurant]

hdf4 is of course experimental and not very tested at all. I don't immediately have a thought, and I'm not sure when I'll have a chance to look into it :| .

[abarciauskas-bgse]

No worries @martindurant thanks for responding. I'll leave this open for now as I'll continue to investigate when I have time. It's exciting that it's working for the other 2 modis products!

[martindurant]

I can get s3 credentials via earthaccess, but I still don't have permissions for the file. I assume I need to approve something, but what?

[abarciauskas-bgse]

I'm guessing the issue is that the code you are running is not located in the same region as the files. NASA Earthdata cloud buckets are only accessible when working in the same region. You can only access them outside that region using HTTPS. If you are interested, I could add you to the VEDA JupyterHub (where you actually don't need S3 credentials at all because it has role-based access to the LPDAAC bucket)

[martindurant]

Yes, please add me.

[abarciauskas-bgse]

Ok you should have an invite to the veda-analytics-access org and all-access github team. Once you have accepted that invitation you should be able to log in to https://hub.openveda.cloud

[martindurant]

OK, the special case for this particular array is in the dataset is, that it's made of exactly one chunk only. Fix will come soon.