Cryo 84: Refactor tutorials to use earthaccess and xarray

[andypbarrett]

This PR updates the SMAP tutorials to use earthaccess and xarray

Methods are included to load SMAP L3 data into xarray.DataTree and add coordinates.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[github-actions]

👈 Launch a binder notebook on this branch for commit 1715ba0

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit a095d54

👈 Launch a binder notebook on this branch for commit 1ceeb9b

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

flamingbear commented on 2025-01-07T16:28:07Z
----------------------------------------------------------------

xarray names dimensions phony_dim0, phony_dim1, etc.

I actually think this is HDF5 and not xarray.

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

flamingbear commented on 2025-01-07T16:28:08Z
----------------------------------------------------------------

Line #1.    dt = xr.open_datatree(filelist[0], phony_dims='sort')

In my notebook I don't need phony_dims keyword, but I might have a different env, and not using h5netcdf. I hope that doesn't really change how the datatrees behave.

asteiker commented on 2025-02-10T16:31:46Z
----------------------------------------------------------------

I also do not need phony_dims and in fact this gives me an error when I use it. I am working in the Openscapes 2i2c hub with xarray v2024.11.0

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

flamingbear commented on 2025-01-07T16:28:08Z
----------------------------------------------------------------

I think here you can just access the dataset object and skip the update. see below

dt["Soil_Moisture_Retrieval_Data_AM"].ds = \
  dt["Soil_Moisture_Retrieval_Data_AM"].ds.rename(
      {
          'phony_dim_0': 'y', 
          'phony_dim_1': 'x', 
          'phony_dim_2': 'igbp_class'
      }

 

asteiker commented on 2025-02-10T16:37:27Z
----------------------------------------------------------------

Confirmed this updated using Matt's code w/o dt.update()

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

flamingbear commented on 2025-01-07T16:28:09Z
----------------------------------------------------------------

Of course I just said that you didn't need to do update but I don't actually see the Dimensions in the root doing it my way. And I just verified that behavior. It's probably a bug. So if you want the dims recognized I think you'll have to use the update function.

asteiker commented on 2025-02-10T16:40:39Z
----------------------------------------------------------------

I see the updated Dimensions in the root w/o using dt.update() so it works for me at least. Scratch that, I was looking at dimensions in those particular groups, not at the root level.

asteiker commented on 2025-02-10T16:43:48Z
----------------------------------------------------------------

Somehow I edited Matt's comment..?! And now I can't go back to re-edit. Confirming that it does look like dt.update() is needed to update dimensions at the root level.

[flamingbear]

@andypbarrett Unfortunately I think you are doing it properly for the moment, but this looks like the right direction. If I get a chance I'll see if I can create a stripped down example and file a bug on the repo.

[flamingbear]

Oh. One last thing. I noticed the versions for smap in the download data notebooks were 008 and I couldn't get any data without bumping them to 009.

[asteiker]

I also do not need phony_dims and in fact this gives me an error when I use it. I am working in the Openscapes 2i2c hub with xarray v2024.11.0

---

View entire conversation on ReviewNB

[asteiker]

Confirmed this updated using Matt's code w/o dt.update()

---

View entire conversation on ReviewNB

[asteiker]

I see the updated Dimensions in the root w/o using dt.update() so it works for me at least.

---

View entire conversation on ReviewNB

[asteiker]

Somehow I edited Matt's comment..?! And now I can't go back to re-edit. Confirming that it does look like dt.update() is needed to update dimensions at the root level.

---

View entire conversation on ReviewNB

[asteiker]

I received an error in this code block, pointing to line 6:

`ValueError: Invalid location identifier (invalid identifier type to function)

Reply via ReviewNB

[andypbarrett]

This error results from using h5py. We can now use xarray.DataTree so this notebook is largely obsolete. I wonder if we should remove it and combine any relevant info into other notebooks.

[asteiker]

I get the same error, this time for root.visititems(get_vars):

ValueError: Invalid location identifier (invalid identifier type to function)

Reply via ReviewNB

[andypbarrett]

See comment above

[asteiker]

Update from the on-prem URL to cloud: https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/SMAP/SPL4SMGP/007/2015/03/31/SMAP_L4_SM_gph_20150331T013000_Vv7032_001.h5

Reply via ReviewNB

[andypbarrett]

I wonder if we leave this rasterio access pattern out of the tutorials for the moment. As I recall I was playing around with regridding.

[asteiker]

I don't actually have this file in my smap_data directory. But I don't even see that directory in your branch: https://github.com/nsidc/NSIDC-Data-Tutorials/tree/CRYO-84/notebooks/SMAP Maybe this is missing?

Reply via ReviewNB

[asteiker]

After adding the file to my smap_data directory, I still get this error:

OSError: [Errno -101] NetCDF: HDF error: '/home/jovyan/other-repos/NSIDC-Data-Tutorials/notebooks/SMAP/smap_data/SMAP_L4_SM_gph_20150331T013000_Vv7032_001.h5'

I'm using xarray v2024.11.0 and rioxarray v0.17.0

Reply via ReviewNB

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

asteiker commented on 2025-02-10T21:23:57Z
----------------------------------------------------------------

I also see that h5py is not defined when I tried to run this code block. I don't want to lose this work but maybe this content should be moved elsewhere out of this notebook to be refined in a future iteration?

[asteiker]

@andypbarrett These are great additions to the existing SMAP notebooks, and appreciate the updates to point to the cloud copies. One big-picture question after reviewing the notebooks: Do these make sense to add as numbered tutorials that flow with the existing 01_download..., 02_read..., 03_smap_quality_flags tutorials? Potentially iterations of the 02_read tutorial?

[andypbarrett]

I was thinking the same thing about the numbered tutorials. I think the SMAP as DataTree tutorial can replace 01 and 02. However, the time series section of 02 requires open_dataset because datatree does not have a multi-file option.