Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling for HTTPFileSystem URIs #24

Merged
merged 1 commit into from
May 17, 2023
Merged

Conversation

alexgleith
Copy link
Contributor

In order to load EMIT data over HTTP, it's best to use a token to authenticate using fsspec as documented in this comment.

This below is working code, but it requires this attached fix.

s3_url = "s3://lp-prod-protected/EMITL2ARFL.001/EMIT_L2A_RFL_001_20230123T004529_2302216_003/EMIT_L2A_RFL_001_20230123T004529_2302216_003.nc"
s3_url = s3_url.replace("s3://", "https://data.lpdaac.earthdatacloud.nasa.gov/")

fs = HTTPFileSystem(headers={
    "Authorization": f"bearer {token}"
})
# ds = xr.open_dataset(fs.open(s3_url))
ds = emit_xarray(fs.open(s3_url))
ds

@ebolch
Copy link
Contributor

ebolch commented May 17, 2023

Thanks @alexgleith

Glad you were able to get HTTPS access working. We are planning add some instructions for HTTPS access to a future notebook. For other users in the meantime, the easiest way to access and work with an EMIT granule via HTTPS would be something similar to what you posted above:

# Use earthaccess library to login and retrieve a token
earthaccess.login()

# Provide granule URL
url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/EMITL2ARFL.001/EMIT_L2A_RFL_001_20220903T163129_2224611_012/EMIT_L2A_RFL_001_20220903T163129_2224611_012.nc'

# Get an https fsspec session
fs = earthaccess.get_fsspec_https_session()

# open granule using xarray or emit_xarray function from emit_tools.py
with fs.open(url) as file:
    
    # Using xarray (only reads root group)
    dataset = xarray.open_dataset(file)

    # Or using the emit_xarray function
    dataset = emit_xarray(file, ortho=False)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants