auth.get_s3_credentials() requires a .netrc

[scottyhq]

I was experimenting with only using a Bearer token to authorize all requests (#188) but it seems s3 credentials endpoints like https://data.nsidc.earthdatacloud.nasa.gov/s3credentials do not recognize this so you get re-directed to URS servers that expect basic user:password authentication:

(on a machine without a ~/.netrc)

import earthaccess #0.5.0
auth = earthaccess.login('interactive')
auth.get_s3_credentials('NSIDC')
#You're now authenticated with NASA Earthdata Login
#Using token with expiration date: 04/22/2023
#Using user provided credentials for EDL
#https://data.nsidc.earthdatacloud.nasa.gov/s3credentials
#Authentication with Earthdata Login failed with:
#HTTP Basic: Access denied.

[scottyhq]

Interestingly it looks like ASF credentials do work with just a token! looking at the curl logs the authentication path is different, with some endpoints requiring an accessToken cookie (NOTE: not my EDL_TOKEN, but something different), whereas the ASF endpoint only requires the asf-urs cookie which is what is used for actually downloading data:

import earthaccess #0.5.0
auth = earthaccess.login('interactive')
auth.get_s3_credentials('ASF')

Or, look at logs or cookies produced from:

curl -v -L -b cookie.txt -c cookie.txt -n $S3CREDSURL

It would be nice if all those endpoints used the same authentication flow (or if there was just one endpoint for all DAACs...)

[JessicaS11]

@scottyhq Turns out you were a day ahead of me in this discovery. @betolink and I were troubleshooting this yesterday. It turns out that NSIDC s3 endpoints aren't yet configured to handle the bearer tokens (as you said), but v0.5.0 wasn't fully tested for this. We're (mostly Luis, really) in the process of addressing this and a few other bugs (and migrate some testing from icepyx) for a release sometime [early] next week.

[scottyhq]

Yeah I appreciate the situation is tricky right now across DAACs!

I find myself more and more using remote servers where I'd rather not have a .netrc with my personal password (also best-practice from a security standpoint to just use time-limited tokens that are easily revoked). So my usage pattern of this library so far is 1. get temporary credentials on my laptop, then 2. copy them over to another machine and work. This works but is annoying.

It would be amazing to use a unified access pattern regardless of machine where we directly supply a token (or read it from the environment variable) earthaccess.login() #"using EDL_TOKEN". Happy to open a PR with the understanding that this might not work for all scenarios right now.

[betolink]

@scottyhq first thanks for reporting the bug! Yes we definitely can add user-provided tokens with the upcoming fixes, the issue like you mentioned is that some DAACs and services require the EDL session cookie, for which we need the user's credentials e.g. the /S3Credentials endpoint. I know there is an ongoing migration to only support EDL bearer tokens everywhere, eventually that would be "the way".