Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document exactly why curl does not work by default when called from inside us-west-2 with data in AWS #187

Closed
yuvipanda opened this issue Dec 13, 2022 · 2 comments

Comments

@yuvipanda
Copy link

If you make a request from inside us-west-2, the OAuth2 login process for earthdata login + redirect is a bit like this:

Original URL -> Earthdata Login OAuth2 -> Original URL (with a cookie for auth) -> Cloudfront -> Signed S3 URL.

The issue here is that some clients - notably curl, and probably aiohttp - do something that means S3 will reject the request with a 403 forbidden, and not much of an explanatory message. Information from Joseph H Kennedy and Luis (and others) is that the signed S3 URL is being rejected by AWS because it has additional authentication information (in the form of a cookie or headers) it does not understand.

So the issue is that the eventual signed s3 url somtimes gets two bits of authentication:

  1. The signed S3 bits, part of the query string
  2. The earthdata cookie (maybe? Or is it a header? But cookies are also headers soooo)

And S3 rejects this. This isn't manifest in all http clients, it is in some. We should use curl as a reference, and use it.

So we should figure out and document a sequence of curl commands, that when executed outside us-west-2, produce the data - and when executed inside us-west-2, do not produce the data. We can then go from here.

@yuvipanda
Copy link
Author

The end goal here is to make sure that end users can start working on data hosted on AWS without needing to hear the words 'us-west-2' or learn about the internals how how the data is stored. This will have a performance ceiling (due to not using the s3 protocol), but that's alright for a start.

@yuvipanda
Copy link
Author

Closing in favor of #188, which has a ton of relevant info

@betolink betolink mentioned this issue Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant