Add scratch bucket functionality for AWS

[yuvipanda]

• Bump up AWS provider version, as there had been a few deprecations
  in the IAM resources
• Mimic the GCP setup as much as possible
• Write some docs on how to enable this for S3
• Setup IRSA with Terraform so users can be granted IAM roles
• Use the uwhackweeks to test

Ref #1309

[yuvipanda]

This is good to go I think!

[yuvipanda]

@scottyhq can you test out the SCRATCH_BUCKET stuff on the uwhackweeks hub as well? This will make sure that when we deploy it for snowex it'll be all good.

[github-actions]

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/workflows/deploy-hubs.yaml?query=branch%3Amaster

[scottyhq]

can you test out the SCRATCH_BUCKET stuff on the uwhackweeks hub as well?

Thanks! Just did a quick test with AWS CLI and things seem to work well! We'll see if it's gone in a week :)

aws s3 cp scratch-bucket-test.txt $SCRATCH_BUCKET/ 
upload: ./scratch-bucket-test.txt to s3://uwhackweeks-scratch/scottyhq/scratch-bucket-test.txt

[scottyhq]

By the way, i've found for hackweeks people often want to get a bunch of local data (laptop, labgroup computer) into the cloud and the scratch bucket is a pretty great way to kick the tires.

A trick i figured out for last year's event is getting temporary session token from the hub: (copied from snowex-hackweek/jupyterhub#9 (comment))

aws sts assume-role-with-web-identity \
 --role-arn $AWS_ROLE_ARN \
 --role-session-name $JUPYTERHUB_CLIENT_ID \
 --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
 --duration-seconds 1000 > /tmp/irp-cred.txt

export AWS_REGION="us-west-2"
export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")"
export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")"
export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")"
rm /tmp/irp-cred.txt

Do you think it's worth documenting this in https://docs.2i2c.org/en/latest/user/storage.html?highlight=scratch#scratch-bucket ? It'd be pretty cool if there were a file menu to 'get session token' instead of a hacky script, do you know of something like this ?

[damianavila]

@yuvipanda, do you have any thoughts about this one? I presume the permissions you get with these temporary credentials only give you access to the scratch bucket on S3 and anything else, am I correct?

[scottyhq]

It'd be pretty cool if there were a file menu to 'get session token' instead of a hacky script, do you know of something like this ?

@yuvipanda and I (mostly Yuvi :) ) put together this proof of concept jupyter server extension to give Hub users temporary credentials via an API endpoint. It's still a hacky script behind the scenes but works! I probably won't work on it much in the near future but wanted to link here in case people want to develop it further and try out: https://github.com/scottyhq/jupyter-cloud-scoped-creds