-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to write to scratch and persistent buckets on a hub? #3639
Comments
@sgibson91, found this #1322 (comment) that looks relevant and wanted to read on it. |
Also, I'm in the same boat on the above ⬆️ |
I dunno if this helps but in reference to the above, I tried the following in the Showcase Hub and got: (notebook) jovyan@jupyter-jnywong:~$ aws sts assume-role-with-web-identity \
--role-arn $AWS_ROLE_ARN \
--role-session-name $JUPYTERHUB_CLIENT_ID \
--web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity |
@yuvipanda or @consideRatio, do you have any further thoughts about this one? |
Issue 1 - our docs isn't explicit enough (now tracked in #3665)I think our working with object storage docs should explicitly distinguish the following scenarios:
Issue 2 - access to showcase hub's scratch bucketI've attempted to use the code snippet from our docs on scratch buckets and also got an error. import os
import xarray as xr
SCRATCH_BUCKET = os.environ['SCRATCH_BUCKET']
ds = xr.tutorial.open_dataset("rasm") # load example data
ds.to_zarr(f'{SCRATCH_BUCKET}/rasm.zarr') # write data Using the more direct test of Like @sgibson91 and @GeorgianaElena, my expectation is that permissions to access the scratch bucket should be setup automatically. Debugging issue 2So how do I get Explicit understandingMy understanding is that for bucket permissions to be provided to the user servers' users by:
What fails?
|
Thanks for the update @consideRatio ! I will definitely take the lessons learned from this to update the Service Docs accordingly. |
Thank you @jnywong!!! I opened #3663 to extract "Issue 1" from this issue that I closed as part of verifying that bucket access was once again functioning. |
Context
I was asked via support to set up scratch and persistent buckets for the showcase hub, and I did so in:
There was some confusion in the ticket because Jenny was expecting me to send her a
CLIENT_KEY
andCLIENT_SECRET
according to https://docs.2i2c.org/user/topics/data/cloud/, but following https://infrastructure.2i2c.org/howto/features/buckets/ I found no mention of keys and secrets, and nothing I did generated any.As far as I am aware, adding the bucket names to the
bucket_admin_access
section of thehub_cloud_permissions
variable in terraform grants read/write permission to the bucket in a role that is passed to the hub via a kubernetes annotation. Every user server gets this annotation and reading/writing the bucket should "just work". However, Jenny has reported that she is unable to write to the bucket:What is going on here? I don't actually know how researchers actually access/write to buckets and assumed this "just worked" since no one else has had issues, and I assumed I had done everything I needed to from the terraform/hub side of things. Can @2i2c-org/engineering provide any clarity?
Proposal
No response
Updates and actions
No response
The text was updated successfully, but these errors were encountered: