Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure s3 in the containerized environment for file upload e2e tests #457

Merged
merged 10 commits into from
Aug 21, 2024

Conversation

GPortas
Copy link
Contributor

@GPortas GPortas commented Aug 13, 2024

What this PR does / why we need it:

Adds S3 remote storage option using a bucket hosted on AWS.

Initially, I evaluated using localstack as in js-dataverse to avoid using AWS and to keep everything containerized. After encountering issues and limitations, I decided to use a remote S3 storage.

Which issue(s) this PR closes:

Special notes for your reviewer:

Now we need to create a .env under dev-env folder, by copying the .env.example file, and paste there the credentials for AWS. I will be happy to share them, just ping me on Slack. @ekraffmiller @ErykKul @GermanSaracca

The AWS secrets are already added to the repository for the GitHub actions to operate.

Suggestions on how to test this:

Run docker containers locally and create a dataset. Next, verify that direct upload works on JSF.

@coveralls
Copy link

coveralls commented Aug 13, 2024

Coverage Status

coverage: 97.733% (-0.02%) from 97.754%
when pulling 0ca4a1f on 456-localstack-s3-env
into 5832262 on develop.

@GPortas GPortas changed the title Configure localstack s3 in the containerized environment for file upload e2e tests Configure s3 in the containerized environment for file upload e2e tests Aug 15, 2024
@GPortas GPortas marked this pull request as ready for review August 15, 2024 12:38
@GPortas GPortas added SPA Size: 3 A percentage of a sprint. 2.1 hours. labels Aug 15, 2024
@g-saracca g-saracca self-assigned this Aug 15, 2024
g-saracca
g-saracca previously approved these changes Aug 15, 2024
Copy link
Contributor

@g-saracca g-saracca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed the steps and was able to successfully create a collection with s3 configured as Storage, and then create a dataset and upload a file to it.
Screenshot 2024-08-15 at 14 51 13
Screenshot 2024-08-15 at 14 47 58

@g-saracca g-saracca removed their assignment Aug 15, 2024
@GPortas
Copy link
Contributor Author

GPortas commented Aug 16, 2024

@GermanSaracca @ErykKul @ekraffmiller

Thanks to Eryk's help, I've managed to set S3 as the default and unique storage option, so there's no need for the intermediate step of creating a dataverse with S3 storage. 🎉

Screenshot 2024-08-16 at 11 50 31

@GPortas GPortas requested a review from g-saracca August 16, 2024 10:53
g-saracca
g-saracca previously approved these changes Aug 16, 2024
Copy link
Contributor

@g-saracca g-saracca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GPortas @ErykKul Works great, S3 is now the default Storage. I was able to create datasets and upload files into them.

@g-saracca g-saracca merged commit 0933f3f into develop Aug 21, 2024
14 checks passed
@GPortas GPortas added SPA.Q3.5 Enhanced File upload Original size: 3 labels Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Original size: 3 Size: 3 A percentage of a sprint. 2.1 hours. SPA.Q3.5 Enhanced File upload SPA
Projects
Status: Done 🧹
Development

Successfully merging this pull request may close these issues.

Configure localstack s3 in the containerized environment for file upload e2e tests
3 participants