Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s3_bucket transfer accleration #2046

Merged

Conversation

abraverm
Copy link
Contributor

@abraverm abraverm commented Apr 9, 2024

Copy link

github-actions bot commented Apr 9, 2024

Docs Build 📝

Thank you for contribution!✨

This PR has been merged and your docs changes will be incorporated when they are next published.

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/ab7140304faf441495fbf7b7dd4ffdc8

✔️ ansible-galaxy-importer SUCCESS in 4m 38s
✔️ build-ansible-collection SUCCESS in 14m 13s
✔️ ansible-test-splitter SUCCESS in 5m 25s
integration-amazon.aws-1 FAILURE in 7m 38s
Skipped 43 jobs

@abraverm
Copy link
Contributor Author

abraverm commented Apr 9, 2024

Integation test failed on "AccessDenied", I have verified the integration test with my own AWS account with S3 admin user, so I have no idea what permissions are missing or how to change them for the CI to pass.

@abraverm
Copy link
Contributor Author

ping

@alinabuzachis
Copy link
Collaborator

alinabuzachis commented Apr 23, 2024

@abraverm Thank you for taking time to work on this feature idea. Can you please open a pull request and add the missing permission PutBucketAccelerateConfiguration to this file https://github.com/mattclay/aws-terminator/blob/main/aws/policy/storage-services.yaml. Thank you.

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/b2ae2c59bae74c6fa0a654b97d7a78f8

✔️ ansible-galaxy-importer SUCCESS in 5m 04s
✔️ build-ansible-collection SUCCESS in 15m 46s
✔️ ansible-test-splitter SUCCESS in 6m 40s
integration-amazon.aws-1 FAILURE in 6m 53s
Skipped 43 jobs

@abraverm
Copy link
Contributor Author

I have updated the PR to match the recent changes but CI is still blocked by ansible/aws-ci-admin#5

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/90810f6758364fcaa3de95905aa4b004

✔️ ansible-galaxy-importer SUCCESS in 5m 39s
✔️ build-ansible-collection SUCCESS in 15m 58s
✔️ ansible-test-splitter SUCCESS in 6m 21s
integration-amazon.aws-1 FAILURE in 9m 09s
Skipped 43 jobs

@gravesm
Copy link
Member

gravesm commented May 8, 2024

recheck

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/1b5e85ba3e1647d1b4d9ede80f069b37

✔️ ansible-galaxy-importer SUCCESS in 5m 12s
✔️ build-ansible-collection SUCCESS in 16m 45s
✔️ ansible-test-splitter SUCCESS in 7m 45s
✔️ integration-amazon.aws-1 SUCCESS in 10m 12s
Skipped 43 jobs

Copy link
Contributor

@hakbailey hakbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good! See a few minor comments inline.

plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
@abraverm
Copy link
Contributor Author

@hakbailey thank you for the review, I have addressed your comments in my last commit

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/d3db87fd2d8a401bb1da743bb1134220

✔️ ansible-galaxy-importer SUCCESS in 6m 13s
✔️ build-ansible-collection SUCCESS in 15m 47s
✔️ ansible-test-splitter SUCCESS in 6m 25s
integration-amazon.aws-1 RETRY_LIMIT in 7m 06s
Skipped 43 jobs

@abraverm
Copy link
Contributor Author

I have no clue why the CI failed, the integration tests passed locally

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/7a99e3c3826646d794d8f6fd64fbfff8

✔️ ansible-galaxy-importer SUCCESS in 9m 02s
✔️ build-ansible-collection SUCCESS in 24m 48s
ansible-test-splitter RETRY_LIMIT in 5m 26s
⚠️ integration-amazon.aws-1 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-2 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-3 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-4 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-5 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-6 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-7 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-8 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-9 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-10 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-11 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-12 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-13 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-14 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-15 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-16 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-17 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-18 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-19 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-20 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-21 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-22 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-1 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-2 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-3 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-4 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-5 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-6 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-7 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-8 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-9 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-10 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-11 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-12 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-13 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-14 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-15 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-16 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-17 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-18 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-19 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-20 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-21 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-22 SKIPPED Skipped due to failed job ansible-test-splitter

@abraverm
Copy link
Contributor Author

abraverm commented Jun 6, 2024

recheck

Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/7511cf92171e4bc9961458c3b27ccb44

✔️ ansible-galaxy-importer SUCCESS in 5m 46s
✔️ build-ansible-collection SUCCESS in 16m 27s
ansible-test-splitter POST_FAILURE in 7m 39s
⚠️ integration-amazon.aws-1 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-2 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-3 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-4 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-5 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-6 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-7 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-8 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-9 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-10 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-11 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-12 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-13 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-14 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-15 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-16 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-17 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-18 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-19 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-20 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-21 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-amazon.aws-22 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-1 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-2 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-3 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-4 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-5 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-6 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-7 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-8 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-9 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-10 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-11 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-12 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-13 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-14 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-15 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-16 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-17 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-18 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-19 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-20 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-21 SKIPPED Skipped due to failed job ansible-test-splitter
⚠️ integration-community.aws-22 SKIPPED Skipped due to failed job ansible-test-splitter

@abraverm
Copy link
Contributor Author

abraverm commented Jun 6, 2024

recheck

Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/5ef36deb1322438eb3fa1d65298445cc

✔️ ansible-galaxy-importer SUCCESS in 5m 30s
✔️ build-ansible-collection SUCCESS in 15m 19s
✔️ ansible-test-splitter SUCCESS in 7m 09s
✔️ integration-amazon.aws-1 SUCCESS in 8m 17s
Skipped 43 jobs

@hakbailey hakbailey added this to the 8.1.0 milestone Jun 26, 2024
Copy link
Contributor

@hakbailey hakbailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more minor comments. This will also need to be rebased onto main to fix the conflicts. Thanks!

plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
plugins/modules/s3_bucket.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@alinabuzachis alinabuzachis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abraverm Thank you for working on this feature idea. Can you also please add a changelog fragment (https://docs.ansible.com/ansible/latest/community/development_process.html#creating-a-changelog-fragment) fro this change?

@alinabuzachis alinabuzachis added the backport-8 PR should be backported to the stable-8 branch label Jun 28, 2024
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/d3e9be038fe14af6874192497901d0d6

✔️ ansible-galaxy-importer SUCCESS in 4m 34s
✔️ build-ansible-collection SUCCESS in 12m 19s
✔️ ansible-test-splitter SUCCESS in 5m 57s
✔️ integration-amazon.aws-1 SUCCESS in 6m 54s
Skipped 43 jobs

Signed-off-by: Alina Buzachis <[email protected]>
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/5b1deee03e4f4e05926f1db8f8bf6024

✔️ ansible-galaxy-importer SUCCESS in 5m 18s
✔️ build-ansible-collection SUCCESS in 12m 45s
✔️ ansible-test-splitter SUCCESS in 5m 04s
✔️ integration-amazon.aws-1 SUCCESS in 6m 49s
Skipped 43 jobs

@alinabuzachis alinabuzachis added the mergeit Merge the PR (SoftwareFactory) label Jun 28, 2024
Copy link
Contributor

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/29c4168d3b094ea08068d8721cb4ca0c

✔️ ansible-galaxy-importer SUCCESS in 4m 45s
✔️ build-ansible-collection SUCCESS in 13m 07s
✔️ ansible-test-splitter SUCCESS in 5m 10s
✔️ integration-amazon.aws-1 SUCCESS in 6m 56s
Skipped 43 jobs

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 3a6e448 into ansible-collections:main Jun 28, 2024
36 checks passed
Copy link

patchback bot commented Jun 28, 2024

Backport to stable-8: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 3a6e448 on top of patchback/backports/stable-8/3a6e448a93a24c474d281aa49492a3903b662ef7/pr-2046

Backporting merged PR #2046 into main

  1. Ensure you have a local repo clone of your fork. Unless you cloned it
    from the upstream, this would be your origin remote.
  2. Make sure you have an upstream repo added as a remote too. In these
    instructions you'll refer to it by the name upstream. If you don't
    have it, here's how you can add it:
    $ git remote add upstream https://github.com/ansible-collections/amazon.aws.git
  3. Ensure you have the latest copy of upstream and prepare a branch
    that will hold the backported code:
    $ git fetch upstream
    $ git checkout -b patchback/backports/stable-8/3a6e448a93a24c474d281aa49492a3903b662ef7/pr-2046 upstream/stable-8
  4. Now, cherry-pick PR s3_bucket transfer accleration #2046 contents into that branch:
    $ git cherry-pick -x 3a6e448a93a24c474d281aa49492a3903b662ef7
    If it'll yell at you with something like fatal: Commit 3a6e448a93a24c474d281aa49492a3903b662ef7 is a merge but no -m option was given., add -m 1 as follows instead:
    $ git cherry-pick -m1 -x 3a6e448a93a24c474d281aa49492a3903b662ef7
  5. At this point, you'll probably encounter some merge conflicts. You must
    resolve them in to preserve the patch from PR s3_bucket transfer accleration #2046 as close to the
    original as possible.
  6. Push this branch to your fork on GitHub:
    $ git push origin patchback/backports/stable-8/3a6e448a93a24c474d281aa49492a3903b662ef7/pr-2046
  7. Create a PR, ensure that the CI is green. If it's not — update it so that
    the tests and any other checks pass. This is it!
    Now relax and wait for the maintainers to process your pull request
    when they have some cycles to do reviews. Don't worry — they'll tell you if
    any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
alinabuzachis pushed a commit to alinabuzachis/amazon.aws that referenced this pull request Jun 28, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves <[email protected]>
(cherry picked from commit 3a6e448)
@abraverm
Copy link
Contributor Author

Thank you for merging the PR and I'm sorry for the delay on my part to respond.

softwarefactory-project-zuul bot pushed a commit that referenced this pull request Jul 1, 2024
Add support for S3 Bucket Transfer Acceleration:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration.html
Reviewed-by: Helen Bailey [email protected]
Reviewed-by: Alina Buzachis
Reviewed-by: Mike Graves [email protected]
(cherry picked from commit 3a6e448)
SUMMARY


ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Helen Bailey <[email protected]>
Reviewed-by: Mandar Kulkarni <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8 PR should be backported to the stable-8 branch mergeit Merge the PR (SoftwareFactory)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants