[BUG] Sync Continues, Despite Failure to List S3 Bucket #564
Assignees
Milestone
Comments
|
PR - #565 First attempt at writing golang, feedback welcome! |
ilkinulas
added a commit
that referenced
this issue
Aug 4, 2023
Resolves #564 Changes are made: - Added `exit-on-error` flag. Its value is `false` by default. - Added `shouldStopSync` function. It determines whether a sync process should be stopped or not. It does not ignore the errors `AccessDenied` and `NoSuchBucket` regardless of the value of `exit-on-error` flag. - `sync` command stops if an error is received when listing objects from source or destination when the `exit-on-error` flag is `true`. But it always ignores the `ErrNoObjectFound` error. --------- Co-authored-by: İlkin Balkanay <[email protected]> Co-authored-by: Onur Sönmez <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello!
First, I'd like to say thank you for providing an amazing tool,
s5cmdhas been invaluable for a migration project I'm doing at $work; I have achieved amazing performance improvements over standardaws s3 syncon a very large bucket.On Friday I hit a very interesting edge case and would like to open a bug report, I have a PR ready which I'll link to the issue which I think will fix it.
Description:
Last week when running
s5cmd syncI messed up the permissions on the target bucket such that it was possible to write objects but not actually list the bucket. Rather than failing the sync simply continued assuming target bucket was empty. This led to a full sync of 250TiB of data from source to destination, as I have versioning enabled on target bucket these copied objects were treated as additional versions and I had a scary spike in my costs as my target bucket went from 250TiB to 500TiB... ouch!Steps to reproduce:
v2.1.0-beta.1)s3:GetObject, s3:ListBucketon source bucket ands3:PutObject, s3:DeleteObjecton destination bucket.s5cmd sync s3://<source>/* s3://<destination>/Expected result:
The
s5cmd syncshould exit with failure and provide an error message, without ability to successfully list source and destination buckets it's not possible to meet the behaviour of the sync command as provided in the documentation. In addition by not "failing fast" in the sync stage it can lead to an explosion of errors in the cp stage, if for instance you sync to a target bucket that doesn't exist, rather than failing during sync, you'll just get per-object failures as each cp fails with target bucket doesn't exist.Actual result:
s5cmd syncruns and exits with success after failing to list target bucket, the target bucket is treated as empty irrespective of the actual contents.The text was updated successfully, but these errors were encountered: