Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

command/sync: handle object listing errors #597

Merged
merged 30 commits into from
Aug 4, 2023
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
da7d6fa
command/sync: handle object listing errors
ahmethakanbesel Jul 18, 2023
4589b09
command/sync: fix typo in the comments
ahmethakanbesel Jul 18, 2023
899a195
command/sync: fix typo in the comments
ahmethakanbesel Jul 18, 2023
9482b08
command/sync: add --exit-on-error flag
ahmethakanbesel Jul 18, 2023
ea069b4
command/sync: better error handling
ahmethakanbesel Jul 18, 2023
2bf614d
command/sync: add new tests
ahmethakanbesel Jul 18, 2023
219c8a6
command/sync: update not found bucket test
ahmethakanbesel Jul 19, 2023
b89f6d3
command/sync: update tests
ahmethakanbesel Jul 19, 2023
d3aea4e
command/sync: refactor getSourceAndDestinationObjects
ahmethakanbesel Jul 20, 2023
127ff9b
command/run: check ctx cancelled
ahmethakanbesel Jul 21, 2023
c6318bb
Merge branch 'peak:master' into sync-handle-errors
ahmethakanbesel Jul 21, 2023
15b3d58
command/sync: revert refactoring
ahmethakanbesel Jul 21, 2023
10ee234
command/run: revert a change
ahmethakanbesel Jul 24, 2023
c85c3ba
command/sync: add a test case
ahmethakanbesel Jul 24, 2023
8230b85
changelog: add a bug fix
ahmethakanbesel Jul 24, 2023
17b199c
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 24, 2023
68f2b4a
changelog: fix typo
ahmethakanbesel Jul 24, 2023
c3fdf8b
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 25, 2023
966ce06
command/sync: add fallthrough
ahmethakanbesel Jul 25, 2023
723fcf3
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 25, 2023
209ae93
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 27, 2023
11c1435
changelog: update the entry
ahmethakanbesel Jul 27, 2023
427ffb4
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 27, 2023
ff52dea
Merge branch 'master' into sync-handle-errors
ilkinulas Jul 27, 2023
06619ef
command/sync: refactor `shouldStopSync`
ahmethakanbesel Jul 27, 2023
6f5e1ff
Merge branch 'master' into sync-handle-errors
ahmethakanbesel Jul 28, 2023
44261f2
command/sync: revert the ctx
ahmethakanbesel Jul 28, 2023
96fd166
command/run: set reader err if ctx is cancelled
ahmethakanbesel Jul 28, 2023
1189724
command/run: remove unnecessary `return`
ahmethakanbesel Jul 28, 2023
7b79b43
Merge branch 'master' into sync-handle-errors
sonmezonur Aug 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
command/sync: add a test case
  • Loading branch information
ahmethakanbesel committed Jul 24, 2023
commit c85c3ba764fadcf5acb68d207aa1f37693403033
41 changes: 39 additions & 2 deletions e2e/sync_test.go
Original file line number Diff line number Diff line change
@@ -1847,8 +1847,8 @@ func TestSyncS3BucketToEmptyS3BucketWithExitOnErrorFlag(t *testing.T) {
}
}

// sync s3://bucket/* s3://NotExistingBucket/ (destbucket doesn't exist)
func TestSyncS3BucketToS3BucketThatDoesNotExist(t *testing.T) {
// sync --exit-on-error s3://bucket/* s3://NotExistingBucket/ (dest bucket doesn't exist)
func TestSyncExitOnErrorS3BucketToS3BucketThatDoesNotExist(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test case does not adequately test the --exit-on-error flag, because if the bucket does not exist, the flag will have no effect. It would be better to test the flag with an error other than "AccessDenied" or "NoSuchBucket".

t.Parallel()

now := time.Now()
@@ -1883,3 +1883,40 @@ func TestSyncS3BucketToS3BucketThatDoesNotExist(t *testing.T) {
0: contains(`status code: 404`),
})
}

// sync s3://bucket/* s3://NotExistingBucket/ (dest bucket doesn't exist)
func TestSyncS3BucketToS3BucketThatDoesNotExist(t *testing.T) {
t.Parallel()

now := time.Now()
timeSource := newFixedTimeSource(now)
s3client, s5cmd := setup(t, withTimeSource(timeSource))

bucket := s3BucketFromTestName(t)
destbucket := "NotExistingBucket"

createBucket(t, s3client, bucket)

S3Content := map[string]string{
"testfile.txt": "S: this is a test file",
"readme.md": "S: this is a readme file",
"a/another_test_file.txt": "S: yet another txt file",
"abc/def/test.py": "S: file in nested folders",
}

for filename, content := range S3Content {
putFile(t, s3client, bucket, filename, content)
}

src := fmt.Sprintf("s3://%v/*", bucket)
dst := fmt.Sprintf("s3://%v/", destbucket)

cmd := s5cmd("sync", src, dst)
result := icmd.RunCmd(cmd)

result.Assert(t, icmd.Expected{ExitCode: 1})

assertLines(t, result.Stderr(), map[int]compareFunc{
0: contains(`status code: 404`),
})
}