Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SUP-1372 fix compressed #91

Merged
merged 14 commits into from
Sep 14, 2023
Merged

SUP-1372 fix compressed #91

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c3b1a0d
Added check for file/directory to compressed
HugeIRL Sep 5, 2023
ec91690
Updated tests to create files and remove them for testing compressed
HugeIRL Sep 5, 2023
23f438a
Updated README.md version numbers
HugeIRL Sep 5, 2023
4b4d2e0
Moved compression path check to a different location
HugeIRL Sep 6, 2023
7b8cf9e
Fixed tests and added a new path existence test for compression
HugeIRL Sep 6, 2023
b46c1a4
Updated compression check to fail gracefully and be more detailed
HugeIRL Sep 6, 2023
2969b4d
Updated compression test
HugeIRL Sep 6, 2023
370b6bd
Update compression path check to use -n
HugeIRL Sep 6, 2023
fbee13c
Removed compression expression wrap
HugeIRL Sep 7, 2023
3447a39
Updated compression test, add new compression ignore-missing test
HugeIRL Sep 7, 2023
6b257d2
Removed mocked glob pathing in upload compressed tests
HugeIRL Sep 7, 2023
ea94cb1
Add clarification when using compressed with ignore-missing
pzeballos Sep 13, 2023
13f3aa1
Update tests title
pzeballos Sep 13, 2023
3359ae1
Update tests title
pzeballos Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This functionality duplicates the [artifact_paths](https://buildkite.com/docs/pi
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "log/**/*.log"
```

Expand All @@ -20,7 +20,7 @@ You can specify multiple files/globs to upload as artifacts:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: [ "log/**/*.log", "debug/*.error" ]
```

Expand All @@ -30,7 +30,7 @@ And even rename them before uploading them (can not use globs here though, sorry
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload:
- from: log1.log
to: log2.log
Expand All @@ -47,7 +47,7 @@ eg: uploading a public file when using S3
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "coverage-report/**/*"
s3-upload-acl: public-read
```
Expand All @@ -57,7 +57,7 @@ eg: uploading a private file when using GS
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "coverage-report/**/*"
gs-upload-acl: private
```
Expand All @@ -70,7 +70,7 @@ This downloads artifacts matching globs to the local filesystem. See [downloadin
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
download: "log/**/*.log"
```

Expand All @@ -80,7 +80,7 @@ You can specify multiple files/patterns:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
download: [ "log/**/*.log", "debug/*.error" ]
```

Expand All @@ -90,7 +90,7 @@ Rename particular files after downloading them:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
download:
- from: log1.log
to: log2.log
Expand All @@ -102,7 +102,7 @@ And even do so from different builds/steps:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
step: UUID-DEFAULT
build: UUID-DEFAULT-2
download:
Expand Down Expand Up @@ -145,7 +145,7 @@ When uploading, the file or directory specified in the `upload` option will be c
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "log/my-folder"
compressed: logs.zip
```
Expand All @@ -156,14 +156,14 @@ When downloading, this option states the actual name of the artifact to be downl
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
download: "log/file.log"
compressed: logs.tgz
```

### `ignore-missing` (optional, boolean)

If set to `true`, it will ignore errors caused when calling `buildkite-agent artifact` to prevent failures if you expect artifacts not to be present in some situations.
If set to `true`, it will ignore errors caused when calling `buildkite-agent artifact` to prevent failures if you expect artifacts not to be present in some situations. When using the `compressed` property, it will ignore compressing the artifacts that are not present.

### `skip-on-status` (optional, integer or array of integers, uploads only)

Expand All @@ -177,7 +177,7 @@ Skip uploading if the main command failed with exit code 147:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "log/*.log"
skip-on-status: 147
```
Expand All @@ -188,7 +188,7 @@ Alternatively, skip artifact uploading on exit codes 1 and 5:
steps:
- command: ...
plugins:
- artifacts#v1.9.0:
- artifacts#v1.9.1:
upload: "log/*.log"
skip-on-status:
- 1
Expand Down
8 changes: 6 additions & 2 deletions hooks/post-command
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,12 @@ if [[ "${SINGULAR_UPLOAD_OBJECT}" == "true" ]]; then

if [[ "${COMPRESSED}" == "true" ]]; then
echo "~~~ Compressing ${path} to ${BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED}"
"${compress[@]}" "${path}"
path=" ${BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED}"
if [[ ! -e "${path}" ]]; then
echo "+++ 🚨 Unable to compress artifact, '${path}' may not exist or is an empty directory"
else
"${compress[@]}" "${path}"
path=" ${BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED}"
fi
fi

echo "~~~ Uploading artifacts ${EXTRA_MESSAGE}"
Expand Down
86 changes: 75 additions & 11 deletions tests/upload-compressed.bats
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,34 @@ load "${BATS_PLUGIN_PATH}/load.bash"
# Uncomment to enable stub debug output:
# export BUILDKITE_AGENT_STUB_DEBUG=/dev/tty

@test "Compression errors silently when there is nothing to compress" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="non_existent_file.txt"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"

run "$PWD/hooks/post-command"

assert_failure
assert_output --partial "Unable to compress artifact, 'non_existent_file.txt' may not exist or is an empty directory"

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED
}

@test "Compression errors silently when there is nothing to compress and ignore-missing is set to true" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="non_existent_file.txt"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"
export BUILDKITE_PLUGIN_ARTIFACTS_IGNORE_MISSING="true"

run "$PWD/hooks/post-command"

assert_success
assert_output --partial "Unable to compress artifact, 'non_existent_file.txt' may not exist or is an empty directory"
assert_output --partial "Ignoring error in upload of"

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED
}

@test "Invalid compressed format" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="file.log"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.rar"
Expand All @@ -20,9 +48,11 @@ load "${BATS_PLUGIN_PATH}/load.bash"
}

@test "Single value zip" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="*.log"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="file.log"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"

touch "file.log"

stub buildkite-agent \
"artifact upload \* : echo uploaded \$3"

Expand All @@ -32,7 +62,7 @@ load "${BATS_PLUGIN_PATH}/load.bash"
run "$PWD/hooks/post-command"

assert_success
assert_output --partial "Compressing *.log to file.zip"
assert_output --partial "Compressing file.log to file.zip"
assert_output --partial "Uploading artifacts"
assert_output --partial "uploaded file.zip"

Expand All @@ -41,12 +71,16 @@ load "${BATS_PLUGIN_PATH}/load.bash"

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm "file.log"
}

@test "Single value tgz" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="*.log"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="file.log"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.tgz"

touch "file.log"

stub buildkite-agent \
"artifact upload \* : echo uploaded \$3"

Expand All @@ -56,7 +90,7 @@ load "${BATS_PLUGIN_PATH}/load.bash"
run "$PWD/hooks/post-command"

assert_success
assert_output --partial "Compressing *.log to file.tgz"
assert_output --partial "Compressing file.log to file.tgz"
assert_output --partial "Uploading artifacts"
assert_output --partial "uploaded file.tgz"

Expand All @@ -65,6 +99,8 @@ load "${BATS_PLUGIN_PATH}/load.bash"

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm "file.log"
}

@test "Single file zip with relocation" {
Expand Down Expand Up @@ -138,10 +174,12 @@ load "${BATS_PLUGIN_PATH}/load.bash"
}

@test "Single value zip with job" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="*.log"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="file.log"
export BUILDKITE_PLUGIN_ARTIFACTS_JOB="12345"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"

touch "file.log"

stub buildkite-agent \
"artifact upload --job \* \* : echo uploaded \$5 with --job \$4"

Expand All @@ -152,23 +190,27 @@ load "${BATS_PLUGIN_PATH}/load.bash"

assert_success
assert_output --partial "Uploading artifacts (extra args: '--job 12345')"
assert_output --partial "Compressing *.log to file.zip"
assert_output --partial "Compressing file.log to file.zip"
assert_output --partial "uploaded file.zip"
refute_output --partial "uploaded *.log"
refute_output --partial "uploaded file.log"

unstub buildkite-agent
unstub zip

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_JOB
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm "file.log"
}

@test "Single value tgz with job" {
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="*.log"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD="file.log"
export BUILDKITE_PLUGIN_ARTIFACTS_JOB="12345"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.tgz"

touch "file.log"

stub buildkite-agent \
"artifact upload --job \* \* : echo uploaded \$5 with --job \$4"

Expand All @@ -179,16 +221,18 @@ load "${BATS_PLUGIN_PATH}/load.bash"

assert_success
assert_output --partial "Uploading artifacts (extra args: '--job 12345')"
assert_output --partial "Compressing *.log to file.tgz"
assert_output --partial "Compressing file.log to file.tgz"
assert_output --partial "uploaded file.tgz"
refute_output --partial "uploaded *.log"
refute_output --partial "uploaded file.log"

unstub buildkite-agent
unstub tar

unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD
unset BUILDKITE_PLUGIN_ARTIFACTS_JOB
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm "file.log"
}

@test "Multiple artifacts zip" {
Expand All @@ -197,6 +241,10 @@ load "${BATS_PLUGIN_PATH}/load.bash"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_2="baz.log"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"

touch /tmp/foo.log
touch bar.log
touch baz.log

stub buildkite-agent \
"artifact upload \* : echo uploaded \$3"

Expand All @@ -220,6 +268,10 @@ load "${BATS_PLUGIN_PATH}/load.bash"
unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_1
unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_2
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm /tmp/foo.log
rm bar.log
rm baz.log
}

@test "Multiple artifacts tgz" {
Expand All @@ -228,6 +280,10 @@ load "${BATS_PLUGIN_PATH}/load.bash"
export BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_2="baz.log"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.tgz"

touch /tmp/foo.log
touch bar.log
touch baz.log

stub buildkite-agent \
"artifact upload \* : echo uploaded \$3"

Expand All @@ -251,6 +307,10 @@ load "${BATS_PLUGIN_PATH}/load.bash"
unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_1
unset BUILDKITE_PLUGIN_ARTIFACTS_UPLOAD_2
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED

rm /tmp/foo.log
rm bar.log
rm baz.log
}

@test "Multiple artifacs zip some relocation" {
Expand All @@ -261,6 +321,8 @@ load "${BATS_PLUGIN_PATH}/load.bash"
export BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED="file.zip"

touch /tmp/foo.log
touch bar.log
touch baz.log

stub buildkite-agent \
"artifact upload \* : echo uploaded \$3"
Expand All @@ -282,6 +344,8 @@ load "${BATS_PLUGIN_PATH}/load.bash"
assert [ -e /tmp/foo2.log ]
assert [ ! -e /tmp/foo.log ]
rm /tmp/foo2.log
rm bar.log
rm baz.log

unstub buildkite-agent
unstub zip
Expand Down Expand Up @@ -365,4 +429,4 @@ load "${BATS_PLUGIN_PATH}/load.bash"
unset BUILDKITE_PLUGIN_ARTIFACTS_DOWNLOAD_0_FROM
unset BUILDKITE_PLUGIN_ARTIFACTS_DOWNLOAD_0_TO
unset BUILDKITE_PLUGIN_ARTIFACTS_COMPRESSED
}
}