Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

specify file which will be pruned by wildcard, and skip checking exis… #34

Merged
merged 1 commit into from
Mar 14, 2022

Conversation

ryu-sato
Copy link
Contributor

@ryu-sato ryu-sato commented Mar 9, 2022

…tence of backup file before pruning

This PR enable to prune backuped files in S3 and GCS.

CI does not work but locally test/s3/e2e.sh succeeds.

changes

  • specify file which will be pruned by wildcard
  • skip checking existence of backup file before pruning
    • because aws s3 ls cannot support wildcard

local check

DELETE_DEVIDE=3 DELETE_TARGET_DAYS_LEFT=0 AWSCLI_ENDPOINT_OPT="http://192.168.32.3/" TARGET_BUCKET_URL=s3://test/ sh /opt/bin/entrypoint.sh prune
+ BACKUPFILE_PREFIX=backup
+ DELETE_DEVIDE=3
+ DELETE_TARGET_DAYS_LEFT=0
++ /usr/bin/dirname ./prune.sh
+ CWD=.
+ cd .
+ . ./functions.sh
++ AWSCLI=/usr/bin/aws
++ AWSCLI_COPY_OPT='s3 cp'
++ AWSCLI_LIST_OPT='s3 ls'
++ AWSCLI_DEL_OPT='s3 rm'
++ AWSCLIOPT=
++ AWSCLI_ENDPOINT_OPT='--endpoint-url http://192.168.32.3/'
++ GCSCLI=/root/google-cloud-sdk/bin/gsutil
++ GCSCLI_COPY_OPT=cp
++ GCSCLI_LIST_OPT=ls
++ GCSCLI_DEL_OPT=rm
++ GCSCLIOPT=
++ DATE_CMD=/bin/date
++ create_past_yyyymmdd 0
++ '[' 1 -ne 1 ']'
+++ /bin/date +%Y%m%d -d '0 days ago'
++ echo 20220310
+ PAST=20220310
+ TARBALL_PAST='backup-20220310*.tar.bz2'
++ echo s3://test/
++ cut -f1 -d:
+ '[' s3 == s3 ']'
+ s3_delete_file_if_delete_backup_day 's3://test/backup-20220310*.tar.bz2' 0 3
+ '[' 3 -ne 3 ']'
+ check_is_delete_backup_day 0 3
+ '[' 2 -ne 2 ']'
+++ /bin/date +%s -d '0 days ago'
++ /usr/bin/expr 1646843839 / 86400 % 3
+ MOD_COUNT=1
+ '[' 1 -ne 0 ']'
+ return 0
+ s3_delete_file 's3://test/backup-20220310*.tar.bz2'
+ '[' 1 -ne 1 ']'
++ dirname 's3://test/backup-20220310*.tar.bz2'
+ BUCKET_URL=s3://test
++ basename 's3://test/backup-20220310*.tar.bz2'
+ OBJECT_NAME='backup-20220310*.tar.bz2'
+ /usr/bin/aws --endpoint-url http://192.168.32.3/ s3 rm s3://test --recursive --exclude '*' --include 'backup-20220310*.tar.bz2'
delete: s3://test/backup-202203102359.tar.bz2
+ echo 'DELETED past backuped file on S3: s3://test/backup-20220310*.tar.bz2'
DELETED past backuped file on S3: s3://test/backup-20220310*.tar.bz2
bash-5.0# aws --endpoint-url http://192.168.32.3/ s3 ls s3://test/
2022-03-10 01:31:47          0 backup-202203102359.tar.gz
2022-03-10 00:52:58          5 temp
2022-03-10 01:05:09          5 temp202202031000.tar.gz
root@850a48d55db2:/# gsutil ls gs://staging.mongodb-awesome-backup.appspot.com/test/weseek_mongodb-awesome-backup_1981862196/dummy-backup-20180101*.tar.bz2
gs://staging.mongodb-awesome-backup.appspot.com/test/weseek_mongodb-awesome-backup_1981862196/dummy-backup-20180101000000.tar.bz2
root@850a48d55db2:/# gsutil rm gs://staging.mongodb-awesome-backup.appspot.com/test/weseek_mongodb-awesome-backup_1981862196/dummy-backup-20180101*.tar.bz2
Removing gs://staging.mongodb-awesome-backup.appspot.com/test/weseek_mongodb-awesome-backup_1981862196/dummy-backup-20180101000000.tar.bz2...
/ [1 objects]

@skomma skomma merged commit 9b7c2a3 into master Mar 14, 2022
@skomma skomma deleted the fix/cannot-prune-backup-file-on-s3 branch March 14, 2022 03:39
@ryu-sato
Copy link
Contributor Author

ryu-sato commented Mar 15, 2022

I've confirmed that it can prune files in the GCS and have updated the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants