Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom s3 endpoint support #28

Merged
merged 9 commits into from
Jun 4, 2020
Merged

Custom s3 endpoint support #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
99bce93
Added new env var for custom endpoint
pitscher May 26, 2020
feeee33
Removed otc endpoint url and added placeholder
pitscher May 26, 2020
55c1e2a
Include custom s3 endpoint option in README
pitscher May 26, 2020
f6d75c5
Rename ENDPOINT env var
pitscher May 28, 2020
05c7fa8
Update README with new renamed ENDPOINT var
pitscher May 28, 2020
571a317
Remove old comment
pitscher Jun 2, 2020
52eccb5
Add check if custom aws endpoint is set
pitscher Jun 2, 2020
8e1f685
Add custom endpoint option to docker run examples
pitscher Jun 2, 2020
4b4c8a2
Remove repetitive if-statements
pitscher Jun 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ What is mongodb-awesome-backup?
-------------------------------

mongodb-awesome-backup is the collection of scripts which backup MongoDB databases to Amazon S3 or Google Cloud Storage.
You can set a custom S3 endpoint to use S3 based services like DigitalOcean Spaces instead of Amazon S3.


Requirements
Expand Down Expand Up @@ -39,6 +40,7 @@ docker run --rm \
[ -e MONGODB_USERNAME=<DB login username> \ ]
[ -e MONGODB_PASSWORD=<DB login password> \ ]
[ -e MONGODB_AUTHDB=<Authentication DB name> \ ]
[ -e AWSCLI_ENDPOINT_OPT=<S3 endpoint URL (ex. https://fra1.digitaloceanspaces.com)> \ ]
[ -v ~:/mab \ ]
weseek/mongodb-awesome-backup
```
Expand Down Expand Up @@ -66,6 +68,7 @@ docker run --rm \
[ -e MONGODB_USERNAME=<DB login username> \ ]
[ -e MONGODB_PASSWORD=<DB login password> \ ]
[ -e MONGODB_AUTHDB=<Authentication DB name> \ ]
[ -e AWSCLI_ENDPOINT_OPT=<S3 endpoint URL (ex. https://fra1.digitaloceanspaces.com)> \ ]
[ -v ~:/mab \ ]
weseek/mongodb-awesome-backup
```
Expand All @@ -90,6 +93,7 @@ docker run --rm \
[ -e MONGODB_PASSWORD=<DB login password> \ ]
[ -e MONGODB_AUTHDB=<Authentication DB name> \ ]
[ -e MONGORESTORE_OPTS=<Options list of mongorestore> \ ]
[ -e AWSCLI_ENDPOINT_OPT=<S3 endpoint URL (ex. https://fra1.digitaloceanspaces.com)> \ ]
[ -v ~:/mab \ ]
weseek/mongodb-awesome-backup restore
```
Expand Down Expand Up @@ -124,6 +128,7 @@ Environment variables
| MONGODB_AUTHDB | Authentication DB name | - |
| CRONMODE | If set "true", this container is executed in cron mode. In cron mode, the script will be executed with the specified arguments and at the time specified by CRON_EXPRESSION. | "false" |
| CRON_EXPRESSION | Cron expression (ex. "CRON_EXPRESSION=0 4 * * *" if you want to run at 4:00 every day) | - |
| AWSCLI_ENDPOINT_OPT | Set a custom S3 endpoint if you use a S3 based service like DigitalOcean Spaces. (ex. AWSCLI_ENDPOINT_OPT="https://fra1.digitaloceanspaces.com") If not set the Amazon S3 standard endpoint will be used. | - |
| HEALTHCHECKS_URL | URL that gets called after a successful backup (eg. https://healthchecks.io) | - |

### For `restore`
Expand Down Expand Up @@ -151,3 +156,4 @@ Environment variables
| MONGODB_PASSWORD | DB login password | - |
| MONGODB_AUTHDB | Authentication DB name | - |
| MONGORESTORE_OPTS | Options list of mongorestore. (ex --drop) | - |
| AWSCLI_ENDPOINT_OPT | Set a custom S3 endpoint if you use a S3 based service like DigitalOcean Spaces. (ex. AWSCLI_ENDPOINT_OPT="https://fra1.digitaloceanspaces.com") If not set the Amazon S3 standard endpoint will be used. | - |
9 changes: 5 additions & 4 deletions bin/functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ AWSCLI_COPY_OPT="s3 cp"
AWSCLI_LIST_OPT="s3 ls"
AWSCLI_DEL_OPT="s3 rm"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since AWSCLIOPT is already provided, I think you can do the same thing by using this environment variable without adding AWSCLI_ENDPOINT_URL_OPTION.

Unfortunately, currently AWSCLIOPT is not documented...

Copy link
Contributor Author

@pitscher pitscher May 28, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To provide the ability to use the default endpoint or a custom one and not over engineer the code we should use a new env var. Therefore I introduced AWSCLI_ENDPOINT_OPT

AWSCLIOPT=${AWSCLIOPT:-}
AWSCLI_ENDPOINT_OPT=${AWSCLI_ENDPOINT_OPT:+"--endpoint-url ${AWSCLI_ENDPOINT_OPT}"}

GCSCLI="/root/google-cloud-sdk/bin/gsutil"
GCSCLI_COPY_OPT="cp"
Expand All @@ -19,7 +20,7 @@ DATE_CMD="/bin/date"
# arguments: 1. s3 url (s3://.../...)
s3_exists() {
if [ $# -ne 1 ]; then return 255; fi
${AWSCLI} ${AWSCLIOPT} ${AWSCLI_LIST_OPT} $1 >/dev/null
${AWSCLI} ${AWSCLI_ENDPOINT_OPT} ${AWSCLIOPT} ${AWSCLI_LIST_OPT} $1 >/dev/null
}
gs_exists() {
if [ $# -ne 1 ]; then return 255; fi
Expand All @@ -29,7 +30,7 @@ gs_exists() {
# Output the list of the files on specified S3 URL.
# arguments: 1. s3 url (s3://...)
s3_list_files() {
${AWSCLI} ${AWSCLIOPT} ${AWSCLI_LIST_OPT} $1
${AWSCLI} ${AWSCLI_ENDPOINT_OPT} ${AWSCLIOPT} ${AWSCLI_LIST_OPT} $1
}
gs_list_files() {
${GCSCLI} ${GCSCLIOPT} ${GCSCLI_LIST_OPT} $1
Expand All @@ -39,7 +40,7 @@ gs_list_files() {
# arguments: 1. s3 url (s3://.../...)
s3_delete_file() {
if [ $# -ne 1 ]; then return 255; fi
${AWSCLI} ${AWSCLIOPT} ${AWSCLI_DEL_OPT} $1
${AWSCLI} ${AWSCLI_ENDPOINT_OPT} ${AWSCLIOPT} ${AWSCLI_DEL_OPT} $1
}
gs_delete_file() {
if [ $# -ne 1 ]; then return 255; fi
Expand All @@ -57,7 +58,7 @@ gs_delete_file() {
# 2. target s3 url (s3://...)
s3_copy_file() {
if [ $# -ne 2 ]; then return 255; fi
${AWSCLI} ${AWSCLIOPT} ${AWSCLI_COPY_OPT} $1 $2
${AWSCLI} ${AWSCLI_ENDPOINT_OPT} ${AWSCLIOPT} ${AWSCLI_COPY_OPT} $1 $2
}
gs_copy_file() {
if [ $# -ne 2 ]; then return 255; fi
Expand Down