Backup with Medusa to S3 #60
Conversation
Add PodDisruptionBudget Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
# Conflicts: # operator/operator.yaml # templates/operator/operator.yaml.template
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Add cleanup for AWS S3 bucket with prefix Fixed backup job Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
…ssues when backup is disabled Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
…-backup # Conflicts: # operator/operator.yaml # templates/operator/operator.yaml.template
…ods but with parallel restore possible, test runs green locally Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
…sandra-operator into an/medusa-backup
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
# Conflicts: # operator/operator.yaml # operator/templates/node-readiness-probe-sh.yaml # operator/templates/pdb.yaml # operator/templates/stateful-set.yaml # templates/operator/operator.yaml.template # templates/operator/templates/stateful-set.yaml.template # tests/go.mod # tests/go.sum
Signed-off-by: Andreas Neumann <[email protected]>
# Conflicts: # operator/operator.yaml # operator/templates/stateful-set.yaml # templates/operator/operator.yaml.template # templates/operator/templates/stateful-set.yaml.template
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
# Conflicts: # run-tests.sh # tests/go.mod
Made backup and restore work with authn Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
Hopefully fixed bug in backup_restore test Signed-off-by: Andreas Neumann <[email protected]>
Signed-off-by: Andreas Neumann <[email protected]>
|
|
||
| COPY medusa/cassandra-sudoer /etc/sudoers.d/ | ||
|
|
||
| COPY medusa/medusa-src /medusa-src |
There was a problem hiding this comment.
This Dockerfile can probably be removed at some point. It's not included in the build.sh, but very useful to test new medusa versions from source.
Signed-off-by: Andreas Neumann <[email protected]>
There was a problem hiding this comment.
👏 Awesome work, this looks well thought out!
I only have some nits. Would be great to see the extra parameters that you marked as TODO in medusa.ini, but doesn't have to be part of this PR. Please run tools/generate_parameters_markdown.py to document the new parameters in docs/parameters.md.
| if [ ! -f /var/lib/cassandra/token_map ]; then | ||
| while [ -z $NODE_TOKENS ]; do | ||
| NODE_ID=`nodetool info | grep ID | sed -n -e 's/ID[[:space:]]*\:[[:space:]]*\(.*\)$/\1/p'` | ||
| NODE_IP=`nodetool status | grep $NODE_ID | sed -n -e 's/UN[[:space:]]*\([0-9.]*\)[[:space:]]*.*/\1/p'` |
There was a problem hiding this comment.
Nit: Do we have to run nodetool info and nodetool status for every iteration of the while-loop? How about running them once and only run nodetool ring in the loop? I'm not sure how many iterations such a loop has, but nodetool is quite expensive and this could reduce performance if there are many iterations.
There was a problem hiding this comment.
tbh, I'm not sure anymore. I think I've used that in one iteration of development to save the token_map. The while loop is there so the script waits until the cluster is fully up and running, but... Yeah, that could probably be optimized.
The script is currently not used, i just didn't want to delete it yet, as figuring out the sed regexps for the fields was a PITA.
I'll probably delete the script once the branch is merged - then it's still available in the git history.
Signed-off-by: Andreas Neumann <[email protected]>
There was a problem hiding this comment.
This is great!
I think this is perfect as a v1 feature for backup/restore feature which can later be evolved using better security features like IAM profiles and better backend configuration for Medusa so we are ready to support GCP storage and ceph storage as backend options which Medusa already provides.
The only change I would ask is if we use BACKUP_RESTORE_ENABLED true and later false, it should leave the kubernetes cluster as it was before enabling the feature.
Signed-off-by: Andreas Neumann <[email protected]>
SSL support is prepared, waiting for thelastpickle/cassandra-medusa#119 to be merged and released