Hybrid VolumeSnapshot/Restic backup (consistent, online, point-in-time filesystem backup)

[asoltesz]

Describe the problem/challenge you have

Velero doesn't have consistent, point-in-time filesystem backups for my storage provider (Rook/Ceph) and several others.

It does have Restic integration which can take a filesystem backup from Persistent Volumes but in order to make it consistent, one has to stop the workload, otherwise the files may change inconsistently with each other as the backup progresses.

Velero also has support for the generic VolumeSnapshot API (snapshot.storage.k8s.io/v1beta1) but that can only take onsite backups, as opposed to storing the backup in an offsite repository (e.g.: S3 bucket). VolumeSnapshots are stored at a location managed by the cloud provider and typically cannot even be moved between regions so they cannot be called offsite.

Describe the solution you'd like

Combine the VolumeSnapshot capability with Restic backups:

• Velero takes a VolumeSnapshot, so a new VolumeSnapshotContent is created by the CSI driver of the storage provider

• Velero creates a standalone, temporary Restic pod with a newly created PVC which is sourced from the VolumeSnapshot taken in the previous step (PVCs support this type of creation from VolumeSnapshots)

• Velero runs Restic on the new PV and stores the files in the Repository in the remote, offsite backup repository (e.g.: s3 bucket). In this case, incremental backups may also be possible so backup is faster and consumes less storage.

• After successful backup, Velero drops the the temporary Restic pod and also drops the temporary backup PV/PVC and the VolumeSnapshot(Content).

Restore would be the same as a normal Restic restore.

This would combine the advantages of both backup mechanism (VolumeSnapshot+Restic) and allow snapshot-style, point-in-time, online and offsite backups for any storage provider that has a CSI plugin supporting the VolumeSnapshot API.

Anything else you would like to add:

Environment:

• Velero version (use velero version): v1.4
• Kubernetes version (use kubectl version): v1.15.11
• Kubernetes installer & version: Rancher 3.5.2
• Cloud provider or hardware configuration: Hetzner Cloud with Rancher/RKE created cluster with Rook/Ceph as storage provider
• OS (e.g. from /etc/os-release): CentOS 7

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[onedr0p]

Honestly it would be nice to have this prioritized, it seems like a very valid usecase for having backups be reliable.

[Lyndon-Li]

Implemented in v1.12