Design doc for upload progress monitoring

[dsu-igeek]

Signed-off-by: Dave Smith-Uchida [email protected]

[ashish-amarnath]

Here is an alternate proposal:
The goal here is to ensure the durability of Velero backups, and that backups, once completed, can be fully restored.
To accomplish this we propose:

1. Introducing a new BackupProgressTracker gRpc service that would be implemented as a plugin. This service would have the familiar AppliesTo and Execute rpcs. The AppliesTo rpc would indicate what resources this plugin applies to, and the Execute (this can be called something else) would take as input ExecuteRequest with fields item and the backup (similar to BackupItemAction's ExecuteRequest), and return, as output, ExecuteResponse with fields similar to the UploadProgress` that's in this proposal.

2. Following changes in the Backup controller:
   i. Here, instead of setting the backup phase as Completed would set the phase as Uploading.
   ii. Here, start a progress tracker goroutine which would periodically, say every 1s, invoke all the BackupProgressTracker plugins' Execute rpc to obtain the status of the backup for each item and patch the backup object's status with this info. Once the UploadProgress from all the plugins have reached a state indicating durability of the backup, the goroutine returns which is when the backup's phase will be set to completed/uploaded.

3. Requires the following changes to the backup API:

   • Introduce a map[string]UploadProgress as a field in the backup status which will be patched by the goroutine explained in 2.ii above. This will be mapping item to its latest UploadProgress

Benefits of this approach:

1. Introducing a new gRpc service is not a breaking change. (So is adding a new rpc to an existing service, but this is more intuitive in the go-plugin framework.)
2. It is more extendable across resource types as it enables tracking upload progress of all resource types, even those that don't have a corresponding BackupItemAction or a VolumeSnapshotter interface.
3. UploadProgress of all items in the backup can be tracked in a consistent way and in one single point.

We may have previously considered this approach but this seems more intuitive to me now than modifying the existing BackupItemAction and VolumeSnapshotter plugins. WDYT?

[dsu-igeek]

Here is an alternate proposal:
The goal here is to ensure the durability of Velero backups, and that backups, once completed, can be fully restored.
To accomplish this we propose:

1. Introducing a new BackupProgressTracker gRpc service that would be implemented as a plugin. This service would have the familiar AppliesTo and Execute rpcs. The AppliesTo rpc would indicate what resources this plugin applies to, and the Execute (this can be called something else) would take as input ExecuteRequest with fields item and the backup (similar to BackupItemAction's ExecuteRequest), and return, as output, ExecuteResponse with fields similar to the UploadProgress` that's in this proposal.

2. Following changes in the Backup controller:
   i. Here, instead of setting the backup phase as Completed would set the phase as Uploading.
   ii. Here, start a progress tracker goroutine which would periodically, say every 1s, invoke all the BackupProgressTracker plugins' Execute rpc to obtain the status of the backup for each item and patch the backup object's status with this info. Once the UploadProgress from all the plugins have reached a state indicating durability of the backup, the goroutine returns which is when the backup's phase will be set to completed/uploaded.

3. Requires the following changes to the backup API:

   • Introduce a map[string]UploadProgress as a field in the backup status which will be patched by the goroutine explained in 2.ii above. This will be mapping item to its latest UploadProgress

Benefits of this approach:

1. Introducing a new gRpc service is not a breaking change. (So is adding a new rpc to an existing service, but this is more intuitive in the go-plugin framework.)
2. It is more extendable across resource types as it enables tracking upload progress of all resource types, even those that don't have a corresponding BackupItemAction or a VolumeSnapshotter interface.
3. UploadProgress of all items in the backup can be tracked in a consistent way and in one single point.

We may have previously considered this approach but this seems more intuitive to me now than modifying the existing BackupItemAction and VolumeSnapshotter plugins. WDYT?

Upload progress may take a long time. We want to be able to recover from restarts of the Velero server while uploads are in progress. Also, if the backup is deleted before the upload finishes we want to terminate any polling on upload status and, if possible, abort the upload.

The design introduces a BackupItemProgress plug-in for BackupItemAction monitoring. In order to monitor the upload progress, we will need to know what to monitor, which is why the BackupItemAction needs to be modified to return a snapshot ID. It might be worthwhile to add a new BackupItemActionWithSnapshot plug-in rather than add the snapshot ID return to the existing plug-ins.

The VolumeSnapshotter plugins work at the volume level, so invoking a BackupItemProgress action for them requires that they map the PV to a volume. Adding the UploadProgress API should not be a breaking change once API versioning has been introduced.

[ashish-amarnath]

The approach LGTM.
Added a few comments that can be addressed/figured out during implementation.

[carlisia]

Couple suggestions, couple comments, couple questions.

Overall lgtm, looks like someone could take it and implemented. Can't wait to have this split in the backup process!

[ashish-amarnath]

@dsu-igeek Looks like the latest commit is missing a sign-off.

[carlisia]

👍