Backup hooks fsfreeze example does nothing useful

[kaovilai]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
Working on a related lab project and noticing that creating a file with pre-hook annotation to a pod's mounted pvc, may not exist on restore if pvc is backed up prior to pre-hookCommand execution.

The fsfreeze example would only do something useful if pvc is guaranteed to backup in between pre and post backup hook of the pod.

What did you expect to happen:
Ability to have pod pre-hook command execute, then mounted pvc gets backed up, only then does the post-hook command execute.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

https://gist.github.com/kaovilai/8e9b814e2a85bb91e05921f3e9ed9aff#file-backup-log-L628-L665

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
This issue maybe partially closed with #4067
At the pod annotation level, there is currently no way to have mounted pvc be backed up in between a pod's pre and post hook.

Environment:

• Velero version (use velero version): v1.7.0
• Velero features (use velero client config get features): not set
• Kubernetes version (use kubectl version): v1.21.1+f36aa36
• Kubernetes installer & version:
• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release): Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[kaovilai]

One universal solution to pvc problem would be, check all pods that will be backed up for pre-post hooks and if those pods also mount pvcs, to backup pvcs between the pre- and post- hooks.

This may assume a pvc belongs to at most one pod.

[reasonerjt]

I don't quite understand why Backup hooks fsfreeze example does nothing useful
It does useful if the pv we want to freeze the fs to quiesce the app?
Therefore the example still makes sense?

It seems you are trying to point out a limitation of the current backup hook mechanism.
To help us better understand it would be helpful if you can elaborate how your workload looks like and how the current hook mechanism cannot ensure successful backup.

[kaovilai]

@reasonerjt In one of my backup hooks experiments, instead of fsfreeze, I replace it with touch <filename> in pre-hook and touch <filename2> for post-hook.

Both pre-hook and post-hook created a file after backup is completed, but pre-hook created file is not persisted in the restore.

In another case, I created a file in the same location (example's mounted pvc) prior to creating backup. Now this file does exist in the restore, but again the prehook and posthook files are not existing in restore.

This confirms the theory that

1. Pre-hook and Post-hook do as they say, they execute before a pod is backed up, and after a pod is backed up. (works correctly)
2. PersistentVolumeClaims are not guaranteed to be backedup during the time between a pod's pre-hook and post-hook. Thus, in the fsfreeze example, where you are freezing a path to PVC but that PVC is either, already backed up.. or PVC won't be backed up until after a pod's post-hook is already executed.

• A file backed up inside the mounted PVC won't be during a state where fsfreeze is applied in the velero example linked. Backedup PVC will be before pre-hook or after post-hook of the intended fsfreeze state.. which is again, not a useful example as the file system is either never frozen, or are already unfrozen during PVC backup.

PS: I chatted with @sseago and he agrees this is the case currently and making annotated pod's mounted pvc backup between the annotated pod's pre and post backup hooks (and making fsfreeze example useful) will require major refactoring.

[kaovilai]

fsfreeze example is only useful when

• mounted pvc /var/log/nginx is not written to (ie. frozen) during pvc backup with the use of pod backup hook annotation
  • this is not the case because
    • pod's pre-backup hooks are not guaranteed to have been applied before mounted pvc is backed up
    • pod's post-backup hook (unfreezing pvc) are not guaranteed to have been applied after mounted pvc is backed up
    • restored pvc when mounted to restored pod are not guaranteed to be from the frozen state after pod's pre-hook execution and before pod's post-hook execution.

[kaovilai]

Sample workload

• pod with mounted pvc
• pvc mounted in a pod

Add something to PVC prior to backup

touch pathToMountedPVC/priorToPrehook

Adding pod annotation

kubectl annotate pod -n <namespace> -l <labelSelector>
pre.hook.backup.velero.io/command='["/bin/bash", "-c", "echo \"pre\" > pathToMountedPVC/lab-pre-backup"]' \
post.hook.backup.velero.io/command='["/bin/bash", "-c", "echo \"post\" > pathToMountedPVC/lab-post-backup"]'

Expected path in restore

• pathToMountedPVC/priorToPrehook
• pathToMountedPVC/lab-pre-backup

Actual path in restore

• pathToMountedPVC/priorToPrehook

[reasonerjt]

@kaovilai
I did some tests with the annotation you mentioned.
It seems there is a time-window that the pre-hook command is returned but the content is not persisted to the disk.
Could you try sleep a few seconds in the pre.hook.backup.velero.io/command and check if the file pathToMountedPVC/lab-pre-backup is in your restore?

In my test I could see the lab-pre-backup in the restore using this file:
https://github.com/reasonerjt/velero/blob/tmp-hook-investigation/examples/nginx-app/with-pv-sleep120.yaml
but I could not see the lab-pre-backup in the restore using this file:
https://github.com/reasonerjt/velero/blob/tmp-hook-investigation/examples/nginx-app/with-pv.yaml

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing the stale issue.