Enhance E2E performance and stability

[qiuming-best]

Describe the problem/challenge you have

• As we have more and more cases, our overall runtime is getting longer. How can we improve efficiency and parallelize?

• How to improve the stability of the cases?

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[qiuming-best]

1. Splitting the vSphere pipeline into two may be the most effective way to distribute time-consuming cases between the two pipelines.

2. Reducing the size of our persistent volume (PV). Currently, the default size for our created PV is 1Gi.

3. Optimize data transmission over the network by deploying Minio in the cluster.

4. While waiting for resources to be ready, use sleep sparingly or think carefully about sleep duration.

5. Each case should cover as many cases as possible to reduce the number of pipelines and free up resources.

6. Minimizing the installation and uninstallation of Velero to avoid duplicating the process unnecessarily.

[qiuming-best]

[Basic][StorageClass] - 329.122 seconds
[Migration][Snapshot] - 1327.754 seconds
[Migration][Snapshot] - 1136.013 seconds
[Upgrade][Restic] - 690.647 seconds
[Upgrade][Restic] - 720.865 seconds
[Backups][Deletion][Snapshot] - 512.234 seconds
[ResourceFiltering][LabelSelector] - 330.462 seconds
[ResourceFiltering][IncludeResources][Backup] - 325.271 seconds
[ResourceFiltering][IncludeNamespaces][Backup] - 222.823 seconds
[Basic][Restic] - 573.447 seconds
[ResourceFiltering][IncludeNamespaces][Restore] - 227.781 seconds
[ResourceFiltering][ExcludeResources][Restore] - 334.572 seconds
[Upgrade][Snapshot] - 759.531 seconds
[Upgrade][Snapshot] - 788.340 seconds
[NamespaceMapping][Single][Restic] - 415.521 seconds
[ResourceFiltering][ExcludeFromBackup] - 231.720 seconds
[Backups][BackupsSync] - 377.814 seconds
[Backups][BackupsSync] - 249.692 seconds
[PrivilegesMgmt][SSR] - 478.820 seconds
[NamespaceMapping][Multiple][Snapshot] - 1059.330 seconds
[APIGroup][Common] - 4096.759 seconds
[ResourceFiltering][ExcludeResources][Backup] - 332.859 seconds
[Basic][Snapshot] - 307.714 seconds
[Basic][Snapshot] - 1586.040 seconds
[NamespaceMapping][Single][Snapshot] - 777.082 seconds
[Basic][SelectedNode] - 249.388 seconds
[BSL][Deletion][Restic] - 323.203 seconds
[Basic][Nodeport] - 213.890 seconds
[Backups][Deletion][Restic] - 458.880 seconds
[ResourceFiltering][ExcludeNamespaces][Restore] - 229.488 seconds
[Basic][ClusterResource] - 189.076 seconds
[Basic][ClusterResource] - 17.494 seconds
[Basic][ClusterResource] - 40.441 seconds
[NamespaceMapping][Multiple][Restic] - 624.968 seconds
[Migration][Restic] - 787.489 seconds
[Migration][Restic] - 735.927 seconds
[ResourceFiltering][ExcludeNamespaces][Backup] - 227.792 seconds
[ResourceFiltering][ResourcePolicies] - 375.412 seconds
[ResourceFiltering][IncludeResources][Restore] - 331.665 seconds
[BSL][Deletion][Snapshot] - 384.762 seconds

TOTAL: 6h29m

[qiuming-best]

Change PV size and Minio:
TOTAL: 6h9m

Sleep time Adjustment:
TOTAL: 5h16m

Velero installation optimize:
TOTAL:4h21m

[qiuming-best]

The latest test result
[ResourceFiltering][ExcludeFromBackup] - 78.798 seconds
[PrivilegesMgmt][SSR] - 307.855 seconds
[ResourceFiltering][LabelSelector] - 162.817 seconds
[ResourceFiltering][ExcludeNamespaces][Backup] - 65.097 seconds
[Backups][Deletion][Restic] - 334.373 seconds
[Backups][Deletion][Snapshot] - 216.324 seconds
[ResourceFiltering][IncludeResources][Backup] - 165.707 seconds
[ResourceFiltering][ExcludeNamespaces][Restore] - 65.740 seconds
[Basic][Restic] - 612.029 seconds
[NamespaceMapping][Multiple][Snapshot] - 554.153 seconds
[Basic][Snapshot] - 718.483 seconds
[Basic][Snapshot] - 1246.558 seconds
[ResourceFiltering][ResourcePolicies] - 372.805 seconds
[ResourceFiltering][IncludeNamespaces][Backup] - 52.767 seconds
[NamespaceMapping][Single][Snapshot] - 213.145 seconds
[BSL][Deletion][Snapshot] - 159.090 seconds
[Backups][BackupsSync] - 214.412 seconds
[Backups][BackupsSync] - 252.170 seconds
[NamespaceMapping][Single][Restic] - 336.228 seconds
[Migration][Snapshot] - 1016.736 seconds
[Migration][Snapshot] - 679.179 seconds
[BSL][Deletion][Restic] - 315.740 seconds
[pv-backup][Opt-Out] - 199.200 seconds
[Basic][ClusterResource] - 20.969 seconds
[Basic][ClusterResource] - 71.788 seconds
[Basic][ClusterResource] - 35.902 seconds
[pv-backup][Opt-In] - 192.004 seconds
[ResourceFiltering][IncludeNamespaces][Restore] - 60.543 seconds
[Basic][Nodeport] - 55.538 seconds
[ResourceFiltering][IncludeResources][Restore] - 171.035 seconds
[Migration][Restic] - 674.018 seconds
[Migration][Restic] - 664.156 seconds
[Upgrade][Snapshot] - 635.856 seconds
[Upgrade][Snapshot] - 622.140 seconds
[ResourceFiltering][ExcludeResources][Restore] - 182.842 seconds
[NamespaceMapping][Multiple][Restic] - 368.462 seconds
[ResourceFiltering][ExcludeResources][Backup] - 185.136 seconds
[APIGroup][APIVersion] - 1303.923 seconds
[Upgrade][Restic] - 565.845 seconds
[Upgrade][Restic] - 590.402 seconds

TOTAL:4h21m