Add cherry-pick commits and changelog for v1.5.4 (#3651)

* Restore CAPI cluster objects in a better order

Restoring CAPI workload clusters without this ordering caused the
capi-controller-manager code to panic, resulting in an unhealthy cluster
state.

This can be worked around
(https://community.pivotal.io/s/article/5000e00001pJyN41611954332537?language=en_US),
but we provide the inclusion of these resources as a default in order to
provide a better out-of-the-box experience.

Signed-off-by: Nolan Brubaker <[email protected]>

* Add changelog

Signed-off-by: Nolan Brubaker <[email protected]>

* Use pod namespace from backup when matching PVBs (#3475)

* Use pod namespace from backup when matching PVBs

In #3051, we introduced an additional check to ensure that a PVB matched
a particular pod by checking both the name and the namespace of the pod.
This caused an issue when using a namespace mapping on restore. In the
case where a namespace mapping is being used, the check for whether a
PVB matches a particular pod will fail as the PVB was created for the
original pod namespace and is not aware of the new namespace mapping
being used. This resulted in PVRs not being created for pods that were
being restored into new namespaces. The restic init containers were
being created to wait on the volume restore, however this would cause
the restored pods to block indefinitely as they would be waiting for a
volume restore that was not scheduled.

To fix this, we use the original namespace of the pod from the backup to
match the PVB to the pod being restored, not the new namespace where
the pod is being restored into.

Fixes #3467.

Signed-off-by: Bridget McErlean <[email protected]>

* Explain why the namespace mapping can't be used

Signed-off-by: Bridget McErlean <[email protected]>

* Allow Dockerfiles to be configurable (#3634)

For internal builds of Velero, we need to be able to specify an
alternative Dockerfile which uses an alternative image registry to pull
the base images from. This change adapts our Makefile such that both the
main Dockerfile and build image Dockerfile can be overridden.

We have some special handling for the build image to only build when the
Dockerfile has changed. In this case, we check whether a custom
Dockerfile has been provided, and always rebuild in that case. For
custom build image Dockerfiles, use a fixed tag rather than the one
based on commit SHA of the original file.

Signed-off-by: Bridget McErlean <[email protected]>

* Combine CRD install verification into 1 job, and update k8s versions (#3448)

* Validate CRDs against latest Kubernetes versions

Add Kubernetes v1.19 and v1.20 series images, and consolidate the job
into a single file to reduce repetition.

Signed-off-by: Nolan Brubaker <[email protected]>

* Ignore job if the changes are only site/design

Signed-off-by: Nolan Brubaker <[email protected]>

* Fix codespell error

Signed-off-by: Nolan Brubaker <[email protected]>

* Cache Velero binary for reuse on workers

This will cache the Velero binary based on the PR number and a SHA256 of
the generated binary.

This way, the runners testing each version of Kubernetes do not need to
build it independently.

Signed-off-by: Nolan Brubaker <[email protected]>

* Fix GitHub event access

Signed-off-by: Nolan Brubaker <[email protected]>

* Wrap output path in quotes

Signed-off-by: Nolan Brubaker <[email protected]>

* Move code checkout to build step

Signed-off-by: Nolan Brubaker <[email protected]>

* Also cache go modules

Signed-off-by: Nolan Brubaker <[email protected]>

* Fix syntax issues

Signed-off-by: Nolan Brubaker <[email protected]>

* Download cached binary on each node

Signed-off-by: Nolan Brubaker <[email protected]>

* Use cached go modules on main CI

Signed-off-by: Nolan Brubaker <[email protected]>

* Add changelog for v1.5.4

Signed-off-by: Bridget McErlean <[email protected]>

Co-authored-by: Nolan Brubaker <[email protected]>

.github/workflows/crds-verify-kind.yaml

@@ -0,0 +1,86 @@
+name: "Verify Velero CRDs across k8s versions"
+on:
+  pull_request:
+    # Do not run when the change only includes these directories.
+    paths-ignore:
+      - "site/**"
+      - "design/**"
+
+jobs:
+  # Build the Velero CLI once for all Kubernetes versions, and cache it so the fan-out workers can get it.
+  build-cli:
+    runs-on: ubuntu-latest
+    steps:
+      # Look for a CLI that's made for this PR
+      - name: Fetch built CLI
+        id: cache
+        uses: actions/cache@v2
+        env:
+          cache-name: cache-velero-cli
+        with:
+          path: ./_output/bin/linux/amd64/velero
+          # The cache key a combination of the current PR number, and a SHA256 hash of the Velero binary
+          key: velero-${{ github.event.pull_request.number }}-${{ hashFiles('./_output/bin/linux/amd64/velero') }}
+          # This key controls the prefixes that we'll look at in the cache to restore from
+          restore-keys: |
+            velero-${{ github.event.pull_request.number }}-
+
+      - name: Fetch cached go modules
+        uses: actions/cache@v2
+        if: steps.cache.outputs.cache-hit != 'true'
+        with:
+          path: ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+
+      - name: Check out the code
+        uses: actions/checkout@v2
+        if: steps.cache.outputs.cache-hit != 'true'
+
+      # If no binaries were built for this PR, build it now.
+      - name: Build Velero CLI
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: |
+          make local
+
+  # Check the common CLI against all kubernetes versions
+  crd-check:
+    needs: build-cli
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # Latest k8s versions. There's no series-based tag, nor is there a latest tag.
+        k8s:
+          - 1.15.12
+          - 1.16.15
+          - 1.17.17
+          - 1.18.15
+          - 1.19.7
+          - 1.20.2
+    # All steps run in parallel unless otherwise specified.
+    # See https://docs.github.com/en/actions/learn-github-actions/managing-complex-workflows#creating-dependent-jobs
+    steps:
+      - name: Fetch built CLI
+        id: cache
+        uses: actions/cache@v2
+        env:
+          cache-name: cache-velero-cli
+        with:
+          path: ./_output/bin/linux/amd64/velero
+          # The cache key a combination of the current PR number, and a SHA256 hash of the Velero binary
+          key: velero-${{ github.event.pull_request.number }}-${{ hashFiles('./_output/bin/linux/amd64/velero') }}
+          # This key controls the prefixes that we'll look at in the cache to restore from
+          restore-keys: |
+            velero-${{ github.event.pull_request.number }}-
+      - uses: engineerd/[email protected]
+        with:
+          image: "kindest/node:v${{ matrix.k8s }}"
+      - name: Install CRDs
+        run: |
+          kubectl cluster-info
+          kubectl get pods -n kube-system
+          kubectl version
+          echo "current-context:" $(kubectl config current-context)
+          echo "environment-kubeconfig:" ${KUBECONFIG}
+          ./_output/bin/linux/amd64/velero install --crds-only --dry-run -oyaml | kubectl apply -f -

.github/workflows/pr-ci-check.yml

@@ -1,14 +1,20 @@
 name: Pull Request CI Check
 on: [pull_request]
 jobs:
-
   build:
     name: Run CI
     runs-on: ubuntu-latest
     steps:
+      - name: Check out the code
+        uses: actions/checkout@v2
 
-    - name: Check out the code
-      uses: actions/checkout@v2
+      - name: Fetch cached go modules
+        uses: actions/cache@v2
+        with:
+          path: ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
 
-    - name: Make ci
-      run: make ci
+      - name: Make ci
+        run: make ci

Makefile

@@ -26,13 +26,29 @@ REGISTRY ?= velero
 # Image name
 IMAGE ?= $(REGISTRY)/$(BIN)
 
-# Build image handling. We push a build image for every changed version of 
+# We allow the Dockerfile to be configurable to enable the use of custom Dockerfiles
+# that pull base images from different registries.
+VELERO_DOCKERFILE ?= Dockerfile
+BUILDER_IMAGE_DOCKERFILE ?= hack/build-image/Dockerfile
+
+# Calculate the realpath of the build-image Dockerfile as we `cd` into the hack/build
+# directory before this Dockerfile is used and any relative path will not be valid.
+BUILDER_IMAGE_DOCKERFILE_REALPATH := $(shell realpath $(BUILDER_IMAGE_DOCKERFILE))
+
+# Build image handling. We push a build image for every changed version of
 # /hack/build-image/Dockerfile. We tag the dockerfile with the short commit hash
 # of the commit that changed it. When determining if there is a build image in
 # the registry to use we look for one that matches the current "commit" for the
 # Dockerfile else we make one.
+# In the case where the Dockerfile for the build image has been overridden using
+# the BUILDER_IMAGE_DOCKERFILE variable, we always force a build.
+
+ifneq "$(origin BUILDER_IMAGE_DOCKERFILE)" "file"
+	BUILDER_IMAGE_TAG := "custom"
+else
+	BUILDER_IMAGE_TAG := $(shell git log -1 --pretty=%h $(BUILDER_IMAGE_DOCKERFILE))
+endif
 
-BUILDER_IMAGE_TAG := $(shell git log -1 --pretty=%h hack/build-image/Dockerfile)
 BUILDER_IMAGE := $(REGISTRY)/build-image:$(BUILDER_IMAGE_TAG)
 BUILDER_IMAGE_CACHED := $(shell docker images -q ${BUILDER_IMAGE} 2>/dev/null )
 
@@ -170,7 +186,7 @@ endif
 	--build-arg=VERSION=$(VERSION) \
 	--build-arg=GIT_SHA=$(GIT_SHA) \
 	--build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \
-	-f Dockerfile .
+	-f $(VELERO_DOCKERFILE) .
 
 container:
 ifneq ($(BUILDX_ENABLED), true)
@@ -186,7 +202,7 @@ endif
 	--build-arg=GIT_SHA=$(GIT_SHA) \
 	--build-arg=GIT_TREE_STATE=$(GIT_TREE_STATE) \
 	--build-arg=RESTIC_VERSION=$(RESTIC_VERSION) \
-	-f Dockerfile .
+	-f $(VELERO_DOCKERFILE) .
 	@echo "container: $(IMAGE):$(VERSION)"
 
 SKIP_TESTS ?=
@@ -233,11 +249,17 @@ build-dirs:
 	@mkdir -p .go/src/$(PKG) .go/pkg .go/bin .go/std/$(GOOS)/$(GOARCH) .go/go-build .go/golangci-lint
 
 build-env:
-	@# if we detect changes in dockerfile force a new build-image 
+	@# if we have overridden the value for the build-image Dockerfile,
+	@# force a build using that Dockerfile
+	@# if we detect changes in dockerfile force a new build-image
 	@# else if we dont have a cached image make one
 	@# finally use the cached image
-ifneq ($(shell git diff --quiet HEAD -- hack/build-image/Dockerfile; echo $$?), 0)
-	@echo "Local changes detected in hack/build-image/Dockerfile"
+ifneq "$(origin BUILDER_IMAGE_DOCKERFILE)" "file"
+	@echo "Dockerfile for builder image has been overridden to $(BUILDER_IMAGE_DOCKERFILE)"
+	@echo "Preparing a new builder-image"
+	$(MAKE) build-image
+else ifneq ($(shell git diff --quiet HEAD -- $(BUILDER_IMAGE_DOCKERFILE); echo $$?), 0)
+	@echo "Local changes detected in $(BUILDER_IMAGE_DOCKERFILE)"
 	@echo "Preparing a new builder-image"
 	$(MAKE) build-image
 else ifneq ($(BUILDER_IMAGE_CACHED),)
@@ -252,9 +274,9 @@ build-image:
 	@# This makes sure we don't leave the orphaned image behind.
 	$(eval old_id=$(shell docker image inspect  --format '{{ .ID }}' ${BUILDER_IMAGE} 2>/dev/null))
 ifeq ($(BUILDX_ENABLED), true)
-	@cd hack/build-image && docker buildx build --build-arg=GOPROXY=$(GOPROXY) --output=type=docker --pull -t $(BUILDER_IMAGE) .
+	@cd hack/build-image && docker buildx build --build-arg=GOPROXY=$(GOPROXY) --output=type=docker --pull -t $(BUILDER_IMAGE) -f $(BUILDER_IMAGE_DOCKERFILE_REALPATH) .
 else
-	@cd hack/build-image && docker build --build-arg=GOPROXY=$(GOPROXY) --pull -t $(BUILDER_IMAGE) .
+	@cd hack/build-image && docker build --build-arg=GOPROXY=$(GOPROXY) --pull -t $(BUILDER_IMAGE) -f $(BUILDER_IMAGE_DOCKERFILE_REALPATH) .
 endif
 	$(eval new_id=$(shell docker image inspect  --format '{{ .ID }}' ${BUILDER_IMAGE} 2>/dev/null))
 	@if [ "$(old_id)" != "" ] && [ "$(old_id)" != "$(new_id)" ]; then \
@@ -264,7 +286,13 @@ endif
 push-build-image:
 	@# this target will push the build-image it assumes you already have docker
 	@# credentials needed to accomplish this.
-	docker push $(BUILDER_IMAGE)
+	@# Pushing will be skipped if a custom Dockerfile was used to build the image.
+	ifneq "$(origin BUILDER_IMAGE_DOCKERFILE)" "file"
+		@echo "Dockerfile for builder image has been overridden"
+		@echo "Skipping push of custom image"
+	else
+		docker push $(BUILDER_IMAGE)
+	endif
 
 build-image-hugo:
 	cd site && docker build --pull -t $(HUGO_IMAGE) .

changelogs/CHANGELOG-1.5.md

@@ -1,3 +1,20 @@
+## v1.5.4
+### 2021-03-31
+### Download
+https://github.com/vmware-tanzu/velero/releases/tag/v1.5.4
+
+### Container Image
+`velero/velero:v1.5.4`
+
+### Documentation
+https://velero.io/docs/v1.5/
+
+### Upgrading
+https://velero.io/docs/v1.5/upgrade-to-1.5/
+
+  * Fixed a bug where restic volumes would not be restored when using a namespace mapping. (#3475, @zubron)
+  * Add CAPI Cluster and ClusterResourceSets to default restore priorities so that the capi-controller-manager does not panic on restores. (#3446, @nrb)
+
 ## v1.5.3
 ### 2021-01-14
 ### Download

pkg/cmd/server/server.go

@@ -468,6 +468,9 @@ func (s *server) veleroResourcesExist() error {
 //	 have restic restores run before controllers adopt the pods.
 // - Replica sets go before deployments/other controllers so they can be explicitly
 //	 restored and be adopted by controllers.
+// - CAPI Clusters come before ClusterResourceSets because failing to do so means the CAPI controller-manager will panic.
+//	 Both Clusters and ClusterResourceSets need to come before ClusterResourceSetBinding in order to properly restore workload clusters.
+//   See https://github.com/kubernetes-sigs/cluster-api/issues/4105
 var defaultRestorePriorities = []string{
 	"customresourcedefinitions",
 	"namespaces",
@@ -487,6 +490,8 @@ var defaultRestorePriorities = []string{
 	// to ensure that we prioritize restoring from "apps" too, since this is how they're stored
 	// in the backup.
 	"replicasets.apps",
+	"clusters.cluster.x-k8s.io",
+	"clusterresourcesets.addons.cluster.x-k8s.io",
 }
 
 func (s *server) initRestic() error {

pkg/restic/common.go

@@ -95,17 +95,17 @@ func getPodSnapshotAnnotations(obj metav1.Object) map[string]string {
 	return res
 }
 
-func isPVBMatchPod(pvb *velerov1api.PodVolumeBackup, pod metav1.Object) bool {
-	return pod.GetName() == pvb.Spec.Pod.Name && pod.GetNamespace() == pvb.Spec.Pod.Namespace
+func isPVBMatchPod(pvb *velerov1api.PodVolumeBackup, podName string, namespace string) bool {
+	return podName == pvb.Spec.Pod.Name && namespace == pvb.Spec.Pod.Namespace
 }
 
 // GetVolumeBackupsForPod returns a map, of volume name -> snapshot id,
 // of the PodVolumeBackups that exist for the provided pod.
-func GetVolumeBackupsForPod(podVolumeBackups []*velerov1api.PodVolumeBackup, pod metav1.Object) map[string]string {
+func GetVolumeBackupsForPod(podVolumeBackups []*velerov1api.PodVolumeBackup, pod metav1.Object, sourcePodNs string) map[string]string {
 	volumes := make(map[string]string)
 
 	for _, pvb := range podVolumeBackups {
-		if !isPVBMatchPod(pvb, pod) {
+		if !isPVBMatchPod(pvb, pod.GetName(), sourcePodNs) {
 			continue
 		}