Version compatibility matrix

[divbell]

Describe the problem/challenge you have

The release cycle for both Velero and Kubernetes is a fast-moving target. Many users are stuck on various older versions of Kubernetes, particularly in a cloud managed kubernetes scenario, and there is ambiguity on what Velero versions are compatible with what Kubernetes versions.

Describe the solution you'd like

A clear compatibility matrix or description of Velero compatibility with k8s from 1.15 to present.

Anything else you would like to add:

No

Environment:

• Velero version (use velero version):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

[eleanor-millman]

@a-mccarthy I worked with Bridget to get the following info:

Velero versions that can run on K8s versions:
1.4: 1.10-1.21
1.5: 1.10-1.21
1.60-1.6.2: 1.12-1.21
1.6.3-1.7.1: 1.12-latest
1.8: 1.16-latest

Please note that it is much more complicated to figure out which Velero backups can be restored on which Kubernetes versions.

[sseago]

It might be worth writing "1.7.1" as "1.7.z" -- if there's another 1.7 bugfix release, it will still work on 1.12+ since we have the v1beta1 CRDs in the release-1.7 branch.

Also, it's possible that 1.5 also doesn't work with 1.10 and/or 1.11 -- if it's important to get this right, look for when Velero added schema validation to the CRDs, as this is when the CRDs would no longer load on 1.10 (and eventually on 1.11).

[a-mccarthy]

@eleanor-millman how do we know this information? Do we actively test these versions or did we test them at some point? (I genuinely dont know what we test, so feel free to point me to a doc on that if we have one :) )

Looks like 1.8 will drop support for kubernetes 1.12-1.15. Is there anything we need to do to make sure the community knows this is happening? Those versions are pretty old, but will this take folks by surprise?

[eleanor-millman]

@sseago and @a-mccarthy excellent points/questions! @zubron do you mind replying, since you have more context than I?

[zubron]

@a-mccarthy I can't speak for versions of Velero that were released before or just as I joined the team (v1.4 and v1.5) but we performed manual testing on Kubernetes 1.12 at the time of the Velero 1.6 release and also tested it against the latest version of Kubernetes at that time. For the older version of velero, it was explicitly documented that we supported Kubernetes 1.10 so I assume it was tested at some point. As @sseago points out though, Velero v1.5 may not have worked with Kubernetes 1.10 and 1.11. I would need to look into when the changes that broke support were introduced to verify that.

Until recently, we have been testing against Kubernetes 1.15 and later (CRD validation and E2E tests) as part of our CI runs but now only test against Kubernetes v1.16 and later (see https://github.com/vmware-tanzu/velero/actions/runs/1588285070 for an example).

You're right, we will be dropping support for Kubernetes 1.12-1.15 with the Velero 1.8 release, and yes, it may take some users by surprise! I'm pretty sure it has been discussed in the community meeting, but we should also notify on Slack and on the mailing list. Users can still opt to use older versions of Velero on older versions of Kubernetes and restore these on newer Kubernetes clusters using newer versions of Velero (as the backup format is versioned and will still be compatible).

[sseago]

@zubron As I mentioned, the details on when the incompatibilty came in are fuzzy now, but I know for sure that there was a 1.10-incompatible release before we updated the documentation. There was a point where 1.10 failed and 1.11 worked, and then we hit a point where 1.12 was minimum. If we want to be sure, take the 1.5 release and try it with both of those. I'm pretty sure 1.10 will fail. I'm less clear on 1.11.

[sseago]

Regarding dropping 1.15 and older support, I know it was mentioned on the community call (in fact, originally we were going to drop support for those versions with 1.7), but outside of that call, I don't know whether it was really mentioned.

[a-mccarthy]

Thanks for the context @zubron and @sseago! If we are dropping support for kubernetes versions with 1.8, we should

• Bring it up again in a community meeting (cc:@jonas)
• Included in the release notes for 1.8 (this might already be there, but we should make sure)
• Update the docs (i can make a PR for this)

The install page, https://velero.io/docs/v1.7/basic-install/, for example calls for kubernetes 1.12 or later. I can check around for other places were we explicitly list a kubernetes version and make updates.

Based on @zubron's comment here

Following up from a discussion, our CRDs are no longer compatible with Kubernetes v1.11 and earlier. We state that we support v1.10 and later, so we should revisit our list of supported Kubernetes versions.
To summarise the issue, support for the field "type" when using the status subresource in the CRD validation schema was only introduced in Kubernetes v1.12. This means that velero 1.5 and later cannot be installed on Kubernetes 1.11 or earlier.

We should update the version support list to be:
1.4: 1.10-1.21
1.5: 1.12-1.21
1.60-1.6.2: 1.12-1.21
1.6.3-1.7.1: 1.12-latest
1.8: 1.16-latest

@sseago is that the change you were thinking about? or do you think there is still some testing to do?

[sseago]

@a-mccarthy The above is fine. We'd only need additional testing if we want to determine whether 1.5 works with 1.11. I think it's safer to just start with 1.12 as you have above.