Plan for supporting lots of versions in a single CRD

[matthchr]

Some services release API versions very frequently. AKS for example ships a new API version nearly every month. Each version often supports a few new features which might be interesting to a small subset of customers.

We don't currently automatically generate a new CRD version for each API version. Regardless, as time goes on the number of CRD versions we support for ManagedCluster will increase without bound. At some point, we will reach the limit on CRD size and be forced to deprecate some versions (even if the service itself has not deprecated that particular API version yet).

We should define how this deprecation will happen and which APIs will be chosen to remove

For example, we could say that certain API versions every year are LTS (meaning we'll keep them for 3+ years):
20200101 == LTS (supported for 3 years)
20200301 == non-LTS but supported (will remove after a year?)
20200501 == non-LTS
20200801 == non-LTS
20210101 == LTS or something?

[theunrepentantgeek]

Still something we need to consider, especially as the number of CRDs in ASO climbs.

[matthchr]

We also need to define a timeline for how long we announce a deprecation of an API version before we remove it. I've spoken with some customers who want at least a year, which at least for GA APIs seems pretty reasonable to me.

We also need to make sure to use kubebuilder:deprecatedversion do indicate its deprecation so that it shows up in the AKS upgrade warning.

[dtzar]

I think it's fair for preview API versions to only support something like N-1.

For stable API versions, that's harder. IMO it really depends more on how frequently the team releases stable API versions and when ASO supports it. So for this I'd go more in the direction of the age of the API and how long ASO has supported it to make a determination.

[matthchr]

I just ran into this when adding the newer version of AKS ManagedCluster API. It looks like at least for that API, having 6 variants of the API (including preview/GA) along with all of the documentation for each field pushes the CRD file to ~2.5M, which ends up crossing the 1.5M max size limit when we submit the CRD.

I don't fully follow why the filesize differs from the actual limit (I assume something to do with encodings).

In terms of what we should do in the near-term at least, I think we need to announce the deprecation of some of the API versions of ManagedCluster to reduce the size of this CRD to a more manageable level.

At least for this resource, we should commit to 2 preview versions and 3 GA versions? We can fit more if we strip documentation, which I don't love.

[matthchr]

We can also (eventually) fit more when AKS supports #2791, as we can strip the AgentPool from inside MC...

[dtzar]

For AKS - I propose only the latest preview version and N-1 GA versions. People can go to an older ASO version if they absolutely need to. GA versions typically are fairly good at being forward compatible IMO.