Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Flatcar Container Linux template #2890

Merged
merged 3 commits into from
Jan 23, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Add Flatcar Container Linux templates and e2e tests
This commit adds end-user templates and e2e tests for clusters using
Flatcar Container Linux.

Based-on-work-by: Jeremi Piotrowski <[email protected]>
Signed-off-by: Mateusz Gozdek <[email protected]>
jepio authored and invidian committed Jan 23, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 8dad8f074688f1790b08a185ed0a33a6bcf3fd4b
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -706,7 +706,7 @@ kind-create: $(KUBECTL) ## Create capz kind cluster if needed.

.PHONY: tilt-up
tilt-up: install-tools kind-create ## Start tilt and build kind cluster if needed.
EXP_CLUSTER_RESOURCE_SET=true EXP_MACHINE_POOL=true tilt up
EXP_CLUSTER_RESOURCE_SET=true EXP_MACHINE_POOL=true EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true tilt up

.PHONY: delete-cluster
delete-cluster: delete-workload-cluster ## Deletes the example kind cluster "capz".
2 changes: 2 additions & 0 deletions Tiltfile
Original file line number Diff line number Diff line change
@@ -23,6 +23,7 @@ settings = {
"cert_manager_version": "v1.10.0",
"kubernetes_version": "v1.24.6",
"aks_kubernetes_version": "v1.24.6",
"flatcar_version": "3374.2.1",
}

keys = ["AZURE_SUBSCRIPTION_ID", "AZURE_TENANT_ID", "AZURE_CLIENT_SECRET", "AZURE_CLIENT_ID"]
@@ -337,6 +338,7 @@ def deploy_worker_templates(template, substitutions):
"AZURE_CONTROL_PLANE_MACHINE_TYPE": "Standard_B2s",
"WORKER_MACHINE_COUNT": "2",
"AZURE_NODE_MACHINE_TYPE": "Standard_B2s",
"FLATCAR_VERSION": settings.get("flatcar_version"),
}

if flavor == "aks":
1 change: 1 addition & 0 deletions docs/book/src/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -30,6 +30,7 @@
- [Virtual Networks](./topics/custom-vnet.md)
- [VM Identity](./topics/vm-identity.md)
- [Windows](./topics/windows.md)
- [Flatcar](./topics/flatcar.md)
- [Development](./developers/development.md)
- [Kubernetes Developers](./developers/kubernetes-developers.md)
- [Releasing](./developers/releasing.md)
41 changes: 41 additions & 0 deletions docs/book/src/topics/flatcar.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Flatcar Clusters

## Overview

CAPZ enables you to create Kubernetes clusters using Flatcar Container Linux on Microsoft Azure.

### Image creation

The testing reference images are built using [image-builder](https://github.com/kubernetes-sigs/image-builder) by Flatcar maintainers and published to the Flatcar CAPI Community Gallery on Azure with community gallery name `flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0`.

<aside class="note warning">

<h1> Security </h1>

The reference images are not updated with security fixes. They are intended only to facilitate testing and to help users try out Cluster API for Azure.

The reference images should not be used in a production environment. It is highly recommended to [maintain your own custom image](#building-a-custom-image) instead.

</aside>

Find the latest published images:

```console
$ az sig image-definition list-community --location westeurope --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --only-show-errors
HyperVGeneration Location Name OsState OsType UniqueId
------------------ ---------- ---------------------------------- ----------- -------- ---------------------------------------------------------------------------------------------------------------
V2 westeurope flatcar-stable-amd64-capi-v1.23.13 Generalized Linux /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.23.13
V2 westeurope flatcar-stable-amd64-capi-v1.25.4 Generalized Linux /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.25.4
V2 westeurope flatcar-stable-amd64-capi-v1.26.0 Generalized Linux /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.26.0
$
$ az sig image-version list-community --location westeurope --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --only-show-errors --gallery-image-definition flatcar-stable-amd64-capi-v1.26.0
ExcludeFromLatest Location Name PublishedDate UniqueId
------------------- ---------- -------- -------------------------------- --------------------------------------------------------------------------------------------------------------------------------
False westeurope 3227.2.3 2022-12-09T18:05:58.830464+00:00 /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.26.0/Versions/3227.2.3
```

If you would like customize your images please refer to the documentation on building your own [custom images](custom-images.md).

## Trying it out

To create a cluster using Flatcar Container Linux, use `flatcar` cluster flavor.
265 changes: 265 additions & 0 deletions templates/cluster-template-flatcar.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions templates/flavors/flatcar/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
namespace: default
resources:
- ../base
- machine-deployment.yaml
- ../../azure-cluster-identity

patchesStrategicMerge:
- patches/kubeadm-controlplane.yaml
- ../../azure-cluster-identity/azurecluster-identity-ref.yaml
79 changes: 79 additions & 0 deletions templates/flavors/flatcar/machine-deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
clusterName: ${CLUSTER_NAME}
replicas: ${WORKER_MACHINE_COUNT}
selector:
matchLabels: null
template:
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: ${CLUSTER_NAME}-md-0
clusterName: ${CLUSTER_NAME}
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
name: ${CLUSTER_NAME}-md-0
version: ${KUBERNETES_VERSION}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
image:
computeGallery:
gallery: flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0
name: flatcar-stable-amd64-capi-${KUBERNETES_VERSION}
version: ${FLATCAR_VERSION}
osDisk:
diskSizeGB: 128
osType: Linux
sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""}
vmSize: ${AZURE_NODE_MACHINE_TYPE}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
files:
- contentFrom:
secret:
key: worker-node-azure.json
name: ${CLUSTER_NAME}-md-0-azure-json
owner: root:root
path: /etc/kubernetes/azure.json
permissions: "0644"
format: ignition
ignition:
containerLinuxConfig:
additionalConfig: |
systemd:
units:
- name: kubeadm.service
dropins:
- name: 10-flatcar.conf
contents: |
[Unit]
After=oem-cloudinit.service
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
azure-container-registry-config: /etc/kubernetes/azure.json
cloud-config: /etc/kubernetes/azure.json
cloud-provider: azure
name: '@@HOSTNAME@@'
postKubeadmCommands: []
preKubeadmCommands:
- sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' | jq -r .compute.name)/g" /etc/kubeadm.yml
63 changes: 63 additions & 0 deletions templates/flavors/flatcar/patches/kubeadm-controlplane.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: ${CLUSTER_NAME}-control-plane
spec:
kubeadmConfigSpec:
# Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7679.
diskSetup:
filesystems: []
partitions: []
format: ignition
ignition:
containerLinuxConfig:
additionalConfig: |
systemd:
units:
- name: kubeadm.service
dropins:
- name: 10-flatcar.conf
contents: |
[Unit]
After=oem-cloudinit.service
# Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7679.
storage:
disks:
- device: /dev/disk/azure/scsi1/lun0
partitions:
- number: 1
filesystems:
- name: etcd_disk
mount:
device: /dev/disk/azure/scsi1/lun0
format: ext4
label: etcd_disk
options:
- -E
- lazy_itable_init=1,lazy_journal_init=1
initConfiguration:
nodeRegistration:
name: '@@HOSTNAME@@'
joinConfiguration:
nodeRegistration:
name: '@@HOSTNAME@@'
mounts:
- - etcd_disk
- /var/lib/etcddisk
postKubeadmCommands: []
preKubeadmCommands:
- sed -i "s/@@HOSTNAME@@/$(curl -s -H Metadata:true --noproxy '*' 'http://169.254.169.254/metadata/instance?api-version=2020-09-01' | jq -r .compute.name)/g" /etc/kubeadm.yml
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureMachineTemplate
metadata:
name: ${CLUSTER_NAME}-control-plane
spec:
template:
spec:
image:
computeGallery:
gallery: flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0
name: flatcar-stable-amd64-capi-${KUBERNETES_VERSION}
version: ${FLATCAR_VERSION}
270 changes: 270 additions & 0 deletions templates/test/ci/cluster-template-prow-flatcar.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions templates/test/ci/prow-flatcar/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
- ../../../flavors/flatcar/
patchesStrategicMerge:
- ../patches/tags.yaml
- ../patches/controller-manager.yaml
14 changes: 9 additions & 5 deletions test/e2e/azure_logcollector.go
Original file line number Diff line number Diff line change
@@ -237,23 +237,27 @@ func linuxLogs(execToPathFn func(outputFileName string, command string, args ...
return []func() error{
execToPathFn(
"journal.log",
"journalctl", "--no-pager", "--output=short-precise",
"sudo", "journalctl", "--no-pager", "--output=short-precise",
),
execToPathFn(
"kern.log",
"journalctl", "--no-pager", "--output=short-precise", "-k",
"sudo", "journalctl", "--no-pager", "--output=short-precise", "-k",
),
execToPathFn(
"kubelet-version.txt",
"kubelet", "--version",
"PATH=/opt/bin:${PATH}", "kubelet", "--version",
),
execToPathFn(
"kubelet.log",
"journalctl", "--no-pager", "--output=short-precise", "-u", "kubelet.service",
"sudo", "journalctl", "--no-pager", "--output=short-precise", "-u", "kubelet.service",
),
execToPathFn(
"containerd.log",
"journalctl", "--no-pager", "--output=short-precise", "-u", "containerd.service",
"sudo", "journalctl", "--no-pager", "--output=short-precise", "-u", "containerd.service",
),
execToPathFn(
"ignition.log",
"sudo", "journalctl", "--no-pager", "--output=short-precise", "-at", "ignition",
),
execToPathFn(
"cloud-init.log",
31 changes: 30 additions & 1 deletion test/e2e/azure_test.go
Original file line number Diff line number Diff line change
@@ -315,6 +315,34 @@ var _ = Describe("Workload cluster creation", func() {
})
})

Context("Creating a Flatcar cluster [OPTIONAL]", func() {
It("With Flatcar control-plane and worker nodes", func() {
clusterName = getClusterName(clusterNamePrefix, "flatcar")

clusterctl.ApplyClusterTemplateAndWait(ctx, clusterctl.ApplyClusterTemplateAndWaitInput{
ClusterProxy: bootstrapClusterProxy,
ConfigCluster: clusterctl.ConfigClusterInput{
LogFolder: filepath.Join(artifactFolder, "clusters", bootstrapClusterProxy.GetName()),
ClusterctlConfigPath: clusterctlConfigPath,
KubeconfigPath: bootstrapClusterProxy.GetKubeconfigPath(),
InfrastructureProvider: clusterctl.DefaultInfrastructureProvider,
Flavor: "flatcar",
Namespace: namespace.Name,
ClusterName: clusterName,
KubernetesVersion: e2eConfig.GetVariable(FlatcarKubernetesVersion),
ControlPlaneMachineCount: pointer.Int64Ptr(1),
WorkerMachineCount: pointer.Int64Ptr(1),
},
WaitForClusterIntervals: e2eConfig.GetIntervals(specName, "wait-cluster"),
WaitForControlPlaneIntervals: e2eConfig.GetIntervals(specName, "wait-control-plane"),
WaitForMachineDeployments: e2eConfig.GetIntervals(specName, "wait-worker-nodes"),
ControlPlaneWaiters: clusterctl.ControlPlaneWaiters{
WaitForControlPlaneInitialized: EnsureControlPlaneInitialized,
},
}, result)
})
})

Context("Creating a ipv6 control-plane cluster [REQUIRED]", func() {
It("With ipv6 worker node", func() {
clusterName = getClusterName(clusterNamePrefix, "ipv6")
@@ -724,7 +752,8 @@ var _ = Describe("Workload cluster creation", func() {
return AKSMachinePoolSpecInput{
Cluster: result.Cluster,
MachinePools: result.MachinePools,
WaitIntervals: e2eConfig.GetIntervals(specName, "wait-machine-pool-nodes")}
WaitIntervals: e2eConfig.GetIntervals(specName, "wait-machine-pool-nodes"),
}
})
})

3 changes: 3 additions & 0 deletions test/e2e/common.go
Original file line number Diff line number Diff line change
@@ -75,13 +75,16 @@ const (
Timestamp = "TIMESTAMP"
AKSKubernetesVersion = "AKS_KUBERNETES_VERSION"
AKSKubernetesVersionUpgradeFrom = "AKS_KUBERNETES_VERSION_UPGRADE_FROM"
FlatcarKubernetesVersion = "FLATCAR_KUBERNETES_VERSION"
FlatcarVersion = "FLATCAR_VERSION"
SecurityScanFailThreshold = "SECURITY_SCAN_FAIL_THRESHOLD"
SecurityScanContainer = "SECURITY_SCAN_CONTAINER"
ManagedClustersResourceType = "managedClusters"
capiImagePublisher = "cncf-upstream"
capiOfferName = "capi"
capiWindowsOfferName = "capi-windows"
aksClusterNameSuffix = "aks"
flatcarCAPICommunityGallery = "flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0"
)

func Byf(format string, a ...interface{}) {
5 changes: 5 additions & 0 deletions test/e2e/config/azure-dev.yaml
Original file line number Diff line number Diff line change
@@ -144,6 +144,8 @@ providers:
targetName: "clusterclass-ci-default.yaml"
- sourcePath: "${PWD}/templates/test/ci/cluster-template-prow-topology.yaml"
targetName: "cluster-template-topology.yaml"
- sourcePath: "${PWD}/templates/test/ci/cluster-template-prow-flatcar.yaml"
targetName: "cluster-template-flatcar.yaml"
replacements:
- old: "--v=0"
new: "--v=2"
@@ -152,6 +154,8 @@ variables:
AKS_KUBERNETES_VERSION: "latest"
AKS_KUBERNETES_VERSION_UPGRADE_FROM: "latest-1"
KUBERNETES_VERSION: "${KUBERNETES_VERSION:-stable-1.24}"
FLATCAR_KUBERNETES_VERSION: "${FLATCAR_KUBERNETES_VERSION:-stable-1.24}"
mboersma marked this conversation as resolved.
Show resolved Hide resolved
FLATCAR_VERSION: "${FLATCAR_VERSION:-latest}"
ETCD_VERSION_UPGRADE_TO: "3.5.4-0"
COREDNS_VERSION_UPGRADE_TO: "v1.9.3"
KUBERNETES_VERSION_UPGRADE_TO: "${KUBERNETES_VERSION_UPGRADE_TO:-stable-1.25}"
@@ -163,6 +167,7 @@ variables:
EXP_MACHINE_POOL: "true"
EXP_CLUSTER_RESOURCE_SET: "true"
CLUSTER_TOPOLOGY: "true"
EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION: "true"
CONFORMANCE_WORKER_MACHINE_COUNT: "2"
CONFORMANCE_CONTROL_PLANE_MACHINE_COUNT: "${CONFORMANCE_CONTROL_PLANE_MACHINE_COUNT:-1}"
CONFORMANCE_IMAGE: "${CONFORMANCE_IMAGE:-}"
121 changes: 112 additions & 9 deletions test/e2e/helpers.go
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@ import (
"text/tabwriter"
"time"

"github.com/Azure/azure-sdk-for-go/profiles/2020-09-01/compute/mgmt/compute"
"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-03-02/compute"
"github.com/Azure/go-autorest/autorest/azure/auth"
"github.com/blang/semver"
. "github.com/onsi/ginkgo/v2"
@@ -698,6 +698,7 @@ func resolveKubetestRepoListPath(version string, path string) (string, error) {
func resolveKubernetesVersions(config *clusterctl.E2EConfig) {
ubuntuVersions := getVersionsInOffer(context.TODO(), os.Getenv(AzureLocation), capiImagePublisher, capiOfferName)
windowsVersions := getVersionsInOffer(context.TODO(), os.Getenv(AzureLocation), capiImagePublisher, capiWindowsOfferName)
flatcarK8sVersions := getFlatcarK8sVersions(context.TODO(), os.Getenv(AzureLocation), flatcarCAPICommunityGallery)

// find the intersection of ubuntu and windows versions available, since we need an image for both.
var versions semver.Versions
@@ -716,18 +717,44 @@ func resolveKubernetesVersions(config *clusterctl.E2EConfig) {
if config.HasVariable(capi_e2e.KubernetesVersionUpgradeTo) {
resolveKubernetesVersion(config, versions, capi_e2e.KubernetesVersionUpgradeTo)
}
if config.HasVariable(FlatcarKubernetesVersion) && config.HasVariable(FlatcarVersion) {
resolveFlatcarKubernetesVersion(config, flatcarK8sVersions, FlatcarKubernetesVersion)
flatcarVersions := getFlatcarVersions(context.TODO(), os.Getenv(AzureLocation), flatcarCAPICommunityGallery, config.GetVariable(FlatcarKubernetesVersion))
resolveFlatcarVersion(config, flatcarVersions, FlatcarVersion)
}
}

func resolveKubernetesVersion(config *clusterctl.E2EConfig, versions semver.Versions, varName string) {
resolveVariable(config, varName, getLatestVersionForMinor(config.GetVariable(varName), versions, "capi offer"))
}

func resolveVariable(config *clusterctl.E2EConfig, varName, v string) {
oldVersion := config.GetVariable(varName)
v := getLatestSkuForMinor(oldVersion, versions)
if _, ok := os.LookupEnv(varName); ok {
Expect(os.Setenv(varName, v)).To(Succeed())
}
config.Variables[varName] = v
Logf("Resolved %s (set to %s) to %s", varName, oldVersion, v)
}

func resolveFlatcarKubernetesVersion(config *clusterctl.E2EConfig, versions semver.Versions, varName string) {
resolveVariable(config, varName, getLatestVersionForMinor(config.GetVariable(varName), versions, "Flatcar Community Gallery"))
}

func resolveFlatcarVersion(config *clusterctl.E2EConfig, versions semver.Versions, varName string) {
version := config.GetVariable(varName)
if version != "latest" {
Expect(versions).To(ContainElement(semver.MustParse(version)), fmt.Sprintf("Provided Flatcar version %q does not have a corresponding VM image in the Flatcar Community Gallery", version))
}

if version == "latest" {
semver.Sort(versions)
version = versions[0].String()
}

resolveVariable(config, varName, version)
}

// newImagesClient returns a new VM images client using environmental settings for auth.
func newImagesClient() compute.VirtualMachineImagesClient {
settings, err := auth.GetSettingsFromEnvironment()
@@ -741,6 +768,30 @@ func newImagesClient() compute.VirtualMachineImagesClient {
return imagesClient
}

func newCommunityGalleryImagesClient() compute.CommunityGalleryImagesClient {
settings, err := auth.GetSettingsFromEnvironment()
Expect(err).NotTo(HaveOccurred())
subscriptionID := settings.GetSubscriptionID()
authorizer, err := azureutil.GetAuthorizer(settings)
Expect(err).NotTo(HaveOccurred())
communityGalleryImagesClient := compute.NewCommunityGalleryImagesClient(subscriptionID)
communityGalleryImagesClient.Authorizer = authorizer

return communityGalleryImagesClient
}

func newCommunityGalleryImageVersionsClient() compute.CommunityGalleryImageVersionsClient {
settings, err := auth.GetSettingsFromEnvironment()
Expect(err).NotTo(HaveOccurred())
subscriptionID := settings.GetSubscriptionID()
authorizer, err := azureutil.GetAuthorizer(settings)
Expect(err).NotTo(HaveOccurred())
communityGalleryImageVersionsClient := compute.NewCommunityGalleryImageVersionsClient(subscriptionID)
communityGalleryImageVersionsClient.Authorizer = authorizer

return communityGalleryImageVersionsClient
}

// getVersionsInOffer returns a map of Kubernetes versions as strings to semver.Versions.
func getVersionsInOffer(ctx context.Context, location, publisher, offer string) map[string]semver.Version {
Logf("Finding image skus and versions for offer %s/%s in %s", publisher, offer, location)
@@ -781,32 +832,84 @@ func getVersionsInOffer(ctx context.Context, location, publisher, offer string)
return versions
}

// getLatestSkuForMinor gets the latest available patch version in the provided list of sku versions that corresponds to the provided k8s version.
func getLatestSkuForMinor(version string, skus semver.Versions) string {
// getLatestVersionForMinor gets the latest available patch version in the provided list of sku versions that corresponds to the provided k8s version.
func getLatestVersionForMinor(version string, versions semver.Versions, imagesSource string) string {
isStable, match := validateStableReleaseString(version)
if isStable {
// if the version is in the format "stable-1.21", we find the latest 1.21.x version.
major, err := strconv.ParseUint(match[1], 10, 64)
Expect(err).NotTo(HaveOccurred())
minor, err := strconv.ParseUint(match[2], 10, 64)
Expect(err).NotTo(HaveOccurred())
semver.Sort(skus)
for i := len(skus) - 1; i >= 0; i-- {
if skus[i].Major == major && skus[i].Minor == minor {
version = "v" + skus[i].String()
semver.Sort(versions)
for i := len(versions) - 1; i >= 0; i-- {
if versions[i].Major == major && versions[i].Minor == minor {
version = "v" + versions[i].String()
break
}
}
} else if v, err := semver.ParseTolerant(version); err == nil {
if len(v.Pre) == 0 {
// if the version is in the format "v1.21.2", we make sure we have an existing image for it.
Expect(skus).To(ContainElement(v), fmt.Sprintf("Provided Kubernetes version %s does not have a corresponding VM image in the capi offer", version))
Expect(versions).To(ContainElement(v), fmt.Sprintf("Provided Kubernetes version %s does not have a corresponding VM image in the %q", version, imagesSource))
}
}
// otherwise, we just return the version as-is. This allows for versions in other formats, such as "latest" or "latest-1.21".
return version
}

func getFlatcarVersions(ctx context.Context, location, galleryName, k8sVersion string) semver.Versions {
image := fmt.Sprintf("flatcar-stable-amd64-capi-%s", k8sVersion)

Logf("Finding Flatcar versions in community gallery %q in location %q for image %q", galleryName, location, image)
var versions semver.Versions
communityGalleryImageVersionsClient := newCommunityGalleryImageVersionsClient()
imageVersionsIterator, err := communityGalleryImageVersionsClient.List(ctx, location, galleryName, image)
Expect(err).NotTo(HaveOccurred())
imageVersions := imageVersionsIterator.Response()

if imageVersions.Value == nil {
return versions
}

for _, imageVersion := range *imageVersions.Value {
versions = append(versions, semver.MustParse(*imageVersion.Name))
}

return versions
}

func getFlatcarK8sVersions(ctx context.Context, location, communityGalleryName string) semver.Versions {
Logf("Finding Flatcar images and versions in community gallery %q in location %q", communityGalleryName, location)
var versions semver.Versions
k8sVersion := regexp.MustCompile(`flatcar-stable-amd64-capi-v(\d+)\.(\d+).(\d+)`)
communityGalleryImagesClient := newCommunityGalleryImagesClient()
communityGalleryImageVersionsClient := newCommunityGalleryImageVersionsClient()
imagesIterator, err := communityGalleryImagesClient.ListComplete(ctx, location, communityGalleryName)
Expect(err).NotTo(HaveOccurred())
images := imagesIterator.Response()

if images.Value == nil {
return versions
}

for _, image := range *images.Value {
resIterator, err := communityGalleryImageVersionsClient.List(ctx, location, communityGalleryName, *image.Name)
Expect(err).NotTo(HaveOccurred())
res := resIterator.Response()

if res.Value == nil || len(*res.Value) == 0 {
continue
}

match := k8sVersion.FindStringSubmatch(*image.Name)
stringVer := fmt.Sprintf("%s.%s.%s", match[1], match[2], match[3])
versions = append(versions, semver.MustParse(stringVer))
}

return versions
}

// getPodLogs returns the logs of a pod, or an error in string format.
func getPodLogs(ctx context.Context, clientset *kubernetes.Clientset, pod corev1.Pod) string {
req := clientset.CoreV1().Pods(pod.Namespace).GetLogs(pod.Name, &corev1.PodLogOptions{})