🏃 Implement conformance test via e2e test framework #782

sbueringer · 2021-03-13T19:47:24Z

This PR refactors our conformance test to use the cluster-api test framework.
It doesn't introduce additional e2e tests yet, but after this PR this will be fairly easy.
I just didn't want to make the PR even bigger.

I also synced the Makefiles with the ones from AWS .

Notes:

our conformance test now depends on an image on a GitHub release on my image-builder fork:
- I would like to fix this in a follow-up issue: Provide qemu OS images via the capo staging bucket #794
- I had to upgrade because the old image was not compatible with the CAPI test framework
We should do a test release in a few weeks (e.g. a v0.4.0-beta.0) to test the new release Makefile targets and update the documentation
- I also would like to do this in a follow-up isssue: Create a v0.4.0.beta.0 release to verify the new release targets #796

xref: #577

/hold

k8s-ci-robot · 2021-03-13T19:47:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2021-03-15T14:26:46Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-15T14:34:45Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/49730dd5322e43c2a74a9805aefecbf3/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 8s

sbueringer · 2021-03-16T05:22:32Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-16T07:53:07Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/da32635f18774162acce115b2d60c0bc/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 9s

sbueringer · 2021-03-16T17:40:26Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-16T17:41:09Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/a87e7070d21d492a9f8a5634348e970c/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 10s

sbueringer · 2021-03-16T20:18:30Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-16T20:19:15Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/40dd3951c3054656a4b448358cf3256d/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 8s

theopenlab-ci · 2021-03-16T21:24:46Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/5d43de49d738491ea8555070d9572930/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 8s

sbueringer · 2021-03-16T21:24:54Z

/test pull-cluster-api-provider-openstack-make-conformance

sbueringer · 2021-03-16T22:09:58Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-16T22:10:57Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/7fd6c19b38bd4c1bb0dd8d26cda63b70/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 8s

sbueringer · 2021-03-17T05:33:02Z

/test pull-cluster-api-provider-openstack-make-conformance

sbueringer · 2021-03-17T06:05:14Z

/test pull-cluster-api-provider-openstack-make-conformance

sbueringer · 2021-03-17T06:39:00Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-17T07:01:53Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/ed1b32ee0e4547d69912a70a1802c0bc/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 8s

sbueringer · 2021-03-17T16:47:45Z

/test pull-cluster-api-provider-openstack-make-conformance

theopenlab-ci · 2021-03-17T16:48:31Z

Build failed.

cluster-api-provider-openstack-current-acceptance-test-v1.18 : ERROR Unable to find playbook /var/lib/zuul/builds/db8cb9d41aaf4ecfb138f56d3bf65c55/untrusted/project_0/github.com/kubernetes-sigs/cluster-api-provider-openstack/.zuul/playbooks/run.yaml in 10s

sbueringer · 2021-03-24T05:00:21Z

@hidekazuna @prankul88 do you want to review this PR or should we just go ahead? :) It's mostly just about test code

sbueringer · 2021-03-24T05:05:27Z

/test pull-cluster-api-provider-openstack-make-conformance

prankul88 · 2021-03-24T10:13:23Z

templates/cluster-template-external-cloud-provider.yaml

@@ -3,7 +3,6 @@ apiVersion: cluster.x-k8s.io/v1alpha4
 kind: Cluster
 metadata:
  name: ${CLUSTER_NAME}
-  namespace: '${NAMESPACE}'


This was added with #735
@sbueringer I hope it does not conflict with it.

I dropped it because it isn't needed. The namespaces of all CAPI resources are set even without it and our cloudsSecret.namespace defaults to the same namespace as our other resources. Actually are e2e tests are using this because they are deploying to a random generated namespace (the e2e test with multiple flavors from my other PR even 2 clusters at the same time in different namespaces)

Yes, CAPO works without this. Only I followed CAPI recommendation
https://cluster-api.sigs.k8s.io/clusterctl/provider-contract.html#common-variables

I thought someone uses multi cloud provider and they wonder they can not use --target-namespace for CAPO.

I think it was okay that we had it before. My interpretation of the documentation is (now):

If you have places in you're YAML were you need the target namespace, you can/should use this variable. It is not necessary to use it for the metadata.namespace field as that is set by CAPI in any case

I looked at CAPG / CAPA / CAPZ / CAPV and I only could find it in CAPV. As it's not necessary and most of the other providers are not using it I thought it's cleaner to drop it.

prankul88 · 2021-03-24T10:24:56Z

LGTM !! Thankyou

sbueringer · 2021-03-24T17:57:26Z

/test ?

k8s-ci-robot · 2021-03-24T17:57:28Z

@sbueringer: The following commands are available to trigger jobs:

/test pull-cluster-api-provider-openstack-build
/test pull-cluster-api-provider-openstack-test
/test pull-cluster-api-provider-openstack-e2e-test
/test pull-cluster-api-provider-openstack-conformance-test

Use /test all to run the following jobs:

pull-cluster-api-provider-openstack-build
pull-cluster-api-provider-openstack-test

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2021-03-24T17:59:36Z

/test pull-cluster-api-provider-openstack-conformance-test

In case anybody is wondering because the test failed, it's because of a race condition of ssh tunneling and cluster deletion. I fixed this here in the other PR:

cluster-api-provider-openstack/test/e2e/shared/exec.go

Lines 78 to 88 in 9d5eb52

    
           defer timeoutCancel() 
        
           go func() { 
        
           	<-timeout.Done() 
        
           	bastionConn.Close() 
        
           }() 
        
           conn, err := bastionConn.Dial("tcp", fmt.Sprintf("%s:22", machineIP)) 
        
           if err != nil { 
        
           	_, _ = fmt.Fprintf(GinkgoWriter, "couldn't connect from the bastion host %s to the target instance %s: %s\n", bastionIP, machineIP, err) 
        
           	return 
        
           } 
        
           defer conn.Close()

I wouldn't backport it as it happens frequently and I expect we will merge the other PR pretty soon, too.

hidekazuna · 2021-03-25T00:12:57Z

test/e2e/data/infrastructure-openstack/cluster-template-external-cloud-provider.yaml

+apiVersion: addons.cluster.x-k8s.io/v1alpha4
+kind: ClusterResourceSet
+metadata:
+  name:  "${CLUSTER_NAME}-crs-0"
+spec:
+  strategy: ApplyOnce
+  clusterSelector:
+    matchLabels:
+      cni: "${CLUSTER_NAME}-crs-0"
+  resources:
+  - name: "cni-${CLUSTER_NAME}-crs-0"
+    kind: ConfigMap


Do we need to update templates to use ClusterResourceSet in templates directory as well?

I think we should keep it as is. In almost all of the templates of the other providers it is only used for testing. I think it would break the quickstart documentation to change it.

I found ClusterResourceSet is still experimental, after GA we would use it in templates.

hidekazuna · 2021-03-25T00:17:19Z

test/e2e/data/ci-artifacts-platform-kustomization.yaml

+          # Copyright 2020 The Kubernetes Authors.
+          #
+          # Licensed under the Apache License, Version 2.0 (the "License");
+          # you may not use this file except in compliance with the License.
+          # You may obtain a copy of the License at
+          #
+          #     http://www.apache.org/licenses/LICENSE-2.0
+          #
+          # Unless required by applicable law or agreed to in writing, software
+          # distributed under the License is distributed on an "AS IS" BASIS,
+          # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+          # See the License for the specific language governing permissions and
+          # limitations under the License.


This is noisy.. I am not familiar with law, but do we really need this?

The copyright header?

I would keep it as is for now. But, I wanted to do a follow-up anyway to get rid of the whole /usr/local/bin/ci-artifacts.sh file in this YAML anyway. There are some reasons why we are not using the upstream function to add this functionality:

they are using gsutil instead of curl. Thus they install google cloud sdk first which adds ~3-5 minutes to to node startup in our environment (probably in all environments)

they are also pulling the "kube-apiserver" "kube-controller-manager" "kube-scheduler" for worker nodes, which is not necessary and also adds time to the test
3/ when we use the upstream method to patch our KubeadmControlPlane and KubeadmConfigTemplate to add the /usr/local/bin/ci-artifacts.shscript we will lose the/etc/kubernetes/cloud.confand/etc/certs/cacertfiles because the wholefiles` array is overwritten. This is a limitation of kustomize, but I assume with the fixes merged to kustomize a week ago we are able to improve this (Strategic merge support for CRDs kustomize#2825) (although it's not completely trivial)

I wanted to do a follow-up to improve all those things in the cluster-api util func GenerateCIArtifactsInjectedTemplateForDebian I currently copied into tmp_template.go and then drop this in our repo. I think until know I would like to keep the diff as small as possible.

I opened a issue for it here: #804

Thanks, understand.

sbueringer · 2021-03-25T04:14:07Z

/test pull-cluster-api-provider-openstack-make-conformance

/hold

@hidekazuna Would be nice if you can take a look and lgtm in case you agree with my comments. Thank you :)

I would merge after your lgtm and a successful test-run. I rebased onto master because of a conflict in hack/tools/go.mod

k8s-ci-robot · 2021-03-25T04:14:08Z

@sbueringer: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test pull-cluster-api-provider-openstack-build
/test pull-cluster-api-provider-openstack-test
/test pull-cluster-api-provider-openstack-e2e-test
/test pull-cluster-api-provider-openstack-conformance-test

Use /test all to run the following jobs:

pull-cluster-api-provider-openstack-build
pull-cluster-api-provider-openstack-test

In response to this:

/test pull-cluster-api-provider-openstack-make-conformance

/hold

@hidekazuna Would be nice if you can take a look and lgtm in case you agree with my comments. Thank you :)

I would merge after your lgtm and a successful test-run. I rebased onto master because of a conflict in hack/tools/go.mod

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2021-03-25T04:14:17Z

/test pull-cluster-api-provider-openstack-conformance-test

k8s-ci-robot · 2021-03-25T06:18:25Z

@sbueringer: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-cluster-api-provider-openstack-make-conformance	`bade7b9`	link	`/test pull-cluster-api-provider-openstack-make-conformance`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sbueringer · 2021-03-25T07:33:29Z

/test pull-cluster-api-provider-openstack-conformance-test

(let's see if it reoccurs, pretty much had 100% stable tests before the last rebase)

hidekazuna · 2021-03-25T11:44:11Z

/lgtm

sbueringer · 2021-03-25T11:51:21Z

/hold cancel

k8s-ci-robot requested review from jichenjc and prankul88 March 13, 2021 19:47

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 13, 2021

sbueringer mentioned this pull request Mar 14, 2021

🏃 migrate conformance test to Prow #759

Merged

sbueringer force-pushed the caas/sbuerin/capi-prow-e2e-tests branch from 8273622 to 7054730 Compare March 15, 2021 14:25

sbueringer mentioned this pull request Mar 15, 2021

Provide documentation website (like CAPA/CAPZ) #783

Closed

sbueringer force-pushed the caas/sbuerin/capi-prow-e2e-tests branch from 7054730 to 22fb620 Compare March 16, 2021 05:21

sbueringer mentioned this pull request Mar 23, 2021

🏃 Add e2e tests for all flavors #798

Merged

3 tasks

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 24, 2021

sbueringer force-pushed the caas/sbuerin/capi-prow-e2e-tests branch from 1d6e07d to bade7b9 Compare March 24, 2021 05:05

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 24, 2021

prankul88 reviewed Mar 24, 2021

View reviewed changes

hidekazuna reviewed Mar 25, 2021

View reviewed changes

implement conformance test with e2e framework

efa7d41

sbueringer force-pushed the caas/sbuerin/capi-prow-e2e-tests branch from bade7b9 to efa7d41 Compare March 25, 2021 04:10

sbueringer mentioned this pull request Mar 25, 2021

Cleanup GenerateCIArtifactsInjectedTemplateForDebian #804

Closed

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 25, 2021

k8s-ci-robot assigned hidekazuna Mar 25, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 25, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 25, 2021

k8s-ci-robot merged commit 2456b1a into kubernetes-sigs:master Mar 25, 2021

sbueringer deleted the caas/sbuerin/capi-prow-e2e-tests branch March 26, 2021 06:05

mdbooth mentioned this pull request Nov 4, 2021

🐛 Fix image push in Makefile for image-push jobs #1039

Merged

3 tasks

🏃 Implement conformance test via e2e test framework #782

🏃 Implement conformance test via e2e test framework #782

Conversation

sbueringer commented Mar 13, 2021 • edited Loading

k8s-ci-robot commented Mar 13, 2021

sbueringer commented Mar 15, 2021

theopenlab-ci bot commented Mar 15, 2021

sbueringer commented Mar 16, 2021

theopenlab-ci bot commented Mar 16, 2021

sbueringer commented Mar 16, 2021

theopenlab-ci bot commented Mar 16, 2021

sbueringer commented Mar 16, 2021

theopenlab-ci bot commented Mar 16, 2021

theopenlab-ci bot commented Mar 16, 2021

sbueringer commented Mar 16, 2021

sbueringer commented Mar 16, 2021

theopenlab-ci bot commented Mar 16, 2021

sbueringer commented Mar 17, 2021

sbueringer commented Mar 17, 2021

sbueringer commented Mar 17, 2021

theopenlab-ci bot commented Mar 17, 2021

sbueringer commented Mar 17, 2021

theopenlab-ci bot commented Mar 17, 2021

sbueringer commented Mar 24, 2021

sbueringer commented Mar 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidekazuna Mar 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prankul88 commented Mar 24, 2021

sbueringer commented Mar 24, 2021

k8s-ci-robot commented Mar 24, 2021

sbueringer commented Mar 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbueringer commented Mar 25, 2021

k8s-ci-robot commented Mar 25, 2021

sbueringer commented Mar 25, 2021

k8s-ci-robot commented Mar 25, 2021 • edited Loading

sbueringer commented Mar 25, 2021 • edited Loading

hidekazuna commented Mar 25, 2021

sbueringer commented Mar 25, 2021

sbueringer commented Mar 13, 2021 •

edited

Loading

hidekazuna Mar 24, 2021 •

edited

Loading

k8s-ci-robot commented Mar 25, 2021 •

edited

Loading

sbueringer commented Mar 25, 2021 •

edited

Loading