Add E2E test for antrea multiclusters #3124

hjiajing · 2021-12-12T02:48:17Z

build clusters base on the Getting-started of multi-cluster
run test cases to export service and try to test the imported service.

This PR is based on #3024 , only the top commit need to be reviewed.

codecov-commenter · 2021-12-12T02:53:56Z

Codecov Report

Merging #3124 (84e18e9) into feature/multi-cluster (fb5ddfc) will increase coverage by 0.02%.
The diff coverage is 0.00%.

@@                    Coverage Diff                    @@
##           feature/multi-cluster    #3124      +/-   ##
=========================================================
+ Coverage                  41.03%   41.05%   +0.02%     
=========================================================
  Files                        192      191       -1     
  Lines                      22887    22702     -185     
=========================================================
- Hits                        9392     9321      -71     
+ Misses                     12548    12433     -115     
- Partials                     947      948       +1

Flag	Coverage Δ
unit-tests	`41.05% <0.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../controllers/multicluster/clusterset_controller.go	`0.00% <0.00%> (ø)`
...trollers/multicluster/resourceimport_controller.go	`0.00% <0.00%> (ø)`
pkg/agent/cniserver/ipam/antrea_ipam_controller.go	`61.53% <0.00%> (-3.85%)`	⬇️
pkg/controller/externalippool/controller.go	`83.81% <0.00%> (-1.74%)`	⬇️
...lticluster/core/mock_remote_common_area_manager.go
.../multicluster/mock_membercluster_status_manager.go
...llers/multicluster/core/mock_remote_common_area.go

luolanzone · 2021-12-13T01:04:42Z

multicluster/test/yamls/clusterset.yml

+      serviceAccount: antrea-multicluster-controller
+    - clusterID: test-cluster-west
+      serviceAccount: antrea-multicluster-controller
+  namespace: antrea-mcs-ns


missing newline

luolanzone · 2021-12-13T01:05:01Z

multicluster/test/yamls/east-member-cluster.yml

+      server: https://<LEADER_CLUSTER_IP>:6443
+  members:
+    - clusterID: test-cluster-east
+  namespace: antrea-mcs-ns


luolanzone · 2021-12-13T01:05:05Z

multicluster/test/yamls/leader-access-token-secret.yml

+  namespace: antrea-mcs-ns
+  annotations:
+    kubernetes.io/service-account.name: antrea-mc-member-access-sa
+type: kubernetes.io/service-account-token


luolanzone · 2021-12-13T01:05:12Z

multicluster/test/yamls/test-west-serviceexport.yml

+kind: ServiceExport
+metadata:
+  name: west-nginx
+  namespace: antrea-multicluster-test


luolanzone · 2021-12-13T01:05:18Z

multicluster/test/yamls/west-member-cluster.yml

+      server: https://<LEADER_CLUSTER_IP>:6443
+  members:
+    - clusterID: test-cluster-west
+  namespace: antrea-mcs-ns


luolanzone · 2021-12-13T01:49:56Z

ci/jenkins/test.sh

+    kubectl get nodes -o wide --no-headers=true "${LEADER_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/antrea-mcs.tar jenkins@[${IP}]:${WORKDIR}/antrea-mcs.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@"${IP}" "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/antrea-mcs.tar" || true
+    done
+
+    kubectl get nodes -o wide --no-headers=true "${EAST_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/antrea-mcs.tar jenkins@[${IP}]:${WORKDIR}/antrea-mcs.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@${IP} "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/antrea-mcs.tar" || true
+    done
+
+    kubectl get nodes -o wide --no-headers=true "${WEST_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/antrea-mcs.tar jenkins@[${IP}]:${WORKDIR}/antrea-mcs.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@${IP} "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/antrea-mcs.tar" || true
+    done


luolanzone · 2021-12-13T01:50:07Z

ci/jenkins/test.sh

+    kubectl get nodes -o wide --no-headers=true "${LEADER_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/nginx.tar jenkins@["${IP}"]:"${WORKDIR}"/nginx.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@"${IP}" "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/nginx.tar" || true
+    done
+
+    kubectl get nodes -o wide --no-headers=true "${EAST_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/nginx.tar jenkins@["${IP}"]:"${WORKDIR}"/nginx.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@"${IP}" "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/nginx.tar" || true
+    done
+
+    kubectl get nodes -o wide --no-headers=true "${WEST_CLUSTER_CONFIG}"| awk '{print $6}' | while read IP; do
+        rsync -avr --progress --inplace -e "ssh -o StrictHostKeyChecking=no" "${WORKDIR}"/nginx.tar jenkins@["${IP}"]:"${WORKDIR}"/nginx.tar
+        ssh -o StrictHostKeyChecking=no -n jenkins@"${IP}" "${CLEAN_STALE_IMAGES}; docker load -i ${WORKDIR}/nginx.tar" || true
+    done


luolanzone · 2021-12-13T01:52:14Z

ci/jenkins/test.sh

+    kubectl apply -f ./multicluster/build/yamls/antrea-multicluster-member-only.yml "${EAST_CLUSTER_CONFIG}"
+    kubectl rollout status deployment/antrea-mc-controller -n kube-system "${EAST_CLUSTER_CONFIG}"
+    kubectl apply -f ./multicluster/test/yamls/leader-access-token.yml "${EAST_CLUSTER_CONFIG}"
+    kubectl apply -f ./multicluster/test/yamls/east-member-cluster.yml "${EAST_CLUSTER_CONFIG}"
+
+    kubectl apply -f ./multicluster/build/yamls/antrea-multicluster-member-only.yml "${WEST_CLUSTER_CONFIG}"
+    kubectl rollout status deployment/antrea-mc-controller -n kube-system "${WEST_CLUSTER_CONFIG}"
+    kubectl apply -f ./multicluster/test/yamls/leader-access-token.yml "${WEST_CLUSTER_CONFIG}"
+    kubectl apply -f ./multicluster/test/yamls/west-member-cluster.yml "${WEST_CLUSTER_CONFIG}"


replace them by a for?

luolanzone · 2021-12-13T01:54:11Z

ci/jenkins/test.sh

+}
+
+function cleanup_multicluster_antrea {
+    echo "====== Cleanup Multicluster Antrea controller and agent  ======"


I suppose this line should be echo "====== Cleanup Antrea controller and agent ======"?

luolanzone · 2021-12-13T01:57:51Z

ci/jenkins/test.sh

@@ -41,7 +45,10 @@ CONFORMANCE_SKIP="\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]|\[
 NETWORKPOLICY_SKIP="should allow egress access to server in CIDR block|should enforce except clause while egress access to server in CIDR block"

 # TODO: change to "control-plane" when testbeds are updated to K8s v1.20
-CONTROL_PLANE_NODE_ROLE="master"
+CONTROL_PLANE_NODE_ROLE="control-plane,master"


will this change impact existing e2e?

Dyanngg · 2021-12-13T05:36:53Z

ci/jenkins/test.sh

+    kubectl get svc -n "${ns}" --no-headers=true ${kubeconfig}| awk '{print $1}' | while read svc_name; do
+        kubectl delete svc "${svc_name}" -n "${ns}" --force --grace-period 0 ${kubeconfig} --timeout=30s
+    done
+    kubectl delete ns "${ns}" --ignore-not-found=true ${kubeconfig} --timeout=30s || true


nit: pretty sure you could simply delete the Namespace here but deleting the Pods and Services first are fine too

multicluster/test/e2e/service_test.go

Dyanngg · 2021-12-13T06:49:49Z

multicluster/test/e2e/service_test.go

+
+	svc, err := data.getService(eastCluster, multiClusterTestNamespace, fmt.Sprintf("antrea-mc-%s", westClusterTestService))
+	if err != nil {
+		t.Fatalf("Error when getting the import service: %v", err)


s/import/imported/g, also printing out the Service name might be helpful here

luolanzone · 2021-12-13T08:35:16Z

@hjiajing please check the github build failure, you need to add sign-off information to the commit and PR, please refer here https://github.com/antrea-io/antrea/blob/main/CONTRIBUTING.md#sign-off-your-work.
and there is also a golangci failure. https://github.com/antrea-io/antrea/runs/4495889994?check_suite_focus=true
integration test is also failed, https://jenkins.antrea-ci.rocks/job/antrea-auto-integration-for-pull-request/116/console, please check if it's caused by the code change or not. thanks.

Dyanngg · 2021-12-13T18:28:16Z

multicluster/test/e2e/framework.go

@@ -0,0 +1,388 @@
+package e2e


Is there any plan to unify this file with antrea.io/test/e2e/framework in the future? I understand that it will be quite some work because for multi-cluster case we need clients for more than one cluster. However, having a entirely new framework and TestData struct means that for each resource (e.g. Service and ANP, CG in the future) we need to redefine CRUD functions for those, which is a lot of code duplication

Dyanngg · 2021-12-13T18:28:30Z

multicluster/test/e2e/service_test.go

+	corev1 "k8s.io/api/core/v1"
+)
+
+func TestConnectivity(t *testing.T) {


Where is this used?

multicluster/test/e2e/framework.go

luolanzone · 2021-12-15T03:44:11Z

ci/jenkins/test.sh

@@ -677,6 +892,10 @@ if [[ ${TESTCASE} =~ "windows" ]]; then
 elif [[ ${TESTCASE} =~ "e2e" ]]; then
    deliver_antrea
    run_e2e
+elif [[ ${TESTCASE} =~ "multicluster" ]]; then


you need to add a new line here like below:

/test-multicluster-e2e: Multi-cluster e2e tests

We should use the same name prefix for e2e and integration tests, not one with "test-multicluster-", another with "test-mc-"

antoninbas · 2021-12-16T01:16:43Z

It makes sense to me to have a separate framework for "regular" e2e tests and MC e2e tests. I hope that long term the 2 frameworks can share some common library functions if they are going to be co-existing in the same repository (e.g. utility functions to create K8s resources, etc.).

antoninbas · 2021-12-16T01:17:19Z

ci/jenkins/test.sh

@@ -32,6 +32,10 @@ GO_VERSION=$(head -n1 "${WORKSPACE}/build/images/deps/go-version")
 IMAGE_PULL_POLICY="Always"


Is it possible to write a new script for MC tests instead of modifying this one?

okay , I will try to refactor the script.

luolanzone · 2021-12-22T00:43:40Z

CONTRIBUTING.md

@@ -165,6 +165,7 @@ Here are the trigger phrases for individual checks:
 * `/test-ipv6-only-conformance`: Linux IPv6 only conformance tests
 * `/test-ipv6-only-networkpolicy`: Linux IPv6 only networkpolicy tests
 * `/test-flexible-ipam-e2e`: Flexible IPAM e2e tests
+* `/test-multi-cluster-e2e`: Linux IPv4 multi-cluster e2e tests


Hi @hjiajing Could you help to update /test-multi-cluster-e2e to /test-multicluster-e2e? so we can make them consistent with integration test codes, thanks.

Thanks for comment, I will update it.

luolanzone · 2021-12-22T01:15:12Z

ci/jenkins/test-mc.sh

@@ -0,0 +1,300 @@
+#!/usr/bin/env bash
+
+# Copyright 2020 Antrea Authors


s/2020/2021

luolanzone · 2021-12-22T01:16:59Z

ci/jenkins/test-mc.sh

+
+        --kubeconfigs-path            Path of cluster set kubeconfigs.
+        --workdir                     Home path for Go, vSphere information and antrea_logs during cluster setup. Default is $WORKDIR.
+        --testcase                    Antrea multi-cluster e2e testcases on a Linux clusterset.


s/clusterset/cluster set/

ci/jenkins/test-mc.sh

luolanzone · 2021-12-22T01:24:19Z

ci/jenkins/test-mc.sh

+    done
+}
+
+function deliver_multicuster_controller {


s/multicuster/multicluster

luolanzone · 2021-12-22T01:25:16Z

ci/jenkins/test-mc.sh

+
+if [[ ${TEST_FAILURE} == true ]]; then
+    exit 1
+fi


missing newline

multicluster/test/e2e/fixtures.go

luolanzone · 2021-12-22T02:04:49Z

ci/jenkins/test.sh

@@ -684,4 +684,4 @@ fi

 if [[ ${TEST_FAILURE} == true ]]; then
    exit 1
-fi
+fi


missing new line

luolanzone · 2021-12-22T02:09:54Z

multicluster/test/yamls/clusterset.yml

+
+

remove unnecessary empty line

luolanzone · 2021-12-22T02:10:02Z

multicluster/test/yamls/east-member-cluster.yml

+
+

ditto, please check all related yml file with this issue. thanks.

hi @hjiajing have you updated the codes? the commits of this PR doesn't look right. and the comment is still not addressed. please help to double check your change.

ok, I will check it right now. Thanks for your comment.

Dyanngg · 2022-01-04T18:34:27Z

@hjiajing Please sign off your commit as well https://github.com/antrea-io/antrea/pull/3124/checks?check_run_id=4700295107

luolanzone · 2022-01-06T01:35:53Z

ci/jenkins/test-mc.sh

+
+        --kubeconfigs-path            Path of cluster set kubeconfigs.
+        --workdir                     Home path for Go, vSphere information and antrea_logs during cluster setup. Default is $WORKDIR.
+        --testcase                    Antrea multi-cluster e2e testcases on a Linux cluster set.


s/testcases/test cases/
I notice there is also the same typo in test.sh, Could you help to update it as well? thanks.

luolanzone · 2022-01-06T01:50:21Z

multicluster/test/e2e/service_test.go

+	data.testServiceExport(t)
+}
+
+//


I didn't see this is addressed.

hjiajing · 2022-01-06T07:58:31Z

/test-multicluster-e2e

hjiajing · 2022-01-10T02:06:17Z

/test-multicluster-e2e

hjiajing · 2022-01-10T02:51:16Z

/test-multicluster-e2e

hjiajing · 2022-01-10T03:03:50Z

/test-multicluster-e2e

hjiajing · 2022-01-10T04:38:07Z

/test-multicluster-e2e

jianjuns · 2022-01-10T19:26:09Z