Skip to content

Commit

Permalink
Merge branch 'improvement/add-cncf-comformance-test-post-merge' into …
Browse files Browse the repository at this point in the history
…q/123.0
  • Loading branch information
bert-e committed Jul 13, 2022
2 parents 59a8b2c + 5d1b67c commit 39cce08
Show file tree
Hide file tree
Showing 2 changed files with 271 additions and 6 deletions.
146 changes: 146 additions & 0 deletions eve/build-conformance-pr-content.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
#!/bin/bash

DIRECTORY=${DIRECTORY:-pr-content}
SONOBUOY_RES_DIR=${SONOBUOY_RES_DIR:-sonobuoy-results}

K8S_VERSION=${K8S_VERSION:-}
METALK8S_VERSION=${METALK8S_VERSION:-}

dest="${DIRECTORY}/v${K8S_VERSION}/MetalK8s"
doc_url="https://metal-k8s.readthedocs.io/en/development-${METALK8S_VERSION}/"

mkdir -p "$dest"

cat > "$dest/PRODUCT.yaml" << EOF
vendor: Scality
name: MetalK8s
description: "An opinionated Kubernetes distribution with a focus on long-term on-prem deployments"
version: ${METALK8S_VERSION}
type: distribution
website_url: https://github.com/scality/metalk8s/
repo_url: https://github.com/scality/metalk8s.git
product_logo_url: https://raw.githubusercontent.com/scality/metalk8s/development/${METALK8S_VERSION}/artwork/metalk8s-logo-vertical.svg
documentation_url: ${doc_url}
EOF

sed "s%@@DOC_URL@@%${doc_url}%g" > "$dest/README.md" << 'ENDREADME'
# MetalK8s
Official documentation: @@DOC_URL@@
## Prerequisites
- An OpenStack cluster
- The official CentOS 7.9 2009 image pre-loaded in Glance
- Three VMs with 8 vCPUs, 16 GB of RAM, 40GB of local storage
## Provisioning
- Create two private network in the OpenStack cluster with port security
disabled, and a subnet in each:
* Control-plane network: 192.168.1.0/24
* Workload-plane network: 192.168.2.0/24
- Create VM instances using the CentOS 7.9 image, and attach each of them to a
public network (for internet access) and the two private networks.
- Configure the interface for private networks (make sure to fill in the
appropriate MAC address):
```
$ cat > /etc/sysconfig/network-scripts/ifcfg-eth1 << EOF
BOOTPROTO=dhcp
DEVICE=eth1
HWADDR=...
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PEERDNS=no
EOF
$ cat > /etc/sysconfig/network-scripts/ifcfg-eth2 << EOF
BOOTPROTO=dhcp
DEVICE=eth2
HWADDR=...
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
PEERDNS=no
EOF
$ systemctl restart network
```
### Provisioning the Bootstrap Node
On one of the VMs, which will act as the *bootstrap* node, perform the following
steps:
- Set up the Salt Minion ID:
```
$ mkdir /etc/salt; chmod 0700 /etc/salt
$ echo metalk8s-bootstrap > /etc/salt/minion_id
```
- Download MetalK8s ISO to `/home/centos/metalk8s.iso`
- Create `/etc/metalk8s/bootstrap.yaml`:
```
$ mkdir /etc/metalk8s
$ cat > /etc/metalk8s/bootstrap.yaml << EOF
apiVersion: metalk8s.scality.com/v1alpha3
kind: BootstrapConfiguration
networks:
controlPlane:
cidr: 192.168.1.0/24
workloadPlane:
cidr: 192.168.2.0/24
portmap:
cidr: 0.0.0.0/0
nodeport:
cidr: 0.0.0.0/0
ca:
minion: metalk8s-bootstrap
archives:
- /home/centos/metalk8s.iso
EOF
```
- Bootstrap the cluster
```
$ mkdir /mnt/metalk8s
$ mount /home/centos/metalk8s.iso /mnt/metalk8s
$ cd /mnt/metalk8s
$ ./bootstrap.sh
```
### Provisioning the Cluster Nodes
Add the 2 other nodes to the cluster according to the procedure outlined in the
MetalK8s documentation. The easiest way to achieve this is through the MetalK8s
UI.
## Preparing the Cluster to Run Sonobuoy
On the *bootstrap* node:
- Configure access to the Kubernetes API server
```
$ export KUBECONFIG=/etc/kubernetes/admin.conf
```
- Remove taints from the node, which would prevent the Sonobuoy *Pod*s from
being scheduled:
```
$ kubectl taint node metalk8s-bootstrap node-role.kubernetes.io/bootstrap-
node/metalk8s-bootstrap untainted
$ kubectl taint node metalk8s-bootstrap node-role.kubernetes.io/infra-
node/metalk8s-bootstrap untainted
```
## Running Sonobuoy and Collecting Results
Follow the
[instructions](https://github.com/cncf/k8s-conformance/blob/master/instructions.md)
as found in the [CNCF K8s Conformance repository](https://github.com/cncf/k8s-conformance).
ENDREADME

cp "${SONOBUOY_RES_DIR}/plugins/e2e/results/global/e2e.log" "$dest/"
cp "${SONOBUOY_RES_DIR}/plugins/e2e/results/global/junit_01.xml" "$dest/"
131 changes: 125 additions & 6 deletions eve/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -904,6 +904,7 @@ stages:
- bootstrap-restore
- single-node-install-rocky-8
- single-node-no-idp
- k8s-conformance-test
haltOnFailure: true
- TriggerStages:
name: Trigger publish stage
Expand Down Expand Up @@ -1796,7 +1797,7 @@ stages:
# FIXME: We disable offline mode for RHEL as it does not work for now,
# there are some certificates issues when trying to reach RedHat
# repositories through the company proxy.
- SetProperty:
- SetProperty: &disable_offline
name: Set offline property to false
property: offline
value: "false"
Expand Down Expand Up @@ -2352,7 +2353,7 @@ stages:
multiple-nodes:
_metalk8s_internal_info:
junit_info: &_install_multi-node_junit_info
TEST_SUITE: install
TEST_SUITE: "%(prop:run_k8s_conformance:?|conformance|install)s"
CLASS_NAME: "multi node.%(prop:os:-centos-7)s"
TEST_NAME: "1 bootstrap %(prop:nodes_count:-1)s master,etcd"
simultaneous_builds: 20
Expand All @@ -2370,7 +2371,7 @@ stages:
env:
<<: *_env_final_status_artifact_failed
<<: *_install_multi-node_junit_info
STEP_NAME: "multiple-nodes-%(prop:os:-centos-7)s"
STEP_NAME: "%(prop:run_k8s_conformance:?|conformance-tests|multiple-nodes-%(prop:os:-centos-7)s)s"
- ShellCommand: *yum_clean_all
- ShellCommand: *ssh_ip_setup
- ShellCommand: *git_pull_terraform
Expand Down Expand Up @@ -2491,19 +2492,122 @@ stages:
PYTEST_ARGS: "--suppress-no-test-exit-code"
PYTEST_FILTERS: >
post and ci and slow and not bootstrap and not restore
# NOTE: This section only run if property "run_k8s_conformance" is True
# {{{
- SetPropertyFromCommand:
name: Save the Kubernetes version as a property
doStepIf: "%(prop:run_k8s_conformance:-false)s"
property: k8s_version
command: >
ssh -F ssh_config bootstrap sudo
rpm -q --queryformat '%{VERSION}' kubelet | cut -d'.' -f1,2
workdir: *terraform_workdir
haltOnFailure: true
- ShellCommand:
<<: *untaint_bootstrap_ssh
doStepIf: "%(prop:run_k8s_conformance:-false)s"
- ShellCommand:
name: Expose nodeport and portmap on every IPs
doStepIf: "%(prop:run_k8s_conformance:-false)s"
command: |
ssh -F ssh_config bootstrap <<ENDSSH
sudo salt-call --local --retcode-passthrough state.single file.serialize /etc/metalk8s/bootstrap.yaml dataset='{"networks": {"nodeport": {"cidr": "0.0.0.0/0"}, "portmap": {"cidr": "0.0.0.0/0"}}}' merge_if_exists=True
SALT_MASTER=\$(sudo crictl ps --label="io.kubernetes.container.name=salt-master" -q)
sudo crictl exec \$SALT_MASTER salt-run state.sls metalk8s.kubernetes.kube-proxy.deployed saltenv=metalk8s-%(prop:metalk8s_version)s
sudo crictl exec \$SALT_MASTER salt-run state.sls metalk8s.kubernetes.cni.calico.deployed saltenv=metalk8s-%(prop:metalk8s_version)s
sudo crictl exec \$SALT_MASTER salt '*' state.sls metalk8s.addons.nginx-ingress.certs saltenv=metalk8s-%(prop:metalk8s_version)s
ENDSSH
workdir: *terraform_workdir
haltOnFailure: true
- ShellCommand: *wait_pods_stable_ssh
- ShellCommand:
name: Get sonobuoy bin
doStepIf: "%(prop:run_k8s_conformance:-false)s"
env:
VERSION: "0.56.8"
command: |
curl -Lo "sonobuoy.tar.gz" https://github.com/vmware-tanzu/sonobuoy/releases/download/v${VERSION}/sonobuoy_${VERSION}_linux_amd64.tar.gz
tar xvf sonobuoy.tar.gz
haltOnFailure: true
- ShellCommand:
<<: *copy_iso_bootstrap_ssh
name: Copy sonobuoy bin on bootstrap
doStepIf: "%(prop:run_k8s_conformance:-false)s"
env:
<<: *_env_copy_iso_bootstrap_ssh
ARCHIVE: sonobuoy
- ShellCommand:
name: Run conformance test from bootstrap
doStepIf: "%(prop:run_k8s_conformance:-false)s"
command: >
ssh -F ssh_config bootstrap sudo
./sonobuoy run --kubeconfig=/etc/kubernetes/admin.conf --mode=certified-conformance --wait
workdir: *terraform_workdir
haltOnFailure: true
- ShellCommand:
name: Retrieve conformance test from bootstrap
doStepIf: "%(prop:run_k8s_conformance:-false)s"
command: >
ssh -F ssh_config bootstrap sudo
./sonobuoy retrieve --kubeconfig=/etc/kubernetes/admin.conf --filename sonobuoy_result.tar.gz
workdir: *terraform_workdir
haltOnFailure: true
- ShellCommand:
name: Retrieve conformance test result
doStepIf: "%(prop:run_k8s_conformance:-false)s"
env: &_env_conformance_test_dest
DEST_DIR: "%(prop:builddir)s/build/conformance/conformance/sonobuoy-results"
FILENAME: sonobuoy_result.tar.gz
command: |
mkdir -p "$DEST_DIR"
scp -F ssh_config bootstrap:"$FILENAME" "$DEST_DIR"
cd "$DEST_DIR"
tar xvf "$FILENAME"
workdir: *terraform_workdir
alwaysRun: true
- ShellCommand:
name: Check conformance test result
doStepIf: "%(prop:run_k8s_conformance:-false)s"
env: *_env_conformance_test_dest
command: |
failed_tests=$(./sonobuoy results "$DEST_DIR/$FILENAME" --mode=detailed --plugin=e2e | jq 'select(.status=="failed")')
[ -n "$failed_tests" ] && echo $failed_tests && exit 1 || exit 0
haltOnFailure: true
- ShellCommand:
name: Prepare conformance PR content
doStepIf: "%(prop:run_k8s_conformance:-false)s"
env:
DIRECTORY: "pr-content"
SONOBUOY_RES_DIR: sonobuoy-results
K8S_VERSION: "%(prop:k8s_version)s"
METALK8S_VERSION: "%(prop:metalk8s_short_version)s"
command: |
%(prop:builddir)s/build/eve/build-conformance-pr-content.sh
tar cvf pr-content.tar.gz -C "$DIRECTORY" "v$K8S_VERSION"
workdir: "%(prop:builddir)s/build/conformance/conformance"
haltOnFailure: true
- Upload:
name: Upload conformance test result
doStepIf: "%(prop:run_k8s_conformance:-false)s"
source: conformance
alwaysRun: true
urls:
- conformance/sonobuoy-results/sonobuoy_result.tar.gz
- conformance/pr-content.tar.gz
# }}}
- ShellCommand: *generate_report_over_ssh
- ShellCommand:
<<: *collect_report_over_ssh
env:
<<: *_env_collect_report_over_ssh
STEP_NAME: "multiple-nodes-%(prop:os:-centos-7)s"
STEP_NAME: "%(prop:run_k8s_conformance:?|conformance-tests|multiple-nodes-%(prop:os:-centos-7)s)s"
- Upload: *upload_report_artifacts
- ShellCommand:
<<: *add_final_status_artifact_success
env:
<<: *_env_final_status_artifact_success
<<: *_install_multi-node_junit_info
STEP_NAME: "multiple-nodes-%(prop:os:-centos-7)s"
STEP_NAME: "%(prop:run_k8s_conformance:?|conformance-tests|multiple-nodes-%(prop:os:-centos-7)s)s"
- Upload: *upload_final_status_artifact
# NOTE: This section only run if property "generate_snapshot" is True
# {{{
Expand All @@ -2513,7 +2617,7 @@ stages:
<<: *wait_debug
timeout: 14400
env:
STEP_NAME: "multiple-nodes-%(prop:os:-centos-7)s"
STEP_NAME: "%(prop:run_k8s_conformance:?|conformance-tests|multiple-nodes-%(prop:os:-centos-7)s)s"
DURATION: "14400"
- ShellCommand: *terraform_destroy

Expand Down Expand Up @@ -3092,3 +3196,18 @@ stages:
env:
<<: *_env_image_registry
IMAGE_NAME: shell-ui

k8s-conformance-test:
worker:
type: local
steps:
- SetProperty:
name: Enable Kubernetes conformance tests
property: run_k8s_conformance
value: "true"
- SetProperty: *prop_3nodes_count
- SetProperty: *disable_offline
- TriggerStages:
name: Trigger multiple-nodes step with conformance tests enabled
stage_names:
- multiple-nodes

0 comments on commit 39cce08

Please sign in to comment.