Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubernetes-csi: test 1.14 deployment periodically #12482

Merged
merged 3 commits into from
May 8, 2019
Merged

kubernetes-csi: test 1.14 deployment periodically #12482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2a81c00
kubernetes-csi: configure new base image in one place
pohly May 6, 2019
c0e3277
kubernetes-csi: test 1.14 deployment periodically
pohly May 6, 2019
6122799
kubernetes-csi: deploy latest RBAC rules for *-on-master jobs
pohly May 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 154 additions & 6 deletions config/jobs/kubernetes-csi/csi-driver-host-path/csi-driver-host-path-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,111 @@ periodics:
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-1-14-on-kubernetes-1-13
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure if this configuration is valid. A driver deployment for 1.14 could have enabled features that will only work on Kubernetes 1.14 (like the new Leases leader election type).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, it may or may not be compatible. But we know that it will work for the 1.14 deployment, therefore this combination is valid and worth testing. For example, https://kubernetes-csi.github.io/docs/external-attacher.html says that the minimum Kubernetes version is 1.13. This job covers that promise.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we add leader election, topology or any of the CSI driver features, then it will no longer be the case.

I'm thinking in the future if we want to better automate setting up these jobs, then it's simpler to have rules that work across all scenarios

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I consider the kubernetes-1.14 deployment mostly frozen at this point. It enables a certain set of features and those happen to be compatible with 1.13 (when ignoring alpha). IMHO it is useful to keep it this way, to increase test coverage of the stable images also with Kubernetes 1.13.

When we added leader election, I would do it in a separate deployment because we need cover both scenarios (with and without leader election). That new deployment then cannot run on 1.13.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having said that, if you don't care about testing this particular aspect, then I won't argue further in favor of keeping it. It's not a combination that I personally use.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mainly don't want this to be misinterpreted as a general statement that we support running these sidecars against 1.13 in all scenarios. It's very dependent per driver and what features they use.

If we eventually add these 1.14+ features to the hostpath-driver, our deployment management and version skew testing will become very complex if we want to be able to test with all theoretical combinations of new/old features and versions.

decorate: true
extra_refs:
- org: kubernetes-csi
repo: csi-driver-host-path
base_ref: master
labels:
preset-service-account: "true"
preset-dind-enabled: "true"
preset-bazel-remote-cache-enabled: "false"
preset-kind-volume-mounts: "true"
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190420-93fab49-master
command:
- runner.sh
args:
- ./.prow.sh
env:
- name: CSI_PROW_KUBERNETES_VERSION
value: "release-1.13"
- name: CSI_PROW_BUILD_JOB
value: "false"
- name: CSI_PROW_DEPLOYMENT
value: "kubernetes-1.14"
- name: CSI_PROW_TESTS
value: "sanity serial parallel"
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-1-14-on-kubernetes-1-14
decorate: true
extra_refs:
- org: kubernetes-csi
repo: csi-driver-host-path
base_ref: master
labels:
preset-service-account: "true"
preset-dind-enabled: "true"
preset-bazel-remote-cache-enabled: "false"
preset-kind-volume-mounts: "true"
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190420-93fab49-master
command:
- runner.sh
args:
- ./.prow.sh
env:
- name: CSI_PROW_KUBERNETES_VERSION
value: "release-1.14"
- name: CSI_PROW_BUILD_JOB
value: "false"
- name: CSI_PROW_DEPLOYMENT
value: "kubernetes-1.14"
- name: CSI_PROW_TESTS
value: "sanity serial parallel"
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-1-14-on-kubernetes-master
decorate: true
extra_refs:
- org: kubernetes-csi
repo: csi-driver-host-path
base_ref: master
labels:
preset-service-account: "true"
preset-dind-enabled: "true"
preset-bazel-remote-cache-enabled: "true"
preset-kind-volume-mounts: "true"
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190420-93fab49-master
command:
- runner.sh
args:
- ./.prow.sh
env:
- name: CSI_PROW_KUBERNETES_VERSION
value: "latest"
- name: CSI_PROW_BUILD_JOB
value: "false"
- name: CSI_PROW_DEPLOYMENT
value: "kubernetes-1.14"
- name: CSI_PROW_TESTS
value: "sanity serial parallel"
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-alpha-1-13-on-kubernetes-1-13
decorate: true
Expand Down Expand Up @@ -390,6 +495,41 @@ periodics:
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-alpha-1-14-on-kubernetes-1-14
decorate: true
extra_refs:
- org: kubernetes-csi
repo: csi-driver-host-path
base_ref: master
labels:
preset-service-account: "true"
preset-dind-enabled: "true"
preset-bazel-remote-cache-enabled: "false"
preset-kind-volume-mounts: "true"
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190420-93fab49-master
command:
- runner.sh
args:
- ./.prow.sh
env:
- name: CSI_PROW_KUBERNETES_VERSION
value: "release-1.14"
- name: CSI_PROW_BUILD_JOB
value: "false"
- name: CSI_PROW_DEPLOYMENT
value: "kubernetes-1.14"
- name: CSI_PROW_TESTS
value: "serial-alpha parallel-alpha"
# docker-in-docker needs privileged mode
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
- interval: 6h
name: ci-kubernetes-csi-canary-on-kubernetes-1-13
decorate: true
Expand Down Expand Up @@ -418,7 +558,9 @@ periodics:
# Replace images....
- name: CSI_PROW_HOSTPATH_CANARY
value: "canary"
# ... but not the RBAC rules.
# ... but the RBAC rules only when testing on master.
# The other jobs test against the unmodified deployment for
# that Kubernetes version, i.e. with the original RBAC rules.
- name: UPDATE_RBAC_RULES
value: "false"
- name: CSI_PROW_TESTS
Expand Down Expand Up @@ -457,7 +599,9 @@ periodics:
# Replace images....
- name: CSI_PROW_HOSTPATH_CANARY
value: "canary"
# ... but not the RBAC rules.
# ... but the RBAC rules only when testing on master.
# The other jobs test against the unmodified deployment for
# that Kubernetes version, i.e. with the original RBAC rules.
- name: UPDATE_RBAC_RULES
value: "false"
- name: CSI_PROW_TESTS
Expand Down Expand Up @@ -496,9 +640,11 @@ periodics:
# Replace images....
- name: CSI_PROW_HOSTPATH_CANARY
value: "canary"
# ... but not the RBAC rules.
# ... but the RBAC rules only when testing on master.
# The other jobs test against the unmodified deployment for
# that Kubernetes version, i.e. with the original RBAC rules.
- name: UPDATE_RBAC_RULES
value: "false"
value: "true"
- name: CSI_PROW_TESTS
value: "sanity serial parallel"
# docker-in-docker needs privileged mode
Expand Down Expand Up @@ -535,9 +681,11 @@ periodics:
# Replace images....
- name: CSI_PROW_HOSTPATH_CANARY
value: "canary"
# ... but not the RBAC rules.
# ... but the RBAC rules only when testing on master.
# The other jobs test against the unmodified deployment for
# that Kubernetes version, i.e. with the original RBAC rules.
- name: UPDATE_RBAC_RULES
value: "false"
value: "true"
- name: CSI_PROW_TESTS
value: "serial-alpha parallel-alpha"
# docker-in-docker needs privileged mode
Expand Down
26 changes: 15 additions & 11 deletions config/jobs/kubernetes-csi/gen-jobs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@

base="$(dirname $0)"

# We need this image because it has Docker in Docker and go.
dind_image="gcr.io/k8s-testimages/kubekins-e2e:v20190420-93fab49-master"

# All kubernetes-csi repos which are part of the hostpath driver example.
# For these repos we generate the full test matrix.
hostpath_example_repos="
Expand Down Expand Up @@ -192,7 +195,7 @@ EOF
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -238,7 +241,7 @@ EOF
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -270,7 +273,7 @@ EOF
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -308,7 +311,7 @@ EOF
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -347,7 +350,7 @@ EOF
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -377,9 +380,8 @@ cat >>"$base/csi-driver-host-path/csi-driver-host-path-config.yaml" <<EOF
periodics:
EOF

# TODO: decide how we want to test the kubernetes-1.14 deployment
for tests in non-alpha alpha; do
for deployment in 1.13; do
for deployment in 1.13 1.14; do
for kubernetes in 1.13 1.14 master; do
# No version skew testing of alpha features, deployment has to match Kubernetes.
if [ "$tests" = "alpha" ] && ! echo "$kubernetes" | grep -q "^$deployment"; then
Expand All @@ -402,7 +404,7 @@ for tests in non-alpha alpha; do
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand Down Expand Up @@ -451,7 +453,7 @@ for kubernetes in 1.13.3 1.14.0 master; do
spec:
containers:
# We need this image because it has Docker in Docker and go.
- image: gcr.io/k8s-testimages/kubekins-e2e:v20190329-811f7954b-master
- image: ${dind_image}
command:
- runner.sh
args:
Expand All @@ -464,9 +466,11 @@ for kubernetes in 1.13.3 1.14.0 master; do
# Replace images....
- name: CSI_PROW_HOSTPATH_CANARY
value: "canary"
# ... but not the RBAC rules.
# ... but the RBAC rules only when testing on master.
# The other jobs test against the unmodified deployment for
# that Kubernetes version, i.e. with the original RBAC rules.
- name: UPDATE_RBAC_RULES
value: "false"
value: "$([ "$kubernetes" = "master" ] && echo "true" || echo "false")"
- name: CSI_PROW_TESTS
value: "$(expand_tests "$tests")"
# docker-in-docker needs privileged mode
Expand Down
20 changes: 20 additions & 0 deletions testgrid/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3693,8 +3693,16 @@ test_groups:
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-1-13-on-kubernetes-1-14
- name: ci-kubernetes-csi-1-13-on-kubernetes-master
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-1-13-on-kubernetes-master
- name: ci-kubernetes-csi-1-14-on-kubernetes-1-13
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-1-14-on-kubernetes-1-13
- name: ci-kubernetes-csi-1-14-on-kubernetes-1-14
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-1-14-on-kubernetes-1-14
- name: ci-kubernetes-csi-1-14-on-kubernetes-master
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-1-14-on-kubernetes-master
- name: ci-kubernetes-csi-alpha-1-13-on-kubernetes-1-13
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-alpha-1-13-on-kubernetes-1-13
- name: ci-kubernetes-csi-alpha-1-14-on-kubernetes-1-14
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-alpha-1-14-on-kubernetes-1-14
- name: ci-kubernetes-csi-canary-on-kubernetes-1-13
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-csi-canary-on-kubernetes-1-13
- name: ci-kubernetes-csi-canary-on-kubernetes-1-14
Expand Down Expand Up @@ -7250,9 +7258,21 @@ dashboards:
- name: 1-13-on-master
test_group_name: ci-kubernetes-csi-1-13-on-kubernetes-master
description: Kubernetes-CSI non-alpha tests with Kubernetes master and 1.13 sidecars
- name: 1-14-on-1-13
test_group_name: ci-kubernetes-csi-1-14-on-kubernetes-1-13
description: Kubernetes-CSI non-alpha tests with Kubernetes 1.13.x and 1.14 sidecars
- name: 1-14-on-1-14
test_group_name: ci-kubernetes-csi-1-14-on-kubernetes-1-14
description: Kubernetes-CSI non-alpha tests with Kubernetes 1.14.x and 1.14 sidecars
- name: 1-14-on-master
test_group_name: ci-kubernetes-csi-1-14-on-kubernetes-master
description: Kubernetes-CSI non-alpha tests with Kubernetes master and 1.14 sidecars
- name: alpha-1-13-on-1-13
test_group_name: ci-kubernetes-csi-alpha-1-13-on-kubernetes-1-13
description: Kubernetes-CSI alpha tests with Kubernetes 1.13.x and 1.13 sidecars
- name: alpha-1-14-on-1-14
test_group_name: ci-kubernetes-csi-alpha-1-14-on-kubernetes-1-14
description: Kubernetes-CSI alpha tests with Kubernetes 1.14.x and 1.14 sidecars
- name: canary-on-1-13
test_group_name: ci-kubernetes-csi-canary-on-kubernetes-1-13
description: Kubernetes-CSI non-alpha tests with Kubernetes 1.13.x and 1.14 sidecars
Expand Down