test flake: updating labels in default deployment while running – fedora-1_19.olm-operator.API

[pohly]

From https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/blue/organizations/jenkins/pmem-csi/detail/PR-738/2/tests:

/mnt/workspace/pmem-csi_PR-738/test/e2e/operator/deployment_api.go:461
Sep 21 10:17:28.182: validate driver after update and restart
Unexpected error:
<*errors.errorString | 0xc003051a70>: {
s: "timed out waiting for deployment, last error: deployed driver different from expected deployment:\nlabel foo missing for \"pmem-csi-with-defaults-external-provisioner-cfg\" of type \"rbac.authorization.k8s.io/v1, Kind=Role\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-external-provisioner-runner\" of type \"rbac.authorization.k8s.io/v1, Kind=ClusterRole\" in namespace \"\"\nlabel foo missing for \"pmem-csi-with-defaults-csi-provisioner-role-cfg\" of type \"rbac.authorization.k8s.io/v1, Kind=RoleBinding\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-csi-provisioner-role\" of type \"rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding\" in namespace \"\"\nlabel foo missing for \"pmem-csi-with-defaults-controller\" of type \"/v1, Kind=ServiceAccount\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-controller\" of type \"/v1, Kind=Service\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-metrics\" of type \"/v1, Kind=Service\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults\" of type \"storage.k8s.io/v1beta1, Kind=CSIDriver\" in namespace \"\"",
}
timed out waiting for deployment, last error: deployed driver different from expected deployment:
label foo missing for "pmem-csi-with-defaults-external-provisioner-cfg" of type "rbac.authorization.k8s.io/v1, Kind=Role" in namespace "default"
label foo missing for "pmem-csi-with-defaults-external-provisioner-runner" of type "rbac.authorization.k8s.io/v1, Kind=ClusterRole" in namespace ""
label foo missing for "pmem-csi-with-defaults-csi-provisioner-role-cfg" of type "rbac.authorization.k8s.io/v1, Kind=RoleBinding" in namespace "default"
label foo missing for "pmem-csi-with-defaults-csi-provisioner-role" of type "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" in namespace ""
label foo missing for "pmem-csi-with-defaults-controller" of type "/v1, Kind=ServiceAccount" in namespace "default"
label foo missing for "pmem-csi-with-defaults-controller" of type "/v1, Kind=Service" in namespace "default"
label foo missing for "pmem-csi-with-defaults-metrics" of type "/v1, Kind=Service" in namespace "default"
label foo missing for "pmem-csi-with-defaults" of type "storage.k8s.io/v1beta1, Kind=CSIDriver" in namespace ""
occurred
/mnt/workspace/pmem-csi_PR-738/test/e2e/operator/deployment_api.go:458

[pohly]

@avalluri : can you have a look at this? Is the timeout perhaps simply too short?

[avalluri]

I doubt if this failure is with timeout. For some reason the chagnes the operator detected for that deployment is different:

pmem-csi-operator-fbd749dcb-7j4vw/[email protected]: I0921 10:12:33.820351 1 controller_driver.go:97] Deployment: "pmem-csi-with-defaults", state "Running", changes map[deviceMode:{} imagePullPolicy:{} logLevel:{} controllerResources:{} nodeResources:{} nodeSelector:{} pmemPercentage:{} kubeletDir:{}], in cache true

I suspect this is result of some other previous test failures.

[avalluri]

Okey, there is no other test failures, something tricky.

[pohly]

And another one:
https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/blue/organizations/jenkins/pmem-csi/detail/version-skew-2/22/pipeline/93

[Fail] olm-operator API updating provisionerImage in deployment with specific values [It] while running 

[pohly]

And one more: https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/job/pmem-csi/view/change-requests/job/PR-723/33/testReport/junit/fedora-1_19.olm-operator/API/Testing___1_19___updating_controllerResources_in_deployment_with_specific_values_while_running/

[pohly]

In #723 (comment), @avalluri wrote:

We could revert 1a433b7. I see that is causing the operator to detect default values as changes.

[pohly]

While all of the examples above are from PR #723, that's simply because I've been working with that PR the most. It also occurs in "devel":
https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/job/pmem-csi/job/devel/242/execution/node/110/log/?consoleFull

02:29:11.403  Sep 22 13:58:27.783: FAIL: validate driver after update and restart
02:29:11.403  Unexpected error:
02:29:11.403      <*errors.errorString | 0xc003a7be80>: {
02:29:11.403          s: "timed out waiting for deployment, last error: deployed driver different from expected deployment:\nlabel foo missing for \"pmem-csi-with-defaults-external-provisioner-cfg\" of type \"rbac.authorization.k8s.io/v1, Kind=Role\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-external-provisioner-runner\" of type \"rbac.authorization.k8s.io/v1, Kind=ClusterRole\" in namespace \"\"\nlabel foo missing for \"pmem-csi-with-defaults-csi-provisioner-role-cfg\" of type \"rbac.authorization.k8s.io/v1, Kind=RoleBinding\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-csi-provisioner-role\" of type \"rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding\" in namespace \"\"\nlabel foo missing for \"pmem-csi-with-defaults-controller\" of type \"/v1, Kind=ServiceAccount\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-controller\" of type \"/v1, Kind=Service\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults-metrics\" of type \"/v1, Kind=Service\" in namespace \"default\"\nlabel foo missing for \"pmem-csi-with-defaults\" of type \"storage.k8s.io/v1beta1, Kind=CSIDriver\" in namespace \"\"",
02:29:11.403      }
02:29:11.403      timed out waiting for deployment, last error: deployed driver different from expected deployment:
02:29:11.403      label foo missing for "pmem-csi-with-defaults-external-provisioner-cfg" of type "rbac.authorization.k8s.io/v1, Kind=Role" in namespace "default"
02:29:11.403      label foo missing for "pmem-csi-with-defaults-external-provisioner-runner" of type "rbac.authorization.k8s.io/v1, Kind=ClusterRole" in namespace ""
02:29:11.403      label foo missing for "pmem-csi-with-defaults-csi-provisioner-role-cfg" of type "rbac.authorization.k8s.io/v1, Kind=RoleBinding" in namespace "default"
02:29:11.403      label foo missing for "pmem-csi-with-defaults-csi-provisioner-role" of type "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" in namespace ""
02:29:11.403      label foo missing for "pmem-csi-with-defaults-controller" of type "/v1, Kind=ServiceAccount" in namespace "default"
02:29:11.403      label foo missing for "pmem-csi-with-defaults-controller" of type "/v1, Kind=Service" in namespace "default"
02:29:11.403      label foo missing for "pmem-csi-with-defaults-metrics" of type "/v1, Kind=Service" in namespace "default"
02:29:11.403      label foo missing for "pmem-csi-with-defaults" of type "storage.k8s.io/v1beta1, Kind=CSIDriver" in namespace ""
02:29:11.403  occurred

[pohly]

We could revert 1a433b7. I see that is causing the operator to detect default values as changes.

Why is this flaky?

If you think that you have a fix, then please prepare a PR. But it needs to have a good explanation of why the revised code is correct and why the old one wasn't.

[avalluri]

I submitted the fix #743 for the deployment comparison issue. The issue was with the recent change of reverting deployment spec changes. That also resets from the cached copy of that deployment and in next reconcile loop comparing the cached deployment against the updated deployment with set defults, shows all those defaults as changes.

[pohly]

Not seen anymore. We don't know why it's gone now, but let's close the issue.