snapshot tests are failing on master

[msau42]

https://k8s-testgrid.appspot.com/sig-storage-csi#alpha-canary-on-master

Latest failing run:
https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-csi-alpha-canary-on-kubernetes-master/1125363941975789568

[msau42]

/kind bug
/assign @xing-yang

[xing-yang]

I see the following failure in the logs. So this is because the rbac used by the tests don't have permissions to update PVC object.

May 6 11:58:20.966: INFO: At 2019-05-06 11:53:20 +0000 UTC - event for snapshot-694nl: {csi-snapshotter csi-hostpath } SnapshotCreationFailed: Failed to create snapshot: snapshot controller failed to update pvc-8fjdj on API server: persistentvolumeclaims "pvc-8fjdj" is forbidden: User "system:serviceaccount:default:csi-snapshotter" cannot update resource "persistentvolumeclaims" in API group "" in the namespace "snapshotting-4340"

The rbac has been updated in both e2e/testing-manifests and external-snapshotter repo. The one in external-snapshotter is used by csi-driver-host-path in e2e testing. So it looks like this failed test uses rbac file from a different location.

https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/storage-csi/external-snapshotter/rbac.yaml#L28
https://github.com/kubernetes-csi/external-snapshotter/blob/master/deploy/kubernetes/rbac.yaml#L28
https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/deploy/util/deploy-hostpath.sh#L80

[xing-yang]

I see the following in the logs:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.1.0/deploy/kubernetes/rbac.yaml

So it reads rbac.yaml from v1.1.0. The change that requires an update PVC rbac permission is merged after the v1.1.0 release.

[xing-yang]

Should I backport the rbac.yaml change to v1.1.0 branch?

[msau42]

@pohly since this is a canary job, should it be using rbacs from master?

[pohly]

That depends. One aspect of the "canary" jobs is to verify that the new images are still compatible with existing deployments, i.e. also with the RBAC rules used by those deployments. If that test fails, then we know that the next release has a breaking change and thus needs a major version bump.

The deployment controls the RBAC rules, but there is already a knob for overriding that if needed. I've added a new commit to kubernetes/test-infra#12482 which changes that knob for the "on-master" jobs.

Note that this will then also change the RBAC rules for the "canary-on-master" job, not just "alpha-canary-on-master".

[msau42]

Alpha features can break so I think that is fine.

If a stable sidecar required a new rbac rule for an existing feature, then the canary-on-1.13/1.14 jobs would break