Add snapshot webhook build and deployment. Modify controller to label invalid objects. #353

AndiLi99 · 2020-08-12T14:26:06Z

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test

/kind feature

/kind flake

What this PR does / why we need it:
Add a webhook which can be used to perform stricter validation, and to do ratcheting validation as part of the release plan.
Please see this KEP for full context on why a webhook is needed.

Add webhook build, docker, and deployment scripts. Boilerplate for adding the snapshot validation endpoint is in place.
Which issue(s) this PR fixes:

Fixes #187
Fixes #363

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

The validation for volume snapshot objects (VolumeSnapshot and VolumeSnapshotContent) is getting more strict. Due to backwards compatibility this change will occur over multiple releases. The following key changes are highlighted.

1. As part of the first phase of the multi-phased release process, a validating webhook server has been added. This server will perform additional validation (strict) which was not done during the beta release of volume snapshots. It will prevent the cluster from gaining (via create or update) invalid objects.
2. The controller will label objects which fail the additional validation.

The combination of 1 and 2 will allow cluster admins to stop the increase of invalid objects, and provide a way to easily list all objects which currently fail the strict validation. It is the cluster admin's responsibility to install the webhook and to ensure all the invalid objects in the cluster have been deleted or fixed. See the KEP at https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md

k8s-ci-robot · 2020-08-12T14:26:14Z

Welcome @AndiLi99!

It looks like this is your first PR to kubernetes-csi/external-snapshotter 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/external-snapshotter has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2020-08-12T14:26:14Z

Hi @AndiLi99. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yuxiangqian · 2020-08-12T17:10:04Z

cc @xing-yang @msau42

msau42 · 2020-08-12T17:16:37Z

/ok-to-test

yuxiangqian

preliminary comments

Makefile

cmd/webhook/main.go

yuxiangqian · 2020-08-12T22:03:10Z

deploy/kubernetes/webhook-example/README.md

@@ -0,0 +1,67 @@
+# How to deploy the webhook
+
+The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.


should include a prerequisites/dependencies section in this README to include:

k8s version

admission controller is enabled etc

We should also add a brief description of the webhook and add a link to this doc in the top level README: https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md

I added the prereq section. I also added a brief description to top level readme with a link. Please take a look.

deploy/kubernetes/webhook-example/README.md

yuxiangqian · 2020-08-12T22:19:10Z

deploy/kubernetes/webhook-example/README.md

+Patch the `ValidatingWebhookConfiguration` file from the template, filling in the CA bundle field.
+s
+```bash
+cat ./deploy/kubernetes/webhook-example/admission-configuration-template | ./deploy/kubernetes/webhook-example/patch-ca-bundle.sh > ./deploy/kubernetes/webhook-example/admission-configuration.yaml


maybe do not include the repo path?

yuxiangqian · 2020-08-12T22:32:47Z

cmd/webhook/Dockerfile

@@ -0,0 +1,7 @@
+FROM gcr.io/distroless/base:latest
+LABEL maintainers="Kubernetes Authors"
+LABEL description="Snapshot Webhook"


lets call it VolumeSnapshot, ditto everywhere

I'm following the naming convention from the Dockerfile for csi-snapshotter and snapshot-controller. Do you want to change both of those as well, or have an inconsistent naming pattern?

pkg/webhook/main.go

pkg/common-controller/snapshot_controller.go

pkg/utils/util.go

pkg/validation-webhook/convert.go

oleksiys · 2020-08-18T22:56:14Z

pkg/validation-webhook/convert.go

+func convertAdmissionResponseToV1beta1(r *v1.AdmissionResponse) *v1beta1.AdmissionResponse {
+	var pt *v1beta1.PatchType
+	if r.PatchType != nil {
+		t := v1beta1.PatchType(*r.PatchType)


Is it always cast'able?

I'm not sure

I believe it should work, as they're both just strings underneath

pkg/validation-webhook/snapshot.go

yuxiangqian

WIP. First cut.

pkg/common-controller/snapshot_controller.go

AndiLi99 · 2020-08-19T16:54:03Z

pkg/common-controller/snapshot_controller.go

@@ -1285,7 +1302,7 @@ func (ctrl *csiSnapshotCommonController) addSnapshotFinalizer(snapshot *crdv1.Vo
 	}
 	_, err := ctrl.clientset.SnapshotV1beta1().VolumeSnapshots(snapshotClone.Namespace).Update(context.TODO(), snapshotClone, metav1.UpdateOptions{})
 	if err != nil {
-		return newControllerUpdateError(snapshot.Name, err.Error())
+		return newControllerUpdateError(utils.SnapshotKey(snapshot), err.Error())


it looked like a typo, so i changed it

pkg/common-controller/snapshot_controller.go

pkg/utils/util.go

pkg/validation-webhook/main.go

pkg/validation-webhook/snapshot.go

xing-yang · 2020-08-20T03:08:57Z

Can you add a release note?

xing-yang · 2020-08-20T03:12:28Z

deploy/kubernetes/webhook-example/README.md

@@ -0,0 +1,67 @@
+# How to deploy the webhook
+
+The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.


We should also add a brief description of the webhook and add a link to this doc in the top level README: https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md

go.mod

pkg/common-controller/snapshot_controller.go

pkg/utils/util.go

pkg/validation-webhook/snapshot.go

yuxiangqian · 2020-08-20T04:55:46Z

@AndiLi99 some of the unit testings are failing because the newly added label for invalid CRs. Those unit test cases need to be fixed.

pkg/common-controller/snapshot_controller.go

pkg/validation-webhook/snapshot.go

yuxiangqian · 2020-08-24T00:24:58Z

@AndiLi99 please fix existing invalid tests which are failing due to newly added invalid labels.

AndiLi99 · 2020-08-25T15:03:25Z

/retest

yuxiangqian

couple of more nits.
@xing-yang @msau42 can you two take a look as well?

yuxiangqian · 2020-08-25T18:42:29Z

deploy/kubernetes/webhook-example/README.md

+Build the docker image
+
+```bash
+docker build -t gcr.io/your-project-name/webhook:latest -f ./cmd/webhook/Dockerfile .


@xing-yang @msau42 which gcr.io project should this repo use for webhook images?

should it follow the similar pattern here?

Official released images will be like this: k8s.gcr.io/sig-storage/csi-snapshotter:v1.2.2
However, you are showing an example on how to build the image yourself so it won't be the official image. I'd suggest not to specify the image repo name.

docker build -t webhook:latest -f ./cmd/webhook/Dockerfile

yuxiangqian · 2020-08-25T18:57:12Z

deploy/kubernetes/webhook-example/webhook.yaml

+    spec:
+      containers:
+      - name: snapshot-validation
+        image: gcr.io/your-project-name/validation-webhook:latest # change the image if you wish to use your own custom validation server image


please fix the image once my previous comments w.r.t image location is resolved.
example here

Since we are releasing this together with snapshot-controller and csi-snapshotter, the version will be v2.2.0.

k8s.gcr.io/sig-storage/validation-webhook:v2.2.0

pkg/common-controller/snapshot_controller.go

yuxiangqian · 2020-08-25T19:22:42Z

pkg/validation-webhook/config.go

+	}
+	return &tls.Config{
+		Certificates: []tls.Certificate{sCert},
+		// TODO: uses mutual tls after we agree on what cert the apiserver should use.


is there action item for this TODO?

No, this came from the example webhook. I believe it's for the case where the the webhook server verifies the identity of the api server.

yuxiangqian · 2020-08-25T19:29:57Z

pkg/validation-webhook/snapshot.go

+		}
+		return decideSnapshotContent(snapcontent, oldSnapcontent, isCreate)
+	default:
+		err := fmt.Errorf("expect resource to be %s or %s", SnapshotV1Beta1GVR, SnapshotContentV1Beta1GVR)


It's an interesting situation here. If a resource other than Snapshot/SnapshotContent has been mis-configured using this webhook for validation, it will fail all the time which might break existing functionality. Should the error here be ignored rather than populated to API server?

Should the error here be ignored rather than populated to API server?

No, we should break fast. If they misconfigure they should know.

yuxiangqian · 2020-08-25T19:32:30Z

pkg/validation-webhook/snapshot.go

+	// Which allows the remover of finalizers and therefore deletion of this object
+	// Don't rely on the pointers to be nil, because the deserialization method will convert it to
+	// The empty struct value. Instead check the operation type.
+	err := utils.ValidateSnapshot(oldSnapshot)


nit, can be simplified as:
if err := utils.ValidateSnapshot; err != nil { return ...}
this way, you can move line 95 out of the "if isCreate" block and save the call at line 113.
WDYT?

i cleaned it up differently, ptal

pkg/validation-webhook/snapshot.go

yuxiangqian · 2020-08-25T19:34:03Z

pkg/validation-webhook/snapshot.go

+	return *s
+}
+func checkSnapshotImmutableFields(snapshot, oldSnapshot *volumesnapshotv1beta1.VolumeSnapshot) error {
+	if snapshot == nil {


nil checks are no longer needed

I wouldn't want to make any assumptions about the caller. The ptr is never nil now, but that may change (future tests, deserialization change). Do you recommend changing it anyways?

yuxiangqian · 2020-08-25T19:34:22Z

pkg/validation-webhook/snapshot.go

+}
+
+func checkSnapshotContentImmutableFields(snapcontent, oldSnapcontent *volumesnapshotv1beta1.VolumeSnapshotContent) error {
+	if snapcontent == nil {


nil checks are no longer needed.

pkg/validation-webhook/webhook.go

yuxiangqian · 2020-08-25T22:55:40Z

@AndiLi99 please remove [WIP] from the PR title

deploy/kubernetes/webhook-example/README.md

xing-yang · 2020-08-26T04:23:11Z

deploy/kubernetes/webhook-example/webhook.yaml

+kind: Deployment
+metadata:
+  name: snapshot-validation-deployment
+  namespace: default # NOTE: change the namespace


Should this namespace be the same as the namespace of the snapshot-controller?

It doesn't need to be, but it may be easier to organize that way.

xing-yang · 2020-08-26T04:26:27Z

deploy/kubernetes/webhook-example/webhook.yaml

+    spec:
+      containers:
+      - name: snapshot-validation
+        image: gcr.io/your-project-name/validation-webhook:latest # change the image if you wish to use your own custom validation server image


Since we are releasing this together with snapshot-controller and csi-snapshotter, the version will be v2.2.0.

k8s.gcr.io/sig-storage/validation-webhook:v2.2.0

AndiLi99 · 2020-08-26T15:40:41Z

I think this is at a good state. Please take a look @yuxiangqian @msau42 @xing-yang

https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md 1. Ratcheting validation webhook server image 2. Controller labels invalid objects 3. Unit tests for webhook 4. Deployment README and example deployment method with certs 5. Update top-level README Racheting validation: 1. webhook is strict on create 2. webhook is strict on updates where the existing object passes strict validation 3. webhook is relaxed on updates where the existing object fails strict validation (allows finalizer removal, status update, deletion, etc) Additionally the validating wehook server will perform immutability checks on scenario 2 above.

README.md

pkg/validation-webhook/webhook.go

xing-yang · 2020-08-28T03:08:50Z

/retest

Minor cleanup and change default fail policy and timeout on webhook config.

msau42 · 2020-08-28T20:19:54Z

README.md

@@ -79,6 +81,22 @@ Install CSI Driver:
  * kubectl create -f deploy/kubernetes/csi-snapshotter
  * https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/csi-snapshotter

+### Validating Webhook


@xing-yang @yuxiangqian let's look at this before we release, but I think it would be good to revamp the overview to make it more clear of all the pieces in this repo and who is responsible for what:

CRDs + webhook

controller

sidecar

client libraries

msau42 · 2020-08-28T20:25:54Z

This lgtm! Will let @xing-yang and @yuxiangqian give final approvals.

Awesome work Andi!

AndiLi99 · 2020-08-28T21:37:59Z

@msau42 Thank you so much! Everyone's help was instrumental in getting this done :)

xing-yang

/lgtm

Thanks Andi!

xing-yang · 2020-08-28T22:28:32Z

README.md

@@ -79,6 +81,22 @@ Install CSI Driver:
  * kubectl create -f deploy/kubernetes/csi-snapshotter
  * https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/csi-snapshotter

+### Validating Webhook


yuxiangqian · 2020-08-29T01:49:57Z

/lgtm
/approve
GJ @AndiLi99

k8s-ci-robot · 2020-08-29T01:50:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AndiLi99, yuxiangqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuxiangqian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 12, 2020

k8s-ci-robot requested review from jingxu97 and lpabon August 12, 2020 14:26

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 12, 2020

AndiLi99 force-pushed the AndiLi99/webhook branch 6 times, most recently from 4f1e8c8 to 55cd2a5 Compare August 12, 2020 15:32

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 12, 2020

yuxiangqian reviewed Aug 12, 2020

View reviewed changes

AndiLi99 force-pushed the AndiLi99/webhook branch from 3582266 to 47ece7d Compare August 13, 2020 17:38

oleksiys reviewed Aug 18, 2020

View reviewed changes

yuxiangqian reviewed Aug 19, 2020

View reviewed changes

xing-yang mentioned this pull request Aug 20, 2020

VolumeSnapshot cannot be deleted when the VolumeSnapshotClass is missing #306

Closed

xing-yang reviewed Aug 20, 2020

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 20, 2020

yuxiangqian reviewed Aug 20, 2020

View reviewed changes

yuxiangqian reviewed Aug 25, 2020

View reviewed changes

AndiLi99 changed the title ~~[WIP] Add boiler plate for snapshot webhook build and deployment~~ Add boiler plate for snapshot webhook build and deployment Aug 25, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 25, 2020

xing-yang reviewed Aug 26, 2020

View reviewed changes

deploy/kubernetes/webhook-example/README.md Show resolved Hide resolved

xing-yang reviewed Aug 26, 2020

View reviewed changes

AndiLi99 changed the title ~~Add boiler plate for snapshot webhook build and deployment~~ Add snapshot webhook build and deployment. Modify controller to label invalid objects. Aug 26, 2020

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2020

AndiLi99 force-pushed the AndiLi99/webhook branch from bd4aefc to 42b6b37 Compare August 28, 2020 00:46

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 28, 2020

msau42 reviewed Aug 28, 2020

View reviewed changes

README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

pkg/validation-webhook/webhook.go Show resolved Hide resolved

yuxiangqian mentioned this pull request Aug 28, 2020

Update volume snapshot CRDs to v1 version #367

Merged

Andi Li added 2 commits August 28, 2020 12:06

Update documentation.

06989d7

Minor cleanup and change default fail policy and timeout on webhook config.

Documentation style match new line after bullet point.

8e8ab7b

msau42 reviewed Aug 28, 2020

View reviewed changes

xing-yang reviewed Aug 28, 2020

View reviewed changes

k8s-ci-robot assigned xing-yang Aug 28, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 28, 2020

k8s-ci-robot assigned yuxiangqian Aug 29, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 29, 2020

k8s-ci-robot merged commit 4f3b02a into kubernetes-csi:master Aug 29, 2020

		@@ -0,0 +1,67 @@
		# How to deploy the webhook

		The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.

Add snapshot webhook build and deployment. Modify controller to label invalid objects. #353

Add snapshot webhook build and deployment. Modify controller to label invalid objects. #353

Conversation

AndiLi99 commented Aug 12, 2020 • edited Loading

k8s-ci-robot commented Aug 12, 2020

k8s-ci-robot commented Aug 12, 2020

yuxiangqian commented Aug 12, 2020

msau42 commented Aug 12, 2020

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

xing-yang commented Aug 20, 2020

Choose a reason for hiding this comment

yuxiangqian commented Aug 20, 2020

yuxiangqian commented Aug 24, 2020

AndiLi99 commented Aug 25, 2020

yuxiangqian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian commented Aug 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndiLi99 commented Aug 26, 2020

xing-yang commented Aug 28, 2020

msau42 Aug 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Aug 28, 2020

AndiLi99 commented Aug 28, 2020

xing-yang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuxiangqian commented Aug 29, 2020

k8s-ci-robot commented Aug 29, 2020

AndiLi99 commented Aug 12, 2020 •

edited

Loading

msau42 Aug 28, 2020 •

edited

Loading