gVisor: initial support for gVisor container runtime #7661

cristicalin · 2021-05-29T19:00:23Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
This PR adds support to use gVisor as a container runtime. According to the related issue, docker is missing the capability to change the RuntimeClass, so only containerd and cri-o are viable option.

This PR add support for both containerd and cri-o to configure the gVisor runtime but I have yet to be able to get cri-o to successfully launch gVisor workloads and, as per the linked issue it seems gVisor is also not fully support it either as such the validation step only allows gVisor to be enabled with containerd.

Which issue(s) this PR fixes:

Fixes #7650

Special notes for your reviewer:
I implemented molecule testing for ubuntu 20.04 and centos 8, in order to cover centos 8 I also had to move some stuff around between the docker and containerd-common roles so I can reliably test with containerd on centos.

Does this PR introduce a user-facing change?:

gVisor: add initial support for gVisor container runtime

k8s-ci-robot · 2021-05-29T19:00:31Z

Hi @cristicalin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sathieu · 2021-05-31T17:06:06Z

roles/kubernetes/preinstall/tasks/0020-verify-settings.yml

+- name: Stop if gvisor_enabled is enabled when container_manager is not containerd
+  assert:
+    that: container_manager == 'containerd'
+    msg: "gvisor_enabled support only compatible with containerd. See https://github.com/kubernetes-sigs/kubespray/issues/7650 for details"
+  when: gvisor_enabled
+


The doc mentions cri-o too. Which one is correct?

I updated the docs. As per @ianlewis' comment, cri-o is not currently usable though it does support setting the RuntimeClass; because of this I restricted the validation to containerd. Once gVisor adds support for cri-o, we can change this validation and the code to support it is already in this PR.

Thanks @cristicalin for this PR!

oomichi · 2021-06-01T15:37:00Z

/ok-to-test

oomichi · 2021-06-08T22:53:03Z

roles/kubernetes/preinstall/tasks/0020-verify-settings.yml

@@ -293,6 +293,12 @@
    msg: "kata_containers_enabled support only for containerd and crio-o. See https://github.com/kata-containers/documentation/blob/1.11.4/how-to/run-kata-with-k8s.md#install-a-cri-implementation for details"
  when: kata_containers_enabled

+- name: Stop if gvisor_enabled is enabled when container_manager is not containerd
+  assert:
+    that: container_manager == 'containerd'


The additional lines of roles/container-engine/meta/main.yml

- role: container-engine/gvisor when: - gvisor_enabled - container_manager in ['docker', 'containerd']

seem docker also can be container_manager.
I guess I am missing some point.

@oomichi see the discussion on #7650, initially I wanted to support all 3 of our runtimes but it turns out only containerd is well supported by gVisor today. docker is actually not able to select the RuntimeClass and cri-o support is stil not mature enough, though I'm actually hoping that might evolve in gVisor itself.

ianlewis · 2021-06-01T09:34:03Z

roles/container-engine/cri-o/tasks/main.yaml

+- name: Build a list of crio runtimes with gVisor runtime
+  set_fact:
+    crio_runtimes: "{{ crio_runtimes + [gvisor_runtime] }}"
+  when:
+    - gvisor_enabled


I assume you wouldn't need this if you aren't going to support cri-o with runsc?

I left this part here because I'm actually hoping gVisor will eventually support cri-o when google/gvisor#193 is fixed.

Is there any intention to properly support cri-o in gVisor? If not then I will remove this part from the PR.

I guess I'd remove it for now. cri-o will need some work in order to better support runtimes that aren't vanilla containers. It's not just a matter of supporting systemd-cgroups.

Understood, in this case I dropped the support for cri-o until gVisor project decides to support it.

oomichi

The feature works fine for me, thanks for doing this.
/lgtm

Just a question: I'd like to know how to see OCI runtime (gvisor in this case) on kubectl. I can see containerd on kubectl describe node, but gvisor is not there.

oomichi · 2021-06-17T23:22:38Z

docs/gvisor.md

+## Usage
+
+To enable gVisor you should be using a container manager that is compatible with selecting the [RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) such as `containerd`.
+


nit: After testing this pull request locally, I see etcd_deployment_type needs to be host to pass Stop if etcd deployment type is not host when container_manager != docker.
This thing is already written on docs/containerd.md, so I don't have strong opinion to write it here.

ianlewis · 2021-06-21T02:51:30Z

The feature works fine for me, thanks for doing this.
/lgtm

Just a question: I'd like to know how to see OCI runtime (gvisor in this case) on kubectl. I can see containerd on kubectl describe node, but gvisor is not there.

You can see it if you print the whole yaml via kubectl get pod x -o yaml | grep runtimeClassName or you can print it out via custom columns: kubectl get pods -o=custom-columns='NAME:.metadata.name,RUNTIME CLASS:.spec.runtimeClassName,STATUS:.status.phase'

However, I assume by OCI runtime you mean the the CRI runtime handler, and needs to be viewed on the RuntimeClass object as the RuntimeClass name and handler name can be different. So you can get the RuntimeClass from one of the kubectl get pod commands above and then look up the handler on the RuntimeClass via kubectl get runtimeclass gvisor

floryut

Nice work with that, great to see this implemented in the codebase

k8s-ci-robot · 2021-06-21T12:16:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cristicalin, floryut

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

oomichi · 2021-06-21T16:51:06Z

The feature works fine for me, thanks for doing this.
/lgtm
Just a question: I'd like to know how to see OCI runtime (gvisor in this case) on kubectl. I can see containerd on kubectl describe node, but gvisor is not there.

You can see it if you print the whole yaml via kubectl get pod x -o yaml | grep runtimeClassName or you can print it out via custom columns: kubectl get pods -o=custom-columns='NAME:.metadata.name,RUNTIME CLASS:.spec.runtimeClassName,STATUS:.status.phase'

However, I assume by OCI runtime you mean the the CRI runtime handler, and needs to be viewed on the RuntimeClass object as the RuntimeClass name and handler name can be different. So you can get the RuntimeClass from one of the kubectl get pod commands above and then look up the handler on the RuntimeClass via kubectl get runtimeclass gvisor

Thanks for your help, @ianlewis :-)

sathieu · 2021-06-21T18:56:06Z

Thanks @cristicalin!

* master: Update kubernetes-reliability.md (kubernetes-sigs#7724) Fix deployment without openstack cacert (kubernetes-sigs#7723) Clean up residual files (kubernetes-sigs#7722) gVisor: initial support for gVisor container runtime (kubernetes-sigs#7661)

…#7661) * Docker/Containerd: move downloads urls to containerd-common * gVisor: initial support for gVisor container runtime

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 29, 2021

cristicalin marked this pull request as draft May 29, 2021 19:00

k8s-ci-robot requested review from EppO and holmsten May 29, 2021 19:00

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 29, 2021

cristicalin force-pushed the gvisor branch 8 times, most recently from 784b35d to 8169002 Compare May 31, 2021 14:57

cristicalin marked this pull request as ready for review May 31, 2021 14:59

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 31, 2021

cristicalin force-pushed the gvisor branch 2 times, most recently from a39a4c8 to e1e1690 Compare May 31, 2021 15:44

sathieu reviewed May 31, 2021

View reviewed changes

cristicalin force-pushed the gvisor branch from e1e1690 to b7a45f8 Compare May 31, 2021 17:14

cristicalin force-pushed the gvisor branch from b7a45f8 to 2f11de8 Compare June 2, 2021 16:28

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 2, 2021

cristicalin force-pushed the gvisor branch from 2f11de8 to d379da1 Compare June 3, 2021 18:34

oomichi reviewed Jun 8, 2021

View reviewed changes

ianlewis reviewed Jun 8, 2021

View reviewed changes

cristicalin force-pushed the gvisor branch from d379da1 to f0793a4 Compare June 9, 2021 08:00

cristicalin added 2 commits June 9, 2021 05:10

Docker/Containerd: move downloads urls to containerd-common

e59f0bc

gVisor: initial support for gVisor container runtime

7c3e766

cristicalin force-pushed the gvisor branch from f0793a4 to 7c3e766 Compare June 9, 2021 09:10

oomichi reviewed Jun 18, 2021

View reviewed changes

k8s-ci-robot assigned oomichi Jun 18, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 18, 2021

floryut approved these changes Jun 21, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 21, 2021

k8s-ci-robot merged commit 282a27a into kubernetes-sigs:master Jun 21, 2021

floryut mentioned this pull request Sep 8, 2021

Release Proposal v2.17 #7951

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gVisor: initial support for gVisor container runtime #7661

gVisor: initial support for gVisor container runtime #7661

cristicalin commented May 29, 2021 •

edited

Loading

k8s-ci-robot commented May 29, 2021

sathieu May 31, 2021

cristicalin May 31, 2021

sathieu May 31, 2021

oomichi commented Jun 1, 2021

oomichi Jun 8, 2021

cristicalin Jun 9, 2021

ianlewis Jun 1, 2021

cristicalin Jun 9, 2021

ianlewis Jun 9, 2021

cristicalin Jun 9, 2021

oomichi left a comment

oomichi Jun 17, 2021

ianlewis commented Jun 21, 2021 •

edited

Loading

floryut left a comment

k8s-ci-robot commented Jun 21, 2021

oomichi commented Jun 21, 2021

sathieu commented Jun 21, 2021

		## Usage

		To enable gVisor you should be using a container manager that is compatible with selecting the [RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) such as `containerd`.

gVisor: initial support for gVisor container runtime #7661

gVisor: initial support for gVisor container runtime #7661

Conversation

cristicalin commented May 29, 2021 • edited Loading

k8s-ci-robot commented May 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oomichi commented Jun 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oomichi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianlewis commented Jun 21, 2021 • edited Loading

floryut left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 21, 2021

oomichi commented Jun 21, 2021

sathieu commented Jun 21, 2021

cristicalin commented May 29, 2021 •

edited

Loading

ianlewis commented Jun 21, 2021 •

edited

Loading