Add functional test for system versions of container engines

[afbjorklund]

(This is issue is about KIC on Linux)

Currently we are testing with the vendor versions of the container engines:

• docker.com
  • https://docs.docker.com/engine/install/
• podman.io
  • https://podman.io/getting-started/installation

But maybe we also need to test with the system versions, in the distributions:

• debian
  • https://packages.debian.org/docker.io
  • https://packages.debian.org/podman
• ubuntu
  • https://packages.ubuntu.com/docker.io
  • https://packages.ubuntu.com/podman
• fedora
  • https://src.fedoraproject.org/rpms/moby-engine (Provides: docker)
  • https://src.fedoraproject.org/rpms/podman
• centos
  • docker (el7: 1.13)
  • podman (el8: 2.0)

Or at least add some documentation, as to which "variants" are supported.

Currently users expect to be able to install the packages from anywhere...

---

Note that this issue is about the container engine to be used for the node:

• Docker
• Podman

It's not about the container runtime to be used inside the container node:

• Docker (dockershim/cri-dockerd)
• CRI-O
• containerd

For those we still only support the vendor versions, as per Kubernetes upstream.
There are system versions in some other dedicated Kubernetes operating systems.
But we don't use those, so it is up to each Kubernetes distribution to support them.
Example is our own custom Buildroot distribution for the ISO, with system packages.

[afbjorklund]

Otherwise we risk seeing distribution-specific issues like #10088

Could also be a good idea to document supported versions: #9077

[afbjorklund]

Using vagrant for regression/release testing is a great idea.

Unfortunately it only works for real servers, not nested VMs.

[afbjorklund]

Partly related to #3552, for adding support for fedora/centos.

Also need some good way to aggregate/present the results ?

[afbjorklund]

I started on some test infrastructure, for generic: #4733

Should clean it up and make it general... #9545 (comment)

Ubuntu 16.04
Ubuntu 18.04
Ubuntu 20.04
CentOS 7
CentOS 8
Fedora 32
Debian 9
Debian 10

One Vagrantfile for each environment, multiplied by above.

[medyagh]

do u mind adding more clarificaiton, I am a bit confused, if you are talking about the inside minikube podman engine?

[afbjorklund]

do u mind adding more clarificaiton, I am a bit confused, if you are talking about the inside minikube podman engine?

I tried to clarify:

Note that this issue is about the container engine to be used for the node.

It's not about the container runtime to be used inside the container node.

When using docker-in-docker, it can easilly get confusing which is "docker" and which is "docker". 🙂

We control the inside, and there we only run the vendor variant. Not whatever is in the distribution.
This is the same as kubernetes upstream* (kubeadm), except we don't have the same version lock.

* https://kubernetes.io/docs/setup/production-environment/container-runtimes/

Except for the "none" driver, but there are really no guarantees if you try to use something else...
Ultimately we should probably always provision the runtime - also for the none and generic drivers.

So this issue was all about the outside, if we should support the distribution versions as well ?

[afbjorklund]

Main reason to support the standard versions, would be to make it simpler for the user.

sudo apt install docker.io

vs

$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io

It also makes it easier to access the source code, and to integrate with other packages.

That is important to some people, especially Debian users. They don't want the binaries.

---

More details: https://www.collabora.com/news-and-blog/blog/2018/07/04/docker-io-debian-package-back-to-life/

The story for podman is more or less the same, even though it is from another distribution rather than a company...

[medyagh]

this is a good idea ! for the starting point we should at least add ubuntu 20.04 and 18.04 test to our matrix. @afbjorklund

For a starting point we could add 3 new functional test on Gihtub Action. for 3 different versions of ubuntu.

https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#:~:text=GitHub%20offers%20runners%20with%20Linux,each%20job%20in%20a%20workflow.

[afbjorklund]

Maybe weigh the test matrix, against what minikube users are actually using ? Like after doing a poll, or opt-in registration

[ilya-zuyev]

We need to add functional tests for older versions of ubuntu and included docker versions

[afbjorklund]

Actually this issue doesn't have so much to do with the OS versions that minikube is doing the testing with.

It was about testing with the system package of the container runtimes, as well as the vendor packages ?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[medyagh]

I dont know if we have ifnra bandwith to add more tests infra just for this. if Github Actions would provide older OS or Debian we could add functioan on github action if not we could close this issue for now till that infra is available for us