k3d: Introduce k3d SR-IOV provider

[oshoval]

What this PR does / why we need it:
Since kubernetes-sigs/kind#2999 blocks us from updating
to newer k8s versions using kind (because we use cpu manager),
we are introducing k3d.

Current k8s version: v1.25.6+k3s1 (K3D_TAG=v5.4.7)

Changes:

• Support of local multi instances was removed,
  we are not using it, and it shouldn't affect multi instances on CI,
  once we want to introduce it.
• Added gracefully releasing of the SR-IOV nics.
  It reduces the downtime between cluster-down and cluster-up nicely,
  as the nics disappear for few minutes otherwise.
• Only one PF per node is supported, we don't need more for now.
• Use the k3d local registry instead one of our own.
• The provider supports only the following configuration:
  1 server (control-plane node) and 2 agents (workers).
  If we will need other configuration it can be done on follow PR,
  for now there is no reason to support other config.
• Few cosmetical refactoring.

Notes for reviewers:
In order to ease reviews, the first commit is a pure copy of the current kind folders.
The rest are the changes themselves.
Main files are provider.sh and common.sh, maybe better to look on them as a whole
instead of the diff because there are a lot of changes for them.

Project infra PR:

• kubevirtci, k3d-sriov: Add presubmit job project-infra#2633

Tested on kubevirt itself as well:

• Kubevirt - e2e tests: Support k3d SR-IOV lane kubevirt#9326
• Kubevirt Infra - k3d-sriov: Replace kind with k3d for SR-IOV jobs project-infra#2634

Action items on follow PRs:

• Use in memory etcd
• Add CI proxy support
• Support Podman on CI (works locally atm)
• We might need to mirror k3s image to quay.

Will be done according severity.

Potential nice to have follow PRs:

• Use cluster.yaml for the cluster creation
• Preprovision the image to reduce local setup time and potential pull flakes.
• Gracefully detach the nics from CI job / update CNI DEL command to do it.
• Reduce prints during cluster-up (present verbose/debug mode maybe).

Can be extracted to an issue if needed.

[kubevirt-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[oshoval]

Basic sanity checks passed (docker only)
but there is lots of work still

https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_project-infra/2633/rehearsal-check-up-k3d-1.25-sriov/1630148954723192832

[enp0s3]

/uncc
@oshoval Feel free to CC me if needed

[oshoval]

/uncc @oshoval Feel free to CC me if needed

Thanks, note that uncc will continue to nag emails, you can unsubscribe instead and that would help
reducing the noise.
Sorry for the noise, even if i uncced you it wouldn't stop the emails IIRC.
@enp0s3

[oshoval]

Added info about bumping moving parts
once someone actually bump, he can update those docs in case needed.

[EdDev]

Thank you!

It will be nice as a follow up to drop the use of Calico, as we do not really need it.

[oshoval]

Thank you!

It will be nice as a follow up to drop the use of Calico, as we do not really need it.

It is bit problematic, there is an internal flannel, that even if you want flannel, it need to be disabled first and deployed manually (with some changes possibly), because we are using multus and the internal's settings doesn't go well with our multus.
It is doable but will need deeper investigation on how to deploy flannel such that it will work with multus.
k8snetworkplumbingwg/multus-cni#564
k8snetworkplumbingwg/multus-cni#549
Didnt work for me out of the box yet, so i used the suggested calico and it worked great, at least for now
Maybe with a fresher look it will be easier in the future.

EDIT - managed to advance, not sure if all perfect yet though.

[oshoval]

Thank you!

It will be nice as a follow up to drop the use of Calico, as we do not really need it.

Update:
With this 5ab5de3
all the tests are passing
but need to mirror the images first and i want to validate few things
we can do it on a follow PR
Note that it isn't faster, it is lighter, but for example if we will want to use k3d for development of more non SR-IOV tests,
calico might be better isn't it?

[EdDev]

With this 5ab5de3
all the tests are passing
but need to mirror the images first and i want to validate few things
we can do it on a follow PR
Note that it isn't faster, it is lighter, but for example if we will want to use k3d for development of more non SR-IOV tests,
calico might be better isn't it?

I do not know why Callico will be better.
What feature of it are we using that requires it?
My interest of dropping it is to depend on less components and having a simple setup.

[oshoval]

I do not know why Callico will be better. What feature of it are we using that requires it? My interest of dropping it is to depend on less components and having a simple setup.

dual stack ? single stack ipv6 ?
not sure, maybe flannel also has it
we must use flannel instead if so and install it manually because of the issues mentioned above
so it is 1:1 unless you prefer flannel, in case calico doesnt have interesting features that may help us
for developing locally etc in the future, anyhow we can always switch if needed

Cons of flannel:
We will need to mirror the images to quay, it seems calico already has mirrors at least for some versions.
I wonder if flannel will include fixes in the future in case needed as fast as calico, it seems calico adapts faster ?
(but i might be wrong here)

[qinqon]

First pass, in general can we move the sriov things to cluster-up/cluster/k3d/sriov.sh ?

[qinqon]

/lgtm
/approve

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qinqon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [qinqon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment