multus - mixed type of container engine support commit fails

[blackluck]

Environment:

• Cloud provider or hardware configuration:
  Private cloud on vmware

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 5.4.0-125-generic x86_64
  NAME="Ubuntu"
  VERSION="20.04.5 LTS (Focal Fossa)"

• Version of Ansible (ansible --version): ansible==5.7.1 , ansible-core==2.12.5 . It's an awx execution environment created with ansible-builder 3.0, base image: centos:stream9

• Version of Python (python --version): Python 3.9.16

Kubespray version (commit) (git rev-parse --short HEAD): release-2.21

Network plugin used: none, cni is installed manually without kubespray, 'kube_network_plugin: cni'

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"): This gonna take too much and it's not important

Command used to invoke ansible: cluster upgrade with limited to some workers, run from awx

Output of ansible run:

TASK [kubernetes-apps/network_plugin/multus : Multus | Start resources] ********
failed: [k8swafsb001 -> {{ groups['kube_control_plane'][0] }}] (item=None) => {"ansible_loop_var": "item", "changed": false, "item": null, "msg": "Failed to template loop_control.label: 'None' has no attribute 'item'", "skip_reason": "Conditional result was False"}

Anything else do we need to know:

Tried to upgrade our k8s cluster (last time upgraded with release-2.20) with release-2.21 kubespray but got above multus error.
But we don't use multus or any cni from kubespray 'kube_network_plugin: cni' and still doesn't skip this [kubernetes-apps/network_plugin/multus : Multus | Start resources] task.
Maybe relevant extra info, we upgrade cluster nodes in separate batches, first upgrade all masters (that worked), then tried to upgrade only some workers (get facts on all node, then upgrade-cluster with --limit someworkers), because on a bigger cluster this is more convinient and docs says it's also an acceptable way of upgrade.

In the issues saw #10105 , tried to update release-2.21 with that commit but got the same issue.
Then as others mentioned reverted #9224 and after that upgrade worked. So this commit still needs some fixes, until that this probably will affect newer versions too.

[hadi2f244]

I commented this part in the following file and everything works great.

roles/kubernetes-apps/network_plugin/meta/main.yml
roles/network_plugin/meta/main.yml

  # - role: kubernetes-apps/network_plugin/multus
  #   when: kube_network_plugin_multus
  #   tags:
  #     - multus

[hadi2f244]

It seems to be resolved by this change (v2.23):
[multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)

[nicolas-goudry]

@blackluck Could you please confirm this is fixed with v2.23 so we can close this issue ? Thanks

[VannTen]

Let's go ahead and close it, it can be reopened as needed
/close

[k8s-ci-robot]

@VannTen: Closing this issue.

In response to this:

Let's go ahead and close it, it can be reopened as needed
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.