Kubespray deployment fails on run growpart task when root partition is on logical volume

[belissner]

Environment:

• Cloud provider or hardware configuration:
  Virtual machines on Azure.

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 5.15.0-6.80.3.1.el9uek.x86_64 x86_64
  NAME="Oracle Linux Server"
  VERSION="9.1"
  ID="ol"
  ID_LIKE="fedora"
  VARIANT="Server"
  VARIANT_ID="server"
  VERSION_ID="9.1"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Oracle Linux Server 9.1"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:oracle:linux:9:1:server"
  HOME_URL="https://linux.oracle.com/"
  BUG_REPORT_URL="https://github.com/oracle/oracle-linux"
  ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9"
  ORACLE_BUGZILLA_PRODUCT_VERSION=9.1
  ORACLE_SUPPORT_PRODUCT="Oracle Linux"
  ORACLE_SUPPORT_PRODUCT_VERSION=9.1

• Version of Ansible (ansible --version):
  ansible [core 2.11.11]

• Version of Python (python --version):
  Python 3.6.8

Kubespray version (commit) (git rev-parse --short HEAD):
919e666

Network plugin used:
Calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

https://gist.github.com/belissner/e71990f51416310245e16ab868dca319#file-gistfile1-txt
Command used to invoke ansible:

Output of ansible run:

TASK [kubernetes/preinstall : run growpart] ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
fatal: [worker-2]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.005607", "end": "2023-02-01 16:44:39.690812", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:39.685205", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}
fatal: [master-1]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.010419", "end": "2023-02-01 16:44:39.678498", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:39.668079", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}
fatal: [worker-1]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.005641", "end": "2023-02-01 16:44:39.691971", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:39.686330", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}
fatal: [master-3]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.011215", "end": "2023-02-01 16:44:39.696315", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:39.685100", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}
fatal: [master-2]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.010183", "end": "2023-02-01 16:44:39.714299", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:39.704116", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}
fatal: [worker-3]: FAILED! => {"changed": true, "cmd": ["growpart", "/dev/mapper/rootvg-rootlv", "/dev/mapper/rootvg-rootlv"], "delta": "0:00:00.005699", "end": "2023-02-01 16:44:40.219349", "msg": "non-zero return code", "rc": 2, "start": "2023-02-01 16:44:40.213650", "stderr": "", "stderr_lines": [], "stdout": "FAILED: partition-number must be a number", "stdout_lines": ["FAILED: partition-number must be a number"]}

Anything else do we need to know:

If the root partition is on LV , the task run growpart fails in kubespray/roles/kubernetes/preinstall/tasks/0120-growpart-azure-centos-7.yml.
The check if growpart needs to be run task register the following:

ok: [worker-3] => {
    "growpart_needed": {
        "changed": true,
        "cmd": [
            "growpart",
            "-N",
            "/dev/mapper/rootvg-rootlv",
            "/dev/mapper/rootvg-rootlv"
        ],
        "delta": "0:00:00.005603",
        "end": "2023-02-01 16:44:36.019321",
        "failed": false,
        "failed_when_result": false,
        "msg": "non-zero return code",
        "rc": 2,
        "start": "2023-02-01 16:44:36.013718",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "FAILED: partition-number must be a number",
        "stdout_lines": [
            "FAILED: partition-number must be a number"
        ]
    }

The task run growpart will be triggerd, because the changed_when: "'NOCHANGE:' not in growpart_needed.stdout" statement in check if growpart needs to be run task will be true, but it will fail because the device and partition variables are /dev/mapper/rootvg-rootlv

[belissner]

The deployment was succesfull with this temporary workaround:

- name: check if growpart needs to be run
  command: growpart -N {{ device }} {{ partition }}
  failed_when: False
  changed_when:
    - "'NOCHANGE:' not in growpart_needed.stdout"
    - "'FAILED:' not in growpart_needed.stdout"
  register: growpart_needed
  environment:
    LC_ALL: C

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[pedro-peter]

I'm also seeing this issue running on RHEL8 based vm's on azure:

TASK [kubernetes/preinstall : run growpart] **************************************************************************************************************************************************************************
fatal: [myhost]: FAILED! => {
    "changed": true,
    "cmd": [
        "growpart",
        "/dev/mapper/rootvg-rootlv",
        "/dev/mapper/rootvg-rootlv"
    ],
    "delta": "0:00:00.008903",
    "end": "2023-04-27 05:26:39.535699",
    "rc": 2,
    "start": "2023-04-27 05:26:39.526796"
}

STDOUT:

FAILED: partition-number must be a number

From the comment:
https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubernetes/preinstall/tasks/0120-growpart-azure-centos-7.yml#L3

# Running growpart seems to be only required on Azure, as other Cloud Providers do this at boot time

I'm guessing newly created vm's in azure need their disk resized? We're using terraform & packer and do not need to resize the disks.

I suggest we add a variable to allow disabling the 0120-growpart-azure-centos-7.yml role/tasks for folks who manage their vm disks in azure outside kubespray.

I can create a PR for this.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.