Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [RHEL] Single machine upgrade may fail on preflight check: 'dict object' has no attribute 'size_available' #3029

Closed
8 of 18 tasks
przemyslavic opened this issue Mar 16, 2022 · 1 comment
Closed
8 of 18 tasks

[BUG] [RHEL] Single machine upgrade may fail on preflight check: 'dict object' has no attribute 'size_available' #3029

przemyslavic opened this issue Mar 16, 2022 · 1 comment
Assignees
@atsikham @przemyslavic
Labels
type/bug

Comments

@przemyslavic
Copy link
Collaborator

przemyslavic commented Mar 16, 2022 •
edited by seriva
Loading

Describe the bug
Single machine upgrade fails on TASK [preflight : Set facts on space on /dev/nvme0n1p2].
I was able to reproduce it only on RHEL.

AWS

2022-03-16T13:41:52.1413947Z[38;21m13:41:52 INFO cli.src.ansible.AnsibleCommand - TASK [preflight : Set facts on space on /dev/nvme0n1p2] ************************�[0m
2022-03-16T13:41:52.2245083Z[31;21m13:41:52 ERROR cli.src.ansible.AnsibleCommand - fatal: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'size_available'\n\nThe error appears to be in '/shared/build/13todevawrhflannel/ansible/roles/repository/tasks/preflight/upgrade.yml': line 13, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Set facts on space on {{ var_data_disk.stdout }}\n      ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}

Azure

2022-03-16T13:39:43.0280059Z[38;21m13:39:43 INFO cli.src.ansible.AnsibleCommand - TASK [preflight : Set facts on space on /dev/mapper/rootvg-rootlv] *************
2022-03-16T13:39:43.1081787Z[31;21m13:39:43 ERROR cli.src.ansible.AnsibleCommand - fatal: [ci-10todevazrhflannel-single-machine-vm-0]: FAILED! => {"msg": "The task includes an option with an undefined variable. 
The error was: 'dict object' has no attribute 'size_available'\n\nThe error appears to be in '/shared/build/10todevazrhflannel/ansible/roles/repository/tasks/preflight/upgrade.yml': line 13, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - name: Set facts on space on {{ var_data_disk.stdout }}\n      ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}

How to reproduce
Steps to reproduce the behavior:

  1. Deploy v1.0 or v1.3 RHEL cluster (single machine installation)
  2. Upgrade to 2.0.0dev

Expected behavior
Upgrade should succeed. Preflight checks should pass.

Environment

  • Cloud provider: [AWS | Azure ]
  • OS: [RHEL]

epicli version: upgrade to 2.0.0dev

Additional context
Add any other context about the problem here.

DoD checklist

  • Changelog
    • updated
    • not needed
  • COMPONENTS.md
    • updated
    • not needed
  • Schema
    • updated
    • not needed
  • Backport tasks
    • created
    • not needed
  • Documentation
    • added
    • updated
    • not needed
  • Feature has automated tests
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
    • backup/restore
  • Idempotency tested
  • All conversations in PR resolved
@przemyslavic
Copy link
Collaborator Author

przemyslavic commented Mar 17, 2022 •
edited
Loading

After debugging:

15:02:56 INFO cli.engine.ansible.AnsibleCommand - TASK [preflight : debug] *******************************************************
15:02:56 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-2.compute.amazonaws.com] => {
15:02:56 INFO cli.engine.ansible.AnsibleCommand -     "mount": {
15:02:56 INFO cli.engine.ansible.AnsibleCommand -         "device": "/dev/nvme0n1p2",
15:02:56 INFO cli.engine.ansible.AnsibleCommand -         "fstype": "xfs",
15:02:56 INFO cli.engine.ansible.AnsibleCommand -         "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-init-scripts/pgpool/6",
15:02:56 INFO cli.engine.ansible.AnsibleCommand -         "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:56 INFO cli.engine.ansible.AnsibleCommand -         "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:56 INFO cli.engine.ansible.AnsibleCommand -     }
15:02:56 INFO cli.engine.ansible.AnsibleCommand - }

Indeed, there is no size_available attribute returned by the first match.

15:02:55 INFO cli.engine.ansible.AnsibleCommand - TASK [preflight : debug] *******************************************************
15:02:55 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-2.compute.amazonaws.com] => {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -     "msg": [
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-init-scripts/pgpool/6",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/45d3f64c-441e-4ebd-b609-c7f3933a0e56/volume-subpaths/pgpool-config-files/pgpool/3",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-init-scripts/pgpool-init/0",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/45d3f64c-441e-4ebd-b609-c7f3933a0e56/volume-subpaths/pgpool-shared-init-dir/pgpool/5",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/45d3f64c-441e-4ebd-b609-c7f3933a0e56/volume-subpaths/pgpool-config-files/pgpool/4",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/bb181ad3-1328-4f29-a78c-e0ec6a8c7758/volume-subpaths/pgpool-shared-init-dir/pgpool/5",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-config-files/pgpool/4",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-config-files/pgpool/3",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-shared-init-dir/pgpool/5",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "block_available": 2395238,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "block_size": 4096,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "block_total": 7861243,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "block_used": 5466005,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "inode_available": 15611610,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "inode_total": 15727600,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "inode_used": 115990,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "size_available": 9810894848,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "size_total": 32199651328,
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/bb181ad3-1328-4f29-a78c-e0ec6a8c7758/volume-subpaths/pgpool-config-files/pgpool/3",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         {
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p2",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "fstype": "xfs",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/19aaea55-76d5-485b-b47c-6f684bc834c7/volume-subpaths/pgpool-init-scripts/pgpool/7",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "options": "rw,seclabel,relatime,attr2,inode64,noquota,bind",
15:02:55 INFO cli.engine.ansible.AnsibleCommand -             "uuid": "6bd5eb4d-fadd-4063-9d42-d5063758d65d"
15:02:55 INFO cli.engine.ansible.AnsibleCommand -         },

...

It looks the problem will occur only when the pgpool deployment is enabled, and when it is deployed on the repository vm, and this can only happen when having a single machine installation. In other cases it works fine because there is probably only 1 matching record. Reproduced on RHEL.

On Ubuntu it's fine as the first match contains size_available attribute but this is a risky assumption that it will always be so.

16:47:41 INFO cli.src.ansible.AnsibleCommand - TASK [preflight : debug] *******************************************************
16:47:42 INFO cli.src.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-2.compute.amazonaws.com] => {
16:47:42 INFO cli.src.ansible.AnsibleCommand -     "msg": [
16:47:42 INFO cli.src.ansible.AnsibleCommand -         {
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "block_available": 2049039,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "block_size": 4096,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "block_total": 7607140,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "block_used": 5558101,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p1",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "fstype": "ext4",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "inode_available": 3689122,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "inode_total": 3840000,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "inode_used": 150878,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "mount": "/",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "options": "rw,relatime,discard",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "size_available": 8392863744,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "size_total": 31158845440,
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "uuid": "bd626f37-18c7-48e8-8d71-dbe8225627d1"
16:47:42 INFO cli.src.ansible.AnsibleCommand -         },
16:47:42 INFO cli.src.ansible.AnsibleCommand -         {
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p1",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "fstype": "ext4",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/8a06adf1-e1c9-4da1-a565-220aa72f017c/volume-subpaths/pgpool-init-scripts/pgpool-init/0",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "options": "rw,relatime,discard,bind",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "uuid": "bd626f37-18c7-48e8-8d71-dbe8225627d1"
16:47:42 INFO cli.src.ansible.AnsibleCommand -         },
16:47:42 INFO cli.src.ansible.AnsibleCommand -         {
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "device": "/dev/nvme0n1p1",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "fstype": "ext4",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "mount": "/var/lib/kubelet/pods/870631b4-1c7b-42ea-9046-5e5bab555911/volume-subpaths/pgpool-init-scripts/pgpool-init/0",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "options": "rw,relatime,discard,bind",
16:47:42 INFO cli.src.ansible.AnsibleCommand -             "uuid": "bd626f37-18c7-48e8-8d71-dbe8225627d1"
16:47:42 INFO cli.src.ansible.AnsibleCommand -         },

@atsikham atsikham self-assigned this Mar 18, 2022
@przemyslavic przemyslavic changed the title [BUG] [RHEL] Single machine upgrade fails on preflight check: 'dict object' has no attribute 'size_available' [BUG] [RHEL] Single machine upgrade may fail on preflight check: 'dict object' has no attribute 'size_available' Mar 18, 2022
@przemyslavic przemyslavic self-assigned this Mar 18, 2022
@seriva seriva closed this as completed Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@atsikham atsikham

@przemyslavic przemyslavic

Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@atsikham @seriva @przemyslavic