[BUG] [Ubuntu] [PostgreSQL] Apply command fails after upgrading from v1.0.x LTS to v1.3.0

[przemyslavic]

Describe the bug
I'm not able to re-apply configuration after upgrading the cluster from v1.0.x LTS to v1.3.0.
It fails on TASK [postgresql : Extensions | repmgr | Install package(s)]. There is an issue with downgrading repmgr-common package.
Reproduced on Ubuntu.

2022-01-17T19:37:30.2731957Z[38;21m19:37:30 INFO cli.engine.ansible.AnsibleCommand - TASK [postgresql : Extensions | repmgr | Install package(s)] *******************
2022-01-17T19:37:31.2189394Z[31;21m19:37:31 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-10to13azubflannel-postgresql-vm-1]: FAILED! => {"cache_update_time": 1642447402, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'repmgr-common=5.2.1-*'' failed: E: Packages were downgraded and -y was used without --allow-downgrades.
", "rc": 100, "stderr": "E: Packages were downgraded and -y was used without --allow-downgrades.
", "stderr_lines": ["E: Packages were downgraded and -y was used without --allow-downgrades."], "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  repmgr-common
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 3 not upgraded.
", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be DOWNGRADED:", "  repmgr-common", "0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 3 not upgraded."]}
2022-01-17T19:37:31.3099865Z[31;21m19:37:31 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-10to13azubflannel-postgresql-vm-0]: FAILED! => {"cache_update_time": 1642447402, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'repmgr-common=5.2.1-*'' failed: E: Packages were downgraded and -y was used without --allow-downgrades.
", "rc": 100, "stderr": "E: Packages were downgraded and -y was used without --allow-downgrades.
", "stderr_lines": ["E: Packages were downgraded and -y was used without --allow-downgrades."], "stdout": "Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be DOWNGRADED:
  repmgr-common
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 3 not upgraded.
", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be DOWNGRADED:", "  repmgr-common", "0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 3 not upgraded."]}

EDIT: repmgr-common package in version 5.3.0 is installed as dependency by epicli v1.0 LTS as there is no fixed version (it's been introduced in v1.2: https://github.com/epiphany-platform/epiphany/blob/v1.2/core/src/epicli/data/common/ansible/playbooks/roles/postgresql/tasks/extensions/replication/packages.yml#L10)

[ubuntu@ec2-1-1-1-1 ~]$ apt-cache policy repmgr-common
repmgr-common:
  Installed: 5.3.0-1.bionic+1
  Candidate: 5.3.0-1.bionic+1
  Version table:
 *** 5.3.0-1.bionic+1 100
        100 /var/lib/dpkg/status
     4.0.3-1 500
        500 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

How to reproduce
Steps to reproduce the behavior:

1. Deploy v1.0.1 cluster with 2 PostgreSQL vms and repmgr enabled.
2. Upgrade OS to Ubuntu 20.04 (this can be helpful: https://github.com/epiphany-platform/epiphany/blob/develop/ci/ansible/playbooks/os/ubuntu/upgrade-release.yml)
3. Upgrade epicli to v1.3.0
4. Execute apply (v1.3.0)

Expected behavior
The configuration should be applied again with no issues.

Config files

Environment

• Cloud provider: [all]
• OS: [Ubuntu]

epicli version: v1.3.0, develop

Additional context

---

DoD checklist

• Changelog updated (if affected version was released)
• COMPONENTS.md updated / doesn't need to be updated
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
• Case covered by automated test (if possible)
• Idempotency tested
• Documentation updated / doesn't need to be updated
• All conversations in PR resolved
• Backport tasks created / doesn't need to be backported

[przemyslavic]

Packages in epirepo:

[operations@ci-10to13azubflannel-repository-vm-0 ~]$ ls -lah /var/www/html/epirepo/packages/repmgr-common*
-rw-r--r-- 1 root root 179K Nov  4 00:49 /var/www/html/epirepo/packages/repmgr-common_5.2.1-3.pgdg20.04+1_all.deb
-rw-r--r-- 1 root root 181K Nov 13 00:13 /var/www/html/epirepo/packages/repmgr-common_5.3.0-1.pgdg20.04+1_all.deb

PostgreSQL vm:

[operations@ci-10to13azubflannel-postgresql-vm-0 ~]$ apt-cache policy repmgr-common
repmgr-common:
  Installed: 5.3.0-1.bionic+1
  Candidate: 5.3.0-1.pgdg20.04+1
  Version table:
     5.3.0-1.pgdg20.04+1 500
        500 http://10.1.11.4/epirepo/packages ./ Packages
 *** 5.3.0-1.bionic+1 100
        100 /var/lib/dpkg/status
     5.2.1-3.pgdg20.04+1 500
        500 http://10.1.11.4/epirepo/packages ./ Packages

It looks like repmgr-common 5.3.0 is already installed and we try to install 5.2.1.

[przemyslavic]

The issue is that in LTS we install the latest repmgr-common package as a dependency and not a fixed one. Starting from v1.2 we fix the version.

✔️ Tested:

• upgrade from v1.0.1 + re-apply v1.3
• upgrade from v1.2.0 + re-apply v1.3
• apply v1.3 + re-apply v1.3