Failover with two AWX cluster supported or not

[subbucbe25]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

Hi,
I have two AWX cluster setup with one PG pointed to, to handover failover conditions. Job is scheduled in AWX-A cluster and when it is down, the job tries to run on AWX-B cluster and fails with this error. Is this supported or what is the job scheduling logic when we have two AWX cluster with one PG database setup?

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature
h.verify(data[-32:])
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 517, in run
    passwords = self.build_passwords(self.instance, kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 855, in build_passwords
    value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default=''))
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/credential/__init__.py", line 283, in get_input
    return decrypt_field(self, field_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field
    return smart_str(decrypt_value(key, value))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
    value = f.decrypt(encrypted)
            ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data
    self._verify_signature(data)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

AWX version

tags/2.19.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

Almalinux8

Web browser

Firefox

Steps to reproduce

1. Setup two AWX Clusters(AWX-A and AWX-B) with same PG node
2. Schedule a job which runs periodically say every 5mins.
3. Job first run in AWX-A cluster
4. Shutdown the AWX-A cluster
5. Next scheduled job will run in AWX-B

Expected results

The job is expected to run successfully in AWX-B cluster without any error.

Actual results

The job will fail with the following error.

Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature
h.verify(data[-32:])
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 517, in run
    passwords = self.build_passwords(self.instance, kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 855, in build_passwords
    value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default=''))
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/credential/__init__.py", line 283, in get_input
    return decrypt_field(self, field_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field
    return smart_str(decrypt_value(key, value))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
    value = f.decrypt(encrypted)
            ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data
    self._verify_signature(data)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidToken

Additional information

No response