Is there a way to protect an AWX instance from automatically upgrading when the operator gets upgraded?

[Cl0udius]

ISSUE TYPE

• Question

SUMMARY

Is there a way to protect an AWX instance from automatically upgrading when the operator gets upgraded?

ADDITIONAL INFORMATION

We currently have two AWX instances in our K8S cluster installed with the AWX operator.
After the operator upgrade all instances all existing AWX instances PODs get restartet even when we specify all images versions inside the AWX object.
This means we have a short but unwanted downtime for all AWX instances.
We would like to have the possibility to upgrade those instances step by step and not all at once.

Is there some kind of tag we can set at the pod or deployment object of the existing instance to "protect" it from automatically updates?
As i could see it in the code the deployment template will be applied in any case.
Maybe a conditional check on this task could solve our problem here.

ENVIRONMENT

• AWX version: 19.4.0
• Operator version: 0.19.0
• Kubernetes version: 1.23.6

[vrubiolo]

Hi,

Just a question here since AFAIK the operator is namespace scoped: could you just upgrade one operator and not the other? I would expect that only one AWX instance would get upgraded, not the other (but I might be wrong here, I have only one instance in my cluster).

[Cl0udius]

Hi,

i also got such thing in mind but we may could run into problems when there is a CRD update with the new operator version, as the CRDs are not namespace bound AFAIK. So i am not sure if this could lead to side effects.

BR

[fosterseth]

if you look at the operator logs, what are the results for the Apply deployment resources task

awx-operator/roles/installer/tasks/resources_configuration.yml

Line 75 in 0983220

- name: Apply deployment resources

does it say changed?

[Cl0udius]

Hi,

i could not really get it from the logs. I found some tasks which were changed with this

TASK [Apply Resources] ********************************
changed: [localhost] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

The task you mention had the following output:

TASK [installer : Apply deployment resources] **********************************
task path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:75

-------------------------------------------------------------------------------
{"level":"info","ts":1655138956.5544703,"logger":"logging_event_handler","msg":"[playbook task start]","name":"awx-staging02","namespace":"awx02","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"1833956775654613627","EventData.Name":"installer : Apply deployment resources"}
{"level":"info","ts":1655138958.0021372,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/apis/apps/v1/namespaces/awx02/deployments/awx-staging02","Verb":"get","APIPrefix":"apis","APIGroup":"apps","APIVersion":"v1","Namespace":"awx02","Resource":"deployments","Subresource":"","Name":"awx-staging02","Parts":["deployments","awx-staging02"]}}
{"level":"info","ts":1655138958.0118575,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/apis/apps/v1/namespaces/awx02/deployments/awx-staging02","Verb":"get","APIPrefix":"apis","APIGroup":"apps","APIVersion":"v1","Namespace":"awx02","Resource":"deployments","Subresource":"","Name":"awx-staging02","Parts":["deployments","awx-staging02"]}}
{"level":"info","ts":1655138958.0571153,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/apis/apps/v1/namespaces/awx02/deployments/awx-staging02","Verb":"get","APIPrefix":"apis","APIGroup":"apps","APIVersion":"v1","Namespace":"awx02","Resource":"deployments","Subresource":"","Name":"awx-staging02","Parts":["deployments","awx-staging02"]}}
{"level":"info","ts":1655138963.073745,"logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/apis/apps/v1/namespaces/awx02/deployments/awx-staging02","Verb":"get","APIPrefix":"apis","APIGroup":"apps","APIVersion":"v1","Namespace":"awx02","Resource":"deployments","Subresource":"","Name":"awx-staging02","Parts":["deployments","awx-staging02"]}}

The stats of the playbook run:

PLAY RECAP *********************************************************************
localhost                  : ok=75   changed=18   unreachable=0    failed=0    skipped=36   rescued=0    ignored=0

The test config for our AWX:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-staging02
  namespace: awx02
spec:
  ingress_type: Ingress
  hostname: awx-staging02
  admin_password_secret: awx-my-admin-password
  image: "quay.io/ansible/awx" 
  image_version: "19.4.0"
  ee_images: 
    - name: "AWX-EE"
      image: "quay.io/ansible/awx-ee:0.6.0"
  redis_image: "docker.io/redis"
  redis_image_version: "7.0.0"
  postgres_image: "docker.io/postgres"
  postgres_image_version: "12"
  control_plane_ee_image: "quay.io/ansible/awx-ee:0.6.0"

It would be really good if we had some attribute on the AWX object which could stop the operator from automatically rebuild all instances at once.
Deploying two operators would be really just plan B for us because the effort of maintaining this would be quite high.

BR

[Cl0udius]

What i had in mind was something like "auto_upgrade: true|false" flag which could be evaluated in the operator code.
When set to false we could upgrade AWX by recreating the pods manual step by step.