airflow 2.3.3 keeps generating *-reply-celery-pidbox queues in SQS

[avshoram]

Apache Airflow version

2.3.3 (latest released)

What happened

Airflow keeps creating *-reply-celery-pidbox queues in SQS even with AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL=false

What you think should happen instead

The flag according to documentation https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html should prevent doing that.

How to reproduce

Airflow Image: 2.3.3-python3.9
Environment:
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL="sqs://Access_key_ID:Secret_access_key@"
AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL=false
AIRFLOW__CELERY__WORKER_CONCURRENCY=4

Operating System

Linux

Versions of Apache Airflow Providers

2.3.3-python3.9

Deployment

Docker-Compose

Deployment details

Local deployment

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Airflow does not have official support for SQS - only rabbitmq and redis are officially supported

[potiuk]

Converted it into discussion. We have #24019 where we discuss if we should have an SQS support or not but I think we would need to add some tests to support it, otherwise this is not something anyone would be looking at actively.