Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add celery broker configuration example using Amazon SQS #24019

Closed
wants to merge 4 commits into from
Closed

Add celery broker configuration example using Amazon SQS #24019

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a3c4a23
add sqs configuration example
May 30, 2022
756e811
Merge branch 'apache:main' into add-sqs-configuration-doc
tsugumi-sys Jul 22, 2022
d17c40f
add notes for using sqs
tsugumi-sys Jul 24, 2022
7da9e8c
Merge branch 'apache:main' into add-sqs-configuration-doc
tsugumi-sys Jul 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/apache-airflow/executor/celery.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,20 @@ to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
change your ``airflow.cfg`` to point the executor parameter to
``CeleryExecutor`` and provide the related Celery settings.

If you use Amazon SQS as a Celery broker, set environment variables like this:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a note that this is experimental support/use at own risk. It would be great also to add something along "there are anecdotal evidences that Amazon SQS" might be used as the broker backend.
We do not run any tests for SQS as celery backend (as opposed to rabbitmq and redis - we have integration tests for both).

We do not want to give the users false impression that they can expect it to work and ask as question "I followed the documentation and it does not work" - unless we have CI/tests covering the scenario we are talking about.

Or maybe @ferruzzi @o-nikolas @john-jac - there could be some blog post or documentation from Amazon team to describe it. I believe MWAA uses SQS as broker and maybe one of you (or someone from the amazon developer relations team) coudl write a blog post on "How to use SQS as celery broker for Airflow? " that woudl be nice - and then any questions about it could come to Amazon- I am afraid until we make system tests/integration with Amazon, making a statement that "SQS works and here is the configuration" in Airlfow docs is a little too much "leap of faith" :)


.. note::
Using Amazon SQS for a Celery backed in Airflow is experimental and use at your own risk.

.. code-block:: bash

export AIRFLOW__CELERY__BROKER_URL=sqs://@your_sqs_host

# Celery's environment variables.
export AWS_ACCESS_KEY_ID=your_aws_access_key_id
export AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key
export AWS_DEFAULT_REGION=your_aws_region
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if will be great if you can also mention this setting #21507

Copy link
Author

@tsugumi-sys tsugumi-sys Jul 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add environment variable AIRFLOW__CELERY__WORKER_ENABLE_REMOTE_CONTROL=true but new queues are not created.


For more information about setting up a Celery broker, refer to the
exhaustive `Celery documentation on the topic <https://docs.celeryq.dev/en/latest/getting-started/>`_.

Expand Down