From a3c4a238d5f57033e236795251450da383d1221b Mon Sep 17 00:00:00 2001 From: Akira Noda Date: Mon, 30 May 2022 18:24:44 +0900 Subject: [PATCH 1/2] add sqs configuration example --- docs/apache-airflow/executor/celery.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/apache-airflow/executor/celery.rst b/docs/apache-airflow/executor/celery.rst index f61cff47ad41c..daf50b273688f 100644 --- a/docs/apache-airflow/executor/celery.rst +++ b/docs/apache-airflow/executor/celery.rst @@ -26,6 +26,17 @@ to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and change your ``airflow.cfg`` to point the executor parameter to ``CeleryExecutor`` and provide the related Celery settings. +If you use Amazon SQS as a Celery broker, set environment variables like this: + +.. code-block:: bash + + export AIRFLOW__CELERY__BROKER_URL=sqs://@your_sqs_host + + # Celery's environment variables. + export AWS_ACCESS_KEY_ID=your_aws_access_key_id + export AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key + export AWS_DEFAULT_REGION=your_aws_region + For more information about setting up a Celery broker, refer to the exhaustive `Celery documentation on the topic `_. From d17c40fea06ad56b65dfdca9a71cf720a4dd0cc1 Mon Sep 17 00:00:00 2001 From: tsugumi-sys Date: Sun, 24 Jul 2022 18:14:28 +0900 Subject: [PATCH 2/2] add notes for using sqs --- docs/apache-airflow/executor/celery.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/apache-airflow/executor/celery.rst b/docs/apache-airflow/executor/celery.rst index daf50b273688f..0b6b7b1312ccd 100644 --- a/docs/apache-airflow/executor/celery.rst +++ b/docs/apache-airflow/executor/celery.rst @@ -28,6 +28,9 @@ change your ``airflow.cfg`` to point the executor parameter to If you use Amazon SQS as a Celery broker, set environment variables like this: +.. note:: + Using Amazon SQS for a Celery backed in Airflow is experimental and use at your own risk. + .. code-block:: bash export AIRFLOW__CELERY__BROKER_URL=sqs://@your_sqs_host