Skip to content

Commit

Permalink
Document limitations of some async operators (#158)
Browse files Browse the repository at this point in the history
  • Loading branch information
OlympuJupiter committed Mar 24, 2022
1 parent 920d7b7 commit fb097d4
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ class KubernetesPodOperatorAsync(KubernetesPodOperator):
"""
Async (deferring) version of KubernetesPodOperator
.. warning::
The logs would not be available in the Airflow Webserver until the task completes. This is
the main difference between this operator and the
:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`.
:param poll_interval: interval in seconds to sleep between checking pod status
"""

Expand Down
10 changes: 7 additions & 3 deletions astronomer/providers/http/sensors/http.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,23 @@ class HttpSensorAsync(HttpSensor):
Executes a HTTP GET statement and returns False on failure caused by
404 Not Found or `response_check` returning False.
If ``response_check`` is passed, the sync version of the sensor will be used.
.. note::
If ``response_check`` is passed, the sync version of the sensor will be used.
The response check can access the template context to the operator:
.. code-block:: python
def response_check(response, task_instance):
# The task_instance is injected, so you can pull data form xcom
# Other context variables such as dag, ds, execution_date are also available.
xcom_data = task_instance.xcom_pull(task_ids='pushing_task')
xcom_data = task_instance.xcom_pull(task_ids="pushing_task")
# In practice you would do something more sensible with this data..
print(xcom_data)
return True
HttpSensorAsync(task_id='my_http_sensor', ..., response_check=response_check)
HttpSensorAsync(task_id="my_http_sensor", ..., response_check=response_check)
:param http_conn_id: The Connection ID to run the sensor against
:type http_conn_id: str
Expand Down
8 changes: 8 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,11 @@ include =
exclude = venv/*,tox/*,specs/*
ignore = E123,E128,E266,E402,W503,E731,W601
max-line-length = 119
# The following are needed for flake8-rst-docstrings to make it compatible with Sphinx
rst-roles =
class,
func,
ref,
rst-directives =
envvar,
exception,

0 comments on commit fb097d4

Please sign in to comment.