Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document limitations of some async operators #158

Merged
merged 1 commit into from
Mar 24, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ class KubernetesPodOperatorAsync(KubernetesPodOperator):
"""
Async (deferring) version of KubernetesPodOperator

.. warning::
The logs would not be available in the Airflow Webserver until the task completes. This is
the main difference between this operator and the
:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`.

:param poll_interval: interval in seconds to sleep between checking pod status
"""

Expand Down
10 changes: 7 additions & 3 deletions astronomer/providers/http/sensors/http.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,23 @@ class HttpSensorAsync(HttpSensor):
Executes a HTTP GET statement and returns False on failure caused by
404 Not Found or `response_check` returning False.

If ``response_check`` is passed, the sync version of the sensor will be used.
.. note::
If ``response_check`` is passed, the sync version of the sensor will be used.

The response check can access the template context to the operator:

.. code-block:: python

def response_check(response, task_instance):
# The task_instance is injected, so you can pull data form xcom
# Other context variables such as dag, ds, execution_date are also available.
xcom_data = task_instance.xcom_pull(task_ids='pushing_task')
xcom_data = task_instance.xcom_pull(task_ids="pushing_task")
# In practice you would do something more sensible with this data..
print(xcom_data)
return True

HttpSensorAsync(task_id='my_http_sensor', ..., response_check=response_check)

HttpSensorAsync(task_id="my_http_sensor", ..., response_check=response_check)

:param http_conn_id: The Connection ID to run the sensor against
:type http_conn_id: str
Expand Down
8 changes: 8 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,11 @@ include =
exclude = venv/*,tox/*,specs/*
ignore = E123,E128,E266,E402,W503,E731,W601
max-line-length = 119
# The following are needed for flake8-rst-docstrings to make it compatible with Sphinx
rst-roles =
class,
func,
ref,
rst-directives =
envvar,
exception,