Skip to content

Commit

Permalink
Move filesystem sensor to standard provider (apache#43890)
Browse files Browse the repository at this point in the history
* Moved filesystem sensor to standard provider

* Update all references to filesystem sensor

* Update provider.yaml with standard filesystem sensor name

* Typo fix in contributing-docs

* Changing title underline as per static test error msg

* DAT-1897 - Update docs and references as per review comments

* Moved sensor documentation to standard provider

* Removed reference to file sensor from index.rst
  • Loading branch information
kunaljubce authored Nov 17, 2024
1 parent 987c50a commit 36803b8
Show file tree
Hide file tree
Showing 12 changed files with 11 additions and 14 deletions.
4 changes: 2 additions & 2 deletions RELEASE_NOTES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6964,8 +6964,8 @@ The change is backwards compatible, setting ``provide_context`` will add the ``p

PR: `#5990 <https://github.com/apache/airflow/pull/5990>`_

``airflow.sensors.filesystem.FileSensor``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``airflow.providers.standard.sensors.filesystem.FileSensor``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has ``*``\ , ``?``\ , or ``[`` in it then you should replace these with ``[*]``\ , ``[?]``\ , and ``[[]``.

Expand Down
2 changes: 1 addition & 1 deletion airflow/example_dags/example_sensors.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@
from airflow.models.dag import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.sensors.bash import BashSensor
from airflow.providers.standard.sensors.filesystem import FileSensor
from airflow.providers.standard.sensors.python import PythonSensor
from airflow.providers.standard.sensors.time import TimeSensor, TimeSensorAsync
from airflow.providers.standard.sensors.time_delta import TimeDeltaSensor, TimeDeltaSensorAsync
from airflow.providers.standard.sensors.weekday import DayOfWeekSensor
from airflow.sensors.filesystem import FileSensor
from airflow.utils.trigger_rule import TriggerRule
from airflow.utils.weekday import WeekDay

Expand Down
4 changes: 2 additions & 2 deletions airflow/reproducible_build.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
release-notes-hash: 4893030c9c8a4d22d13291823b88ad4d
source-date-epoch: 1730818031
release-notes-hash: c68f3fa23f84c7fc270d73baaa2cc18d
source-date-epoch: 1731415143
2 changes: 1 addition & 1 deletion contributing-docs/03_contributors_quick_start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -478,7 +478,7 @@ or
pipx install pre-commit
You can add ````uv`` support for ``pre-commit`` even you install it with ``pipx`` using the commands
You can add ``uv`` support for ``pre-commit`` even you install it with ``pipx`` using the commands
(then pre-commit will use ``uv`` to create virtualenvs for the hooks):

.. code-block:: bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
FileSensor
==========

Use the :class:`~airflow.sensors.filesystem.FileSensor` to detect files appearing in your local
Use the :class:`~airflow.providers.standard.sensors.filesystem.FileSensor` to detect files appearing in your local
filesystem. You need to have connection defined to use it (pass connection id via ``fs_conn_id``).
Default connection is ``fs_default``.

Expand Down
1 change: 0 additions & 1 deletion docs/apache-airflow/howto/operator/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,4 @@ determine what actually executes when your DAG runs.
.. toctree::
:maxdepth: 2

file
external_task_sensor
3 changes: 0 additions & 3 deletions docs/apache-airflow/operators-and-hooks-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,6 @@ For details see: :doc:`apache-airflow-providers:operators-and-hooks-ref/index`.
* - :mod:`airflow.sensors.external_task`
- :doc:`How to use <howto/operator/external_task_sensor>`

* - :mod:`airflow.sensors.filesystem`
- :ref:`How to use <howto/operator:FileSensor>`


**Hooks:**

Expand Down
2 changes: 1 addition & 1 deletion docs/apache-airflow/tutorial/taskflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ Adding dependencies between decorated and traditional tasks
-----------------------------------------------------------
The above tutorial shows how to create dependencies between TaskFlow functions. However, dependencies can also
be set between traditional tasks (such as :class:`~airflow.providers.standard.operators.bash.BashOperator`
or :class:`~airflow.sensors.filesystem.FileSensor`) and TaskFlow functions.
or :class:`~airflow.providers.standard.sensors.filesystem.FileSensor`) and TaskFlow functions.

Building this dependency is shown in the code below:

Expand Down
1 change: 1 addition & 0 deletions providers/src/airflow/providers/standard/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ sensors:
- airflow.providers.standard.sensors.weekday
- airflow.providers.standard.sensors.bash
- airflow.providers.standard.sensors.python
- airflow.providers.standard.sensors.filesystem
hooks:
- integration-name: Standard
python-modules:
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion tests/cli/commands/test_task_command.py
Original file line number Diff line number Diff line change
Expand Up @@ -411,7 +411,7 @@ def test_cli_test_with_env_vars(self):
@mock.patch("airflow.triggers.file.os.path.getmtime", return_value=0)
@mock.patch("airflow.triggers.file.glob", return_value=["/tmp/test"])
@mock.patch("airflow.triggers.file.os.path.isfile", return_value=True)
@mock.patch("airflow.sensors.filesystem.FileSensor.poke", return_value=False)
@mock.patch("airflow.providers.standard.sensors.filesystem.FileSensor.poke", return_value=False)
def test_cli_test_with_deferrable_operator(
self, mock_pock, mock_is_file, mock_glob, mock_getmtime, caplog
):
Expand Down
2 changes: 1 addition & 1 deletion tests/sensors/test_filesystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

from airflow.exceptions import AirflowSensorTimeout, TaskDeferred
from airflow.models.dag import DAG
from airflow.sensors.filesystem import FileSensor
from airflow.providers.standard.sensors.filesystem import FileSensor
from airflow.triggers.file import FileTrigger
from airflow.utils.timezone import datetime

Expand Down

0 comments on commit 36803b8

Please sign in to comment.