Support DAGS folder being in different location on scheduler and runners #16860

ashb · 2021-07-07T16:03:51Z

There has been some vestigial support for this concept in Airflow for a
while (all the CLI command already turn the literal DAGS_FOLDER in to
the real value of the DAGS folder when loading dags), but sometime
around 1.10.1-1.10.3 it got fully broken and the scheduler only ever
passed full paths to DAG files.

This PR brings back this behaviour

Closes #8061

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ashb · 2021-07-07T16:05:25Z

airflow/jobs/scheduler_job.py

-                mark_success=False,
-                ignore_all_deps=False,
-                ignore_depends_on_past=False,
-                ignore_task_deps=False,
-                ignore_ti_state=False,


These are the defaults, we don't need to pass them again

ashb · 2021-07-07T16:06:32Z

airflow/models/dag.py

@@ -2379,6 +2413,8 @@ class DagModel(Base):
        Index('idx_next_dagrun_create_after', next_dagrun_create_after, unique=False),
    )

+    parent_dag = relationship("DagModel", remote_side=[dag_id])


This relationship was added for "Duck-type compatibility" between a DAG and a DagModel in the case of subdags -- both now have a "parent_dag" attribute.

airflow/models/dag.py

mik-laj · 2021-07-07T22:05:36Z

airflow/models/dag.py

+
+    @property
+    def relative_fileloc(self) -> str:
+        """File location of the importable dag 'file' relative to the configured DAGs folder."""


How will this field behave for example DAGs?

In this case it would be /usr/lib/python3.7/.../airflow/example_dags/example_bash_operator.py -- so not relative at all.

I'll update/expand the docstring to say how it deals with DAGs outside of the dags folder.

Maybe we are also able to handle examples DAGs by adding a suffix? This will allow us to install workers on shared machines without any problems
For samples:
[exaample_dags_folder]/example_bash_operator.py => /usr/lib/python3.7/.../airflow/example_dags/example_bash_operator.py
[dags_folder]/example_bash_operator.py => ~/home/airflow/dags

I think this change would be better done along with something like https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-20+DAG+manifest, so I've not done this for now.

Maybe we shouldn’t call this relative fileloc, but just fileloc…? Not sure about this.

airflow/models/dag.py

github-actions · 2021-07-27T12:46:39Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

airflow/models/dag.py

airflow/models/dagbag.py

uranusjr · 2021-07-30T06:20:57Z

airflow/models/dag.py

        'template_searchpath',
        'last_loaded',
    }

    __serialized_fields: Optional[FrozenSet[str]] = None

+    fileloc: str


This reminds me I was wondering whether fileloc is guaranteed to be absolute or not and had to trace a lot of code. Maybe it’s worthwhile to add this to the docstring (and the one on DagModel).

uranusjr

I think this looks good to me in general

potiuk

LGTM!

potiuk · 2021-08-01T20:31:16Z

@ashb - can you rebase please ? I think most of the issues with CI stability are solved

There has been some vestigial support for this concept in Airflow for a while (all the CLI command already turn the literal `DAGS_FOLDER` in to the real value of the DAGS folder when loading dags), but sometime around 1.10.1-1.10.3 it got fully broken and the scheduler only ever passed full paths to DAG files. This PR brings back this behaviour

…nd runners

ashb · 2021-08-04T12:47:20Z

Looks like transient MSQQL errors :(

  ERROR: for airflow  Container "69eb70e8e951" is unhealthy.
  Encountered errors while bringing up the project.

noelmcloughlin · 2021-08-06T10:13:19Z

Linking #16025 (comment)

boring-cyborg bot added area:Scheduler including HA (high availability) scheduler area:serialization area:UI Related to UI/UX. For Frontend Developers. area:webserver Webserver related Issues labels Jul 7, 2021

ashb changed the title ~~Support DAGS folder being in different location on scheduler and runners~~ 🚧 Support DAGS folder being in different location on scheduler and runners Jul 7, 2021

ashb mentioned this pull request Jul 7, 2021

Added windows extensions #16110

Merged

ashb commented Jul 7, 2021

View reviewed changes

mik-laj reviewed Jul 7, 2021

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

mik-laj reviewed Jul 7, 2021

View reviewed changes

ashb added this to the Airflow 2.2 milestone Jul 27, 2021

ashb force-pushed the dags-folder-in-different-location-workers branch from a72eb83 to 6a7d7ab Compare July 27, 2021 11:08

ashb marked this pull request as ready for review July 27, 2021 11:09

ashb requested review from kaxil, ryanahamilton and XD-DENG as code owners July 27, 2021 11:09

ashb changed the title ~~🚧 Support DAGS folder being in different location on scheduler and runners~~ Support DAGS folder being in different location on scheduler and runners Jul 27, 2021

ashb requested a review from uranusjr July 27, 2021 11:09

kaxil reviewed Jul 27, 2021

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

kaxil approved these changes Jul 27, 2021

View reviewed changes

github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jul 27, 2021

uranusjr reviewed Jul 28, 2021

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

uranusjr reviewed Jul 28, 2021

View reviewed changes

airflow/models/dagbag.py Outdated Show resolved Hide resolved

ashb requested a review from uranusjr July 28, 2021 14:46

uranusjr reviewed Jul 30, 2021

View reviewed changes

uranusjr approved these changes Jul 30, 2021

View reviewed changes

potiuk approved these changes Aug 1, 2021

View reviewed changes

ashb force-pushed the dags-folder-in-different-location-workers branch from be08c89 to af12ddd Compare August 3, 2021 11:08

ashb force-pushed the dags-folder-in-different-location-workers branch from ca640d3 to 0da2b8c Compare August 4, 2021 09:10

fixup! Support DAGS folder being in different location on scheduler a…

5438091

…nd runners

ashb merged commit 9bed58e into apache:main Aug 4, 2021

noelmcloughlin mentioned this pull request Aug 6, 2021

AirflowException: Celery command failed on host: [...] #16025

Closed

noelmcloughlin mentioned this pull request Aug 26, 2021

Duplicate tasks invoked for a single task_id when manually invoked task details modal. #10026

Closed

baryluk mentioned this pull request Nov 18, 2021

Support DAGS folder being in different location on scheduler and worker. #8061

Closed

ashb deleted the dags-folder-in-different-location-workers branch November 18, 2021 16:44

This was referenced Sep 26, 2022

Upgrade Airflow 2.1.4 -> 2.2.0 : Kubernetes executor worker pod can't resolve dag folder #26623

Closed

Fix airflow tasks run --local when dags_folder differs from that of processor #26509

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support DAGS folder being in different location on scheduler and runners #16860

Support DAGS folder being in different location on scheduler and runners #16860

ashb commented Jul 7, 2021

ashb Jul 7, 2021

ashb Jul 7, 2021

mik-laj Jul 7, 2021

ashb Jul 8, 2021

mik-laj Jul 10, 2021

ashb Jul 27, 2021

uranusjr Jul 28, 2021

github-actions bot commented Jul 27, 2021

uranusjr Jul 30, 2021

uranusjr left a comment

potiuk left a comment

potiuk commented Aug 1, 2021

ashb commented Aug 4, 2021

noelmcloughlin commented Aug 6, 2021

Support DAGS folder being in different location on scheduler and runners #16860

Support DAGS folder being in different location on scheduler and runners #16860

Conversation

ashb commented Jul 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 27, 2021

Choose a reason for hiding this comment

uranusjr left a comment

Choose a reason for hiding this comment

potiuk left a comment

Choose a reason for hiding this comment

potiuk commented Aug 1, 2021

ashb commented Aug 4, 2021

noelmcloughlin commented Aug 6, 2021