-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-3702] Add backfill option to run backwards #4676
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4676 +/- ##
==========================================
+ Coverage 74.42% 74.43% +0.01%
==========================================
Files 430 430
Lines 27972 27980 +8
==========================================
+ Hits 20819 20828 +9
+ Misses 7153 7152 -1
Continue to review full report at Codecov.
|
@dima-asana , I have been pretty busy lately. Looking at your code, I think your pr is almost there which only needs to handle the case when the DAG has depend_on_past(we could throw the exception if reverse_backfill and depend_on_past are both true). Do you want to update your pr ? I think we could commit yours and close mine. What do you think? |
22e5eaa
to
81b9f75
Compare
Sure, updated. Note that in addition to the suggested change for depends_on_past handling, I changed the executor queue dict to an ordered dict. I think for this change to be impactful that's necessary -- I didn't realize that the executor queue were orderless until testing this code more. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 small nit. LGTM. thanks for the patch.
airflow/bin/cli.py
Outdated
( | ||
"if set, the backfill will run tasks from the most " | ||
"recent day first " | ||
"instead of throwing exceptions"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about adding one more line "this option will fail if the DAG depends on past."?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
81b9f75
to
2d558f6
Compare
Hi, can the from airflow import DAG
dag = DAG(
dag_id = "...",
scheduled_interval = "@daily",
run_backwards = True,
...
) It seems to be something that can be used only manually through the Airflow cli but wanted to confirm. Is there any way to setup a DAG to run as backfilling job from the most recent DAG to the oldest? |
hi @dima-asana @feng-tao Any updates on the above comment for running backwards dag from dag definition.
|
Make sure you have checked all steps below.
Jira
Description
This adds an optional capability for the backfill CLI command to process dates in reverse order.
Tests
test_jobs::BackfillJobTest::test_backfill_enqueue_backwards
Commits
Documentation
Code Quality
flake8