[AIRFLOW-4343] Show warning in UI if scheduler is not running #5127

ashb · 2019-04-17T18:20:33Z

First commit is from #5125, look at the second commit only for now.
Make sure you have checked all steps below.

Jira

- https://issues.apache.org/jira/browse/AIRFLOW-4343
- In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy. - yes, Cat A

Description

Now that the webserver is more stateless, if the scheduler is not
running the list of dags won't populate, making it harder for new
starters to work out what is going on.

When scheduler has never run (I cleared Jobs table):

**When scheduler not running (either crashed, or has stopped successfully)

(the time has a tooltip to display the full ISO8601 heartbeat)

Login screen/anything not subclassed from AirflowBaseView doesn't display the message.

On Login (not changed, doesn't show warning):

Tests

My PR adds the following unit tests OR does not need testing for this extremely good reason: added tests for helper functions on BaseJob

Commits

My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Documentation

In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
- If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

Passes flake8

/cc @Fokko as discussed.

ashb · 2019-04-17T21:09:18Z

@XD-DENG PTAL too - I've subtly changed the /health endpoint here to use functions from airflow.jobs.

XD-DENG · 2019-04-18T00:12:01Z

Awesome feature 👍Will have a deeper dive later.

feluelle

Nice @ashb I really like this feature 👍

only some comments..

airflow/macros/__init__.py

airflow/jobs.py

airflow/config_templates/default_airflow.cfg

sdikby · 2019-04-24T06:31:13Z

@ashb Nice feature. A suggestion here: adding a link (with the warning) to the scheduler logs would help debug quicker. It should show only a portion of the logs.

ashb · 2019-04-24T11:24:00Z

@sdikby That would be nice but is non-trival to do so I'm not going to do it as part of this PR.

airflow/jobs.py

kaxil · 2019-04-30T20:38:44Z

airflow/jobs.py

+
+        :param grace_multiplier: multiplier of heartrate to require heart beat
+            within
+        :type grace_multiplier: number


Maybe remove this from here and add type hint in L130

Type hint's aren't picked up by docs yet are they?

Good point. I don't know tbh :-D

Just for our future reference, we can use this:
https://github.com/agronholm/sphinx-autodoc-typehints

airflow/jobs.py

airflow/macros/__init__.py

ashb · 2019-05-01T09:34:52Z

I think something odd/unexpected is up with the Kube tests as a result of this PR. They consistently fail but don't on master.

ashb · 2019-05-03T12:57:56Z

airflow/jobs.py

@@ -94,25 +94,54 @@ class BaseJob(Base, LoggingMixin):
        Index('idx_job_state_heartbeat', state, latest_heartbeat),
    )

+    heartrate = conf.getfloat('scheduler', 'JOB_HEARTBEAT_SEC'),


DEAR GOD. That took like 4 hours to get enough of a working set up to get the log for!

codecov-io · 2019-05-03T15:21:51Z

Codecov Report

Merging #5127 into master will increase coverage by <.01%.
The diff coverage is 90.32%.

@@            Coverage Diff             @@
##           master    #5127      +/-   ##
==========================================
+ Coverage   78.93%   78.93%   +<.01%     
==========================================
  Files         480      480              
  Lines       30129    30146      +17     
==========================================
+ Hits        23782    23797      +15     
- Misses       6347     6349       +2

Impacted Files	Coverage Δ
airflow/jobs/base_job.py	`86.42% <100%> (+1.46%)`	⬆️
airflow/www/views.py	`75.72% <100%> (+0.03%)`	⬆️
airflow/macros/__init__.py	`90.9% <33.33%> (-9.1%)`	⬇️
airflow/jobs/scheduler_job.py	`70.32% <83.33%> (+0.07%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a4d32...657a2ee. Read the comment docs.

Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. New dep is BSD-2 which is Cat-A under ASF

…#5127) Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. New dep is BSD-2 which is Cat-A under ASF (cherry picked from commit 3dcfe28)

…#5127) Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. New dep is BSD-2 which is Cat-A under ASF

…#5127) Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. New dep is BSD-2 which is Cat-A under ASF (cherry picked from commit 3dcfe28)

Somehow in the cherry pick these tests got missed.

ashb force-pushed the warning-if-scheduler-not-running branch from cb0bf50 to a00fe0d Compare April 17, 2019 21:08

ashb force-pushed the warning-if-scheduler-not-running branch from a00fe0d to 0af1963 Compare April 17, 2019 21:14

ashb force-pushed the warning-if-scheduler-not-running branch 2 times, most recently from 6b31eae to 37f0f2e Compare April 18, 2019 10:14

ashb marked this pull request as ready for review April 18, 2019 16:14

ashb force-pushed the warning-if-scheduler-not-running branch 2 times, most recently from e4f7d50 to 17dbba0 Compare April 18, 2019 17:11

feluelle reviewed Apr 18, 2019

View reviewed changes

airflow/macros/__init__.py Outdated Show resolved Hide resolved

airflow/jobs.py Outdated Show resolved Hide resolved

airflow/jobs.py Outdated Show resolved Hide resolved

ashb force-pushed the warning-if-scheduler-not-running branch from 17dbba0 to e3115b2 Compare April 23, 2019 08:23

ashb commented Apr 23, 2019

View reviewed changes

airflow/config_templates/default_airflow.cfg Outdated Show resolved Hide resolved

ashb force-pushed the warning-if-scheduler-not-running branch 4 times, most recently from a53202e to 58f1b1f Compare April 29, 2019 08:29

XD-DENG requested changes Apr 29, 2019

View reviewed changes

airflow/jobs.py Outdated Show resolved Hide resolved

XD-DENG approved these changes Apr 29, 2019

View reviewed changes

ashb force-pushed the warning-if-scheduler-not-running branch from 58f1b1f to 47c1bc0 Compare April 29, 2019 21:49

XD-DENG self-requested a review April 30, 2019 02:21

ashb force-pushed the warning-if-scheduler-not-running branch 2 times, most recently from f8f8682 to a5f4f22 Compare April 30, 2019 16:19

kaxil reviewed Apr 30, 2019

View reviewed changes

airflow/jobs.py Outdated Show resolved Hide resolved

kaxil reviewed Apr 30, 2019

View reviewed changes

airflow/jobs.py Outdated Show resolved Hide resolved

kaxil reviewed Apr 30, 2019

View reviewed changes

airflow/jobs.py Outdated Show resolved Hide resolved

kaxil reviewed Apr 30, 2019

View reviewed changes

airflow/macros/__init__.py Show resolved Hide resolved

ashb commented May 3, 2019

View reviewed changes

ashb force-pushed the warning-if-scheduler-not-running branch from a5f4f22 to 6add858 Compare May 3, 2019 13:03

ashb force-pushed the warning-if-scheduler-not-running branch 3 times, most recently from 17ff921 to a2b1370 Compare May 28, 2019 13:14

[AIRFLOW-4343] Show warning in UI if scheduler is not running

657a2ee

Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. New dep is BSD-2 which is Cat-A under ASF

ashb force-pushed the warning-if-scheduler-not-running branch from a2b1370 to 657a2ee Compare May 28, 2019 15:33

ashb merged commit 3dcfe28 into apache:master May 29, 2019

ashb deleted the warning-if-scheduler-not-running branch May 29, 2019 09:50

ashb added a commit to ashb/airflow that referenced this pull request Oct 10, 2019

[AIRFLOW-XXX] Add tests that got missed form apache#5127

c7203bd

Somehow in the cherry pick these tests got missed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-4343] Show warning in UI if scheduler is not running #5127

[AIRFLOW-4343] Show warning in UI if scheduler is not running #5127

ashb commented Apr 17, 2019

ashb commented Apr 17, 2019

XD-DENG commented Apr 18, 2019 •

edited

Loading

feluelle left a comment

sdikby commented Apr 24, 2019

ashb commented Apr 24, 2019

kaxil Apr 30, 2019

ashb Apr 30, 2019

kaxil Apr 30, 2019

ashb commented May 1, 2019

ashb May 3, 2019

codecov-io commented May 3, 2019 •

edited

Loading

[AIRFLOW-4343] Show warning in UI if scheduler is not running #5127

[AIRFLOW-4343] Show warning in UI if scheduler is not running #5127

Conversation

ashb commented Apr 17, 2019

Jira

Description

Tests

Commits

Documentation

Code Quality

ashb commented Apr 17, 2019

XD-DENG commented Apr 18, 2019 • edited Loading

feluelle left a comment

Choose a reason for hiding this comment

sdikby commented Apr 24, 2019

ashb commented Apr 24, 2019

kaxil Apr 30, 2019

Choose a reason for hiding this comment

ashb Apr 30, 2019

Choose a reason for hiding this comment

kaxil Apr 30, 2019

Choose a reason for hiding this comment

ashb commented May 1, 2019

ashb May 3, 2019

Choose a reason for hiding this comment

codecov-io commented May 3, 2019 • edited Loading

Codecov Report

XD-DENG commented Apr 18, 2019 •

edited

Loading

codecov-io commented May 3, 2019 •

edited

Loading