Add dashboard to see DAGs (system wide status) overview

[alexInhert]

Airflow is great in a specific DAG checks. Meaning if you know there is an issue with a specific DAG you have everything you need to check what is going on but if you have a system of 1000+ dags your needs are a bit different.
It's missing some kind of dashboard that will show what is going on in system wise manner.
For example:
TOP X dags that are still in running mode
Pools that are max occupied
last scheduler heartbeat
I can think of many other metrics

The idea here is a dashboard that will show what is going on in Airflow right now (not calculating over history).

There are many ways to get it externally with Grafana and other tools but I think it comes to a time where Airflow should provide it itself.

[junnplus]

Sound awesome. I'm showing it on grafana by reading statsd metrics, is better that if airflow can provide dashboard.

[pierrejeambrun]

I'b be glad to implement a first draft for this with basic metrics and visualization.

Don't hesitate to suggest idea or things that you would like to see in this dashboard :) (I collected a few already)

[raphaelauv]

amazing first draft !

I would suggest a graph view of the airflow "stack"

where each running components and the running task would be nodes of the graph

exemple if there 1 scheduler or more running show them

same for

triggerer(s) and worker(s) and celery backend ( in case of celery or kubernetescelery executor )

show a kubernetes node if its a kubernetesExecutor or celeryKubenertesExecutor

if the setup is localexecutor show the running task connected to the scheduler ( or even inside , to explicit that they are running next to the process )

if the setup is celery then show the running task for each worker(s)

if the setup is kubernetes show the running task

something like this ->