Add option to hide Datasets in Graph View

[ttzhou]

Description

If we have a lot of datasets generated by tasks in a DAG, it can clutter up the Graph view, where sometimes we just want to see the relations between the actual tasks.

Is it possible to add an option (either some kind of toggle in webserver UI and/or an environment variable) that can allow us to hide all the datasets in the Graph view?

Use case/motivation

The hope is that we allow users the ability to reduce clutter in their Graph view when trying to debug/trace lineage of tasks, when many dataset blocks are present.

Related issues

I don't think so, but would be very happy to close this if there already is one!

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[ttzhou]

I may take a stab at a PR when I have some time, but wanting to track my thoughts with this issue.

[uranusjr]

I think we should also have an option to hide the DAGs too (the other way around). cc @bbovenzi sicne I think there’s some planned improvements on the view for 3.0.

[bbovenzi]

Yeah, both graphs can have a lot more parameters. Some thoughts:

DAG graph view:

• add sensors and triggers as external dependencies like we already have for datasets
• have the backend pass dataset conditions and dataset aliases as nodes and edges without the UI having to calculate it
• include a show_external_deps param, when it is true the graph will show datasets/aliases/sensors/triggers. Then the UI can also include a button to toggle datasets on/off

Datasets Dependency Graph

• filter_by param to pass a dag_id, dataset alias name, or a dataset uri to only show the graph connected in some way to that graph
• only_datasets to only render datasets and aliases to see how data flows without worrying about what dags are involved
• add dataset conditions nodes like the DAG graph now has
• add a new endpoint with an event (dag_id + run_id or uri + timestamp) and it will return all of the events upstream and downstream of that event.