Explore integration with OpenLineage

[astrojuanlu]

OpenLineage enables consistent collection of lineage metadata, creating a deeper understanding of how data is produced and used.

https://openlineage.io/

Reference implementation: Marquez https://github.com/MarquezProject/marquez

Validation:

With the release of Airflow 2.7.0, OpenLineage became a built-in feature of Airflow.

What does this mean for us? Should Kedro-Viz consume OpenLineage exports coming from Kedro? (See previous discussion at kedro-org/kedro-viz#1833 (comment))

[astrojuanlu]

A user asked

Is there any way to show log in Databricks for running pipelines/nodes execution such as time spending to complete specific node, memory usage, or cost in graphical manner like the image below?

I was just wondering there are any ways to visualize pipeline execution results/logs in graphical manner like for example, this node takes this much time to complete, etc.

Exporting OL data addresses this.

[astrojuanlu]

Another user:

What we are trying to capture and track is column-level spark transformations across the whole kedro pipeline

(hinted that only Kedro-related lineage would likely be insufficient)