Integrate VizX into Kedro-Viz

[NeroOkwa]

Description

The goal of this ticket is to integrate VizX (a soon to be open source) tool for creating interactive visualisations such as charts and dashboards, with Kedro-Viz.

Context

Kedro-Viz allows users to visualise the relationship between data pipelines, nodes, and datasets for ML development.

Integrating Kedro-Viz with VizX would extend the utility of Kedro-Viz for users, providing an end to end solution from ML development, to operation, and data reporting.

This creates an interoperable suite of tools from data ingestion and processing, to development, and operation.

Outcome of this ticket

An integrated version of Kedro-Viz with VizX working seamlessly together.

Evidence markers

• There may be some evidence to suggest that this ticket will increase the number of translators using Kedro
• In some interviews, translators confused Kedro-Viz and VizX, or they assumed they could do VizX things in Kedro-Viz
• Translators also seem to rely on VizX dashboards in senior stakeholder conversations, and less so with Kedro-Viz

Here are quotes from 3 users with the timestamp of the interview.

First User:

• 12.26 - “I got this node that generates shap values, but the place for me to look in the shap values is to come back here and generate vizx … going to VizX to view the output which is generated from the Kedro-Viz pipeline”.
• 14:28 - “Why would the team want to spend three hours to set this up (Kedro-Viz), to show 10 seconds on the client session where the next thing they wanna do is drill down to here (VizX)”.

Second User:

• 4:41 - “We built three dashboards using VizX. One is a data input, data quality audit, kind of a dashboard. One is a feature engineering dashboard and one is a model performance dashboard”.
• 12:03 -13:00 - “So we used to use this view (Kedro Viz) in technical demos. We kind of stepped away from Kedro-Viz as far as like feature importance. We're now using the VizX dashboards for that instead, how we used to use Kedro-Viz was problem solving as a team. It's not really something that we use in PS with non-technical colleagues”.
• 15:13 - “I think the dashboards are more so what we're sharing with at least the non-technical stakeholders”.

Third User:

• 18:49 - “And then the third thing is like could you at any section in the pipeline almost launch like a BI tool like Power BI or Tableau and basically attach it to the file or whatever is after that node to allow for data exploration?”
• 20:25 - “Yeah, I think there's few purposes. One is like to understand like what the transformations have sort of done. And see if there's any unexpected outputs. I think also certain clients may be interested in certain types of outputs that aren't necessarily the transformations that are in the final table. So being able to provide those visualisations more quickly I think would be helpful”.

[NeroOkwa]

Notes from meeting with @stephkaiser, @rashidakanchwala, and @Joseph-Perkins:

Discussion

There are two high-level approaches we could take:

1. Integrating VizX in Kedro-Viz using an iframe solution.
2. Integrating VizX as a react-component in Kedro-Viz.

Next Steps

The next steps would be experimenting and testing of both options to assess the technical challenges involved.

Once the technical feasibility, risks, and tradeoffs are understood, a decision would be taken on the chosen option for implementation.