Skip to content

Commit

Permalink
[KED-2917] update architecture docs for graphql setup (#682)
Browse files Browse the repository at this point in the history
* updated new data flow diagram with architecture docs

* update architecture docs wording as per suggested

* added in note formatting

* minor wording changes
  • Loading branch information
studioswong authored Jan 10, 2022
1 parent 81c3bd6 commit bcaa303
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 1 deletion.
Binary file removed .github/img/app-architecture-data-flow.png
Binary file not shown.
Binary file added .github/img/app-architecuture-data-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 16 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,19 @@ The `localStorage` state is updated automatically on every Redux store update, v

![Kedro-Viz data flow diagram](/.github/img/app-architecture-data-flow.png)

On initialisation, Kedro-Viz [manually normalises pipeline data](/src/store/normalize-data.js), in order to [make immutable state updates as performant as possible](https://redux.js.org/recipes/structuring-reducers/normalizing-state-shape).
Kedro-Viz currently utilizes two different methods of data ingestion: the Redux setup for the pipeline and flowchart-view related components, and GraphQL via Apollo Client
for the experiment tracking components.

On initialisation for the Redux setup, Kedro-Viz [manually normalises pipeline data](/src/store/normalize-data.js), in order to [make immutable state updates as performant as possible](https://redux.js.org/recipes/structuring-reducers/normalizing-state-shape).

Next, it [initialises the Redux data store](https://github.com/quantumblacklabs/kedro-viz/blob/main/src/store/initial-state.js), by merging this normalised state with other data sources such as saved user preferences from `localStorage`, URL flags, and default values.

During preparation, the initial state is separated into two parts: pipeline and non-pipeline state. This is because the non-pipeline state should persist for the session duration, even if the pipeline state is reset/overwritten - i.e. if the user selects a new top-level pipeline.

All experiment tracking data are stored in a SQLite database that is generated automatically once [experiment tracking is enabled in your Kedro project](https://kedro.readthedocs.io/en/stable/08_logging/02_experiment_tracking.html), where the session store database sits under the `/data` directory of your Kedro project as `session_store.db`. On loading Kedro-Viz from the Kedro project, the Kedro-Viz backend will consume the run data stored in the database and serve the data via the GraphQL endpoint via GraphQL query requests from the Apollo client on the front end.

The server also allows updates to the database for certain fields of the run data (name, notes, etc) via mutations, while subscriptions enabled by ASGI WebSocket will be set up to allow real-time updates of run data.

## React components

React components are all to be found in `/src/components/`. The top-level React component for the standalone app is `Container`, which includes some extra code (e.g. global styles and data loading) that aren't included in the component library. The entry-point component for the library (as set by the `main` property in package.json) is `App`.
Expand All @@ -80,6 +87,8 @@ The `App` component contains the [Redux store Provider](https://react-redux.js.o

## State management

The following sections outline the Redux state management used to manage the data and app state for the flowchart view. The data for the runs view (experiment tracking features) are managed by React hooks and the Apollo GraphQL client.

![Kedro-Viz app architecture](.github/img/app-architecture.png)

Kedro-Viz uses Redux to manage state across the application. For example, the zoom level is synchronised between the MiniMap and FlowChart components by storing the current zoom level and chart dimensions in the central store, and dispatching actions to update this value. These actions first check the origin of the request before dispatching, in order to avoid a circular loop.
Expand All @@ -98,6 +107,12 @@ Selectors can be found in `/src/selectors/`. We use [Reselect](https://github.co

We have used Kedro-Viz to visualize the selector dependency graph - [visit the demo to see it in action](https://quantumblacklabs.github.io/kedro-viz/?data=selectors).

## Apollo

The `src/apollo` directory contains all the related setup for ingesting data from the GraphQL endpoint for the experiment tracking features. This includes the schema that defines all query, mutation, and subscription types, the config that sets up the Apollo Client to be used within React components, and other files containing helper functions, such as mocks to generate random data for the mock server.

Note that the schema checked in this folder is the contract that unites both the back end and front end expectations of the GraphQL endpoint, and should always be up to date with the schema of the graphQL endpoint.

## Utils

The `/src/utils/` directory contains miscellaneous reusable utility functions.
Expand Down

0 comments on commit bcaa303

Please sign in to comment.