Refactor datasets

[rashidakanchwala]

Description

Related to: #1700

Kedro-Viz and Kedro-Datasets are tightly coupled , leading to issues where modifications in Kedro-Datasets can disrupt the functionality of Kedro-Viz. This ticket aims to refactor the codebase to enhance modularity and stability. The goal is to decouple the two components, allowing for smoother integration in the future without excessive interdependencies.

Development notes

We've moved the logic for generating previews for several datasets from kedro-viz to kedro-datasets, reducing the coupling between kedro-viz and kedro-datasets.

Previews are rendered differently in the front-end, so we've introduced aliasing using NewType. Currently, we support four types of previews in the front-end: json, image [png], plotly, and dataframe [as tables]. Users can enable previews for custom datasets as long as they fall into one of these categories.

Previously, we utilized the load function and overrode it in kedro-viz to load dataset previews. However, this logic has been migrated to kedro-datasets within the preview() function. In the kedro-viz API, we no longer include plot, image, or tracking_data as fields of DataNodeMetadata. Instead, we now only send preview and preview_type. Preview can be a preview of any dataset that has the preview function, and preview_type informs the front-end on how to render the preview.

QA notes

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added new entries to the RELEASE.md file
• Added tests to cover my changes

[merelcht]

Can the PR description be updated to add some context? It will make it easier to review.

[ravi-kumar-pilla]

Hi @SajidAlamQB, The reason tests are failing for python3.8 is kedro-datasets >= 2 require python >= 3.9 . I am not sure if we want to drop viz support for python3.8 or do a conditional check for the preview feature that is added to kedro-datasets. If we support python3.8, we need to conditionally add the dataset type checks again.

@merelcht @noklam what do you suggest here ?

[noklam]

@ravi-kumar-pilla I think it's fine to drop Python 3.8 support, with the caveat that is it easy for user find out which version to install?

If an user is using Python 3.8, which combination of kedro/kedro-datasets/kedro-viz would work for them?

It's a result of kedro-plugins (at least kedro-datasets) are moving faster and we are following https://endoflife.date/python.

[ravi-kumar-pilla]

@ravi-kumar-pilla I think it's fine to drop Python 3.8 support, with the caveat that is it easy for user find out which version to install?

If an user is using Python 3.8, which combination of kedro/kedro-datasets/kedro-viz would work for them?

We need to add/update the compatibility matrix in the ReadME which includes kedro-viz/kedro/kedro-datasets.

It's a result of kedro-plugins (at least kedro-datasets) are moving faster and we are following https://endoflife.date/python.

Got it !

[noklam]

Suggested change

	def is_preview_disabled(self):
	"""Checks if the dataset has a preview disabled"""
	if self.viz_metadata:
	preview_value = self.viz_metadata.get("preview")
	return preview_value is not None and not preview_value
	return False
	def is_preview_disabled(self):
	"""Checks if the dataset has a preview disabled"""
	if self.viz_metadata:
	preview_value = self.viz_metadata.get("preview")
	return preview_value is not None and not preview_value
	return False

I found this complicated, can we assume it's True unless specified as False?

[ravi-kumar-pilla]

+1

[SajidAlamQB]

Maybe something like this:

Suggested change

	def is_preview_disabled(self):
	"""Checks if the dataset has a preview disabled"""
	if self.viz_metadata:
	preview_value = self.viz_metadata.get("preview")
	return preview_value is not None and not preview_value
	return False
	def is_preview_disabled(self):
	"""Checks if the dataset has a preview disabled"""
	return self.viz_metadata and self.viz_metadata.get("preview") is False

[ravi-kumar-pilla]

so, the preview feature is always enabled unless the user specifies viz_metadata: preview: false ? @rashidakanchwala

[rashidakanchwala]

yup, that's true. by default

[ravi-kumar-pilla]

if self.viz_metadata is None, the function will return None. Can we have a return value of boolean in all cases ? Can we do something like - return viz_metadata is not None and viz_metadata.get("preview") is False . Thank you

[rashidakanchwala]

ok have more clarity on what you and @merelcht meant. It returns None
so i have changed it to return bool as suggested-- now it will always return False except when preview: False

[jitu5]

Front end part looks good me!

[merelcht]

I left one more question and wanted to check again if the docs will also be updated before release?

Otherwise, this looks all good from a code point of view 👍

[rashidakanchwala]

I left one more question and wanted to check again if the docs will also be updated before release?

Otherwise, this looks all good from a code point of view 👍

Yes, before the release the docs will be updated. I will create a PR tomm for the docs.

[ravi-kumar-pilla]

LGTM. Awesome work !! @rashidakanchwala

We need to make sure to revert the kedro-datasets source in requirements.in, docker_requirements.txt, test_requirements.txt files after kedro-datasets release. Thank you !

[merelcht]

Thanks for clarifying my earlier question @rashidakanchwala. This looks good to be merged now ⭐ 👍

[rashidakanchwala]

Thanks team. I am pushing this, I will revert the dependencies once kedro-datasets is released. I will start on the docs now.