Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Dataset Factory Patterns to Experiment Tracking #1689

Closed
ravi-kumar-pilla opened this issue Dec 18, 2023 · 0 comments · Fixed by #1824
Closed

Add Dataset Factory Patterns to Experiment Tracking #1689

ravi-kumar-pilla opened this issue Dec 18, 2023 · 0 comments · Fixed by #1824
Assignees
Labels
Experiment Tracking Python Pull requests that update Python code

Comments

@ravi-kumar-pilla
Copy link
Contributor

ravi-kumar-pilla commented Dec 18, 2023

Experiment Tracking relies on Tracking Datasets and we create TrackingDatasetModel before we discover and get all the datasets that are defined using dataset factory pattern

Ideal way would be to implement this discovery on the kedro side, but if not possible, we need to discuss on how to discover the datasets. Earlier we used .exists() on the datasets for discovery which caused the below issues -

  1. Users were not able to use Kedro Viz if the dataset defined in the catalog does not exist (This was resolved using exception handlers)
  2. Users received a timed-out issue as the datasets resided remotely and in a distributed cluster

Parent issue - #1480

@ravi-kumar-pilla ravi-kumar-pilla converted this from a draft issue Dec 18, 2023
@rashidakanchwala rashidakanchwala moved this to Inbox in Kedro-Viz Jan 15, 2024
@rashidakanchwala rashidakanchwala moved this from Inbox to Backlog in Kedro-Viz Jan 15, 2024
@rashidakanchwala rashidakanchwala added the Python Pull requests that update Python code label Jan 16, 2024
@rashidakanchwala rashidakanchwala moved this from Backlog to Todo in Kedro-Viz Mar 18, 2024
@ravi-kumar-pilla ravi-kumar-pilla moved this from Todo to In Progress in Kedro-Viz Mar 26, 2024
@ravi-kumar-pilla ravi-kumar-pilla moved this from In Progress to In Review in Kedro-Viz Mar 27, 2024
@github-project-automation github-project-automation bot moved this from In Review to Done in Kedro-Viz Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Experiment Tracking Python Pull requests that update Python code
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants