[Versioning]: Explore Kedro + DVC for versioning #4239

ElenaKhaustova · 2024-10-17T22:25:05Z

Description

At the current stage by versioning we assume mapping a single version number to the corresponding versions of parameters, I/O data, and code. So one is able to retrieve a full project state including data at any point in time.

The goal is to check if we can use DVC to map a single version number to code, parameters, and I/O data within Kedro and how it aligns with Kedro’s workflow.

As a result, we expect a working example of kedro project used with DVC for versioning and some assumptions on:

whether it solves the main task and what are the constraints;
how easy is to set up;
how the workflow looks like;
whether any changes are required on the kedro side;
what data formats are supported;
how easy is to work with local/remote storage;
how demanding is it in terms of dependencies.

Context

#4199

Relates to: #2691 (comment)

Market research

aadarshsingh191198 · 2024-10-24T04:57:18Z

If I am not wrong, what we are asking for here is support in kedro for something similar to

dvc stage add -n train -d train.py -d data -o model.weights.h5

Ref: https://dvc.org/doc/use-cases/versioning-data-and-models/tutorial#automating-capturing

ankatiyar · 2024-11-15T14:39:48Z

Kedro + DVC integration

🔗 REPO LINK: https://github.com/ankatiyar/space-dvc

I tried to integrate DVC into my Spaceflights Kedro project to check the extent of the versioning capabilities. The steps I followed are:

Initialise the Kedro project as a git repository: git init
Initialise the Kedro project as a DVC repository: dvc init

Versioning data with `.dvc` files

Suppose, I have a dataset in my project:

companies:
  type: pandas.CSVDataset
  filepath: data/01_raw/companies.csv

dvc add data/01_raw/companies.csv -> this generates the companies.csv.dvc file which can be committed to git. Note: You would need to update the .gitignore file provided by Kedro to make this work.
git commit -m "First commit"
kedro run - Intermediate and output datasets are generated
dvc add <paths of datasets that were generated>
git add .
git commit -s -m "First run"

Scenario 1: I want to go back to a previous version of the data

Check which git commit I want to go back to
git checkout HEAD~2 data/01_raw/companies.csv.dvc
dvc checkout

Scenario 2: I want to store my data remotely

dvc remote add <name> <url>: https://dvc.org/doc/user-guide/data-management/remote-storage
kedro run: generates new data
git add . and `git commit -m "Update"
dvc push

Scenario 3: I want to go back to a previous version of the data, stored remotely

git checkout <>
dvc checkout
dvc pull

Versioning with DVC Data Pipelines

The previous way does provide enough functionality to version datasets but there's still the following to consider

Users must add all intermediate and output datasets to dvc manually
Parameters and code changes are not explicitly tracked
Can not track artefacts and metrics

Define Kedro pipelines as DVC stages

In dvc.yaml, we can define Kedro pipelines as follows

stages:
  data_processing:
    cmd: kedro run --pipeline data_processing
    deps:
      - data/01_raw/companies.csv
      - data/01_raw/reviews.csv
      - data/01_raw/shuttles.xlsx
    outs:
      - data/02_intermediate/preprocessed_companies.pq
      - data/02_intermediate/preprocessed_shuttles.pq
      - data/03_primary/model_input_table.pq
  
  data_science:
    cmd: kedro run --pipeline data_science
    deps:
      - data/03_primary/model_input_table.pq
      outs:
      - data/06_models/regressor.pickle

run the pipeline with dvc repro

Scenario: One of the datasets is updated

dvc repro will run only pipelines for which the output or the deps have changed

Scenario: I want to track code changes

Add code files to the deps in dvc.yml

stages:
  data_processing:
    cmd: kedro run --pipeline data_processing
    deps:
      - data/01_raw/companies.csv
      - data/01_raw/reviews.csv
      - data/01_raw/shuttles.xlsx
      - src/space_dvc/pipelines/data_processing/nodes.py
      - src/space_dvc/pipelines/data_processing/pipeline.py
    outs:
      - data/02_intermediate/preprocessed_companies.pq
      - data/02_intermediate/preprocessed_shuttles.pq
      - data/03_primary/model_input_table.pq

dvc repro
dvc push

Scenario: I want to track parameters

Add parameters under params in dvc.yml

data_science:
    cmd: kedro run --pipeline data_science
    deps:
      - data/03_primary/model_input_table.pq
      - src/space_dvc/pipelines/data_science/nodes.py
      - src/space_dvc/pipelines/data_science/pipeline.py
    params:
      - conf/base/parameters_data_science.yml:
        - model_options
    outs:
      - data/06_models/regressor.pickle

dvc repro
dvc push

Scenario: I want to run the experiment with different parameters

Change parameter in parameters.yml
dvc repro: This will only run data_science pipeline
dvc params diff: This will show you the comparison of parameters between runs

Experiment tracking with DVCLive

Model output can be versioned with outs in the data pipelines through the yaml and parameters can be versioned with params. DVC offers DVCLive, which is similar to MLFlow to track experiments.

Metrics and plots can be logged alongside all of the above following example code from their tutorial: https://github.com/iterative/example-get-started/blob/main/src/evaluate.py

This logging is not straightforward at the moment but can be simplified with a plugin like kedro-mlflow for dvclive.

datajoely · 2024-11-18T09:13:54Z

This is cool! Would it make sense to automate some of these commands with a pre/post hooks? also wdyt think about some commands to generate/sync the dvc.yml

ElenaKhaustova · 2024-11-18T12:50:00Z

This is cool! Would it make sense to automate some of these commands with a pre/post hooks? also wdyt think about some commands to generate/sync the dvc.yml

It's a good idea to automate some major steps to encapsulate DVC command calls from users, so they work with kedro as usual. We were thinking of making it via plugins so the DVC dependency is not mandatory and one can switch between different versioning solutions (DVC, Iceberg, etc.). In our case plugins can simply extend basic hooks as you suggest.

ankatiyar · 2024-11-18T13:51:30Z

To answer some of the questions mentioned on the ticket description:

whether it solves the main task and what are the constraints;

DVC does solve most of the versioning related issues arising from the research. With the current state of things, it's possible to:

Version datasets and revert to previous versions
Version code and parameters alongside datasets, therefore reproduce previous runs
Do basic levels of experiment tracking, i.e. log plots and metrics with DVCLive. This support could be enhanced in the future with a plugin similar to kedro-mlflow.

Challenges:

User is responsible for maintenance of the versioning data by committing any changes to VCS
If the user is using data pipelines, they need to be familiar with DVC commands because at this point, you run DVC CLI commands like dvc repro and dvc pull/push and not kedro run.
- Any arguments to kedro run will need to be updated with the dvc stage command or in the YAML file directly.
Since interaction with remote storage is handled by DVC, we're restricted by the platforms supported by them: link

how easy is to set up;

Pretty easy and their documentation is great!

how the workflow looks like;

Explained in the comment above.

whether any changes are required on the kedro side;

Not really 🤔

what data formats are supported;

DVC doesn't care about the data formats, so all data formats.

how easy is to work with local/remote storage;

Fairly simple, however, like I mentioned, we would likely be constrained by the remote storages supported by DVC.

how demanding is it in terms of dependencies.

Dependency tree for DVC

dvc==3.56.0
├── attrs [required: >=22.2.0, installed: 24.2.0]
├── celery [required: Any, installed: 5.4.0]
│   ├── billiard [required: >=4.2.0,<5.0, installed: 4.2.1]
│   ├── click [required: >=8.1.2,<9.0, installed: 8.1.7]
│   ├── click-didyoumean [required: >=0.3.0, installed: 0.3.1]
│   │   └── click [required: >=7, installed: 8.1.7]
│   ├── click-plugins [required: >=1.1.1, installed: 1.1.1]
│   │   └── click [required: >=4.0, installed: 8.1.7]
│   ├── click-repl [required: >=0.2.0, installed: 0.3.0]
│   │   ├── click [required: >=7.0, installed: 8.1.7]
│   │   └── prompt_toolkit [required: >=3.0.36, installed: 3.0.48]
│   │       └── wcwidth [required: Any, installed: 0.2.13]
│   ├── kombu [required: >=5.3.4,<6.0, installed: 5.4.2]
│   │   ├── amqp [required: >=5.1.1,<6.0.0, installed: 5.2.0]
│   │   │   └── vine [required: >=5.0.0,<6.0.0, installed: 5.1.0]
│   │   ├── tzdata [required: Any, installed: 2024.2]
│   │   └── vine [required: ==5.1.0, installed: 5.1.0]
│   ├── python-dateutil [required: >=2.8.2, installed: 2.9.0.post0]
│   │   └── six [required: >=1.5, installed: 1.16.0]
│   ├── tzdata [required: >=2022.7, installed: 2024.2]
│   └── vine [required: >=5.1.0,<6.0, installed: 5.1.0]
├── colorama [required: >=0.3.9, installed: 0.4.6]
├── configobj [required: >=5.0.6, installed: 5.0.9]
├── distro [required: >=1.3, installed: 1.9.0]
├── dpath [required: >=2.1.0,<3, installed: 2.2.0]
├── dulwich [required: Any, installed: 0.22.3]
│   └── urllib3 [required: >=1.25, installed: 2.2.3]
├── dvc-data [required: >=3.16.2,<3.17, installed: 3.16.6]
│   ├── attrs [required: >=21.3.0, installed: 24.2.0]
│   ├── dictdiffer [required: >=0.8.1, installed: 0.9.0]
│   ├── diskcache [required: >=5.2.1, installed: 5.6.3]
│   ├── dvc-objects [required: >=4.0.1,<6, installed: 5.1.0]
│   │   ├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
│   │   └── funcy [required: >=1.14, installed: 2.0]
│   ├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
│   ├── funcy [required: >=1.14, installed: 2.0]
│   ├── orjson [required: >=3,<4, installed: 3.10.9]
│   ├── pygtrie [required: >=2.3.2, installed: 2.5.0]
│   ├── sqltrie [required: >=0.11.0,<1, installed: 0.11.1]
│   │   ├── attrs [required: >=22.2.0, installed: 24.2.0]
│   │   ├── orjson [required: Any, installed: 3.10.9]
│   │   └── pygtrie [required: Any, installed: 2.5.0]
│   └── tqdm [required: >=4.63.1,<5, installed: 4.66.5]
├── dvc-http [required: >=2.29.0, installed: 2.32.0]
│   ├── aiohttp-retry [required: >=2.5.0, installed: 2.8.3]
│   │   └── aiohttp [required: Any, installed: 3.10.10]
│   │       ├── aiohappyeyeballs [required: >=2.3.0, installed: 2.4.3]
│   │       ├── aiosignal [required: >=1.1.2, installed: 1.3.1]
│   │       │   └── frozenlist [required: >=1.1.0, installed: 1.4.1]
│   │       ├── attrs [required: >=17.3.0, installed: 24.2.0]
│   │       ├── frozenlist [required: >=1.1.1, installed: 1.4.1]
│   │       ├── multidict [required: >=4.5,<7.0, installed: 6.1.0]
│   │       └── yarl [required: >=1.12.0,<2.0, installed: 1.16.0]
│   │           ├── idna [required: >=2.0, installed: 3.10]
│   │           ├── multidict [required: >=4.0, installed: 6.1.0]
│   │           └── propcache [required: >=0.2.0, installed: 0.2.0]
│   └── fsspec [required: Any, installed: 2024.10.0]
├── dvc-objects [required: Any, installed: 5.1.0]
│   ├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
│   └── funcy [required: >=1.14, installed: 2.0]
├── dvc-render [required: >=1.0.1,<2, installed: 1.0.2]
├── dvc-studio-client [required: >=0.21,<1, installed: 0.21.0]
│   ├── dulwich [required: Any, installed: 0.22.3]
│   │   └── urllib3 [required: >=1.25, installed: 2.2.3]
│   ├── requests [required: Any, installed: 2.32.3]
│   │   ├── certifi [required: >=2017.4.17, installed: 2024.8.30]
│   │   ├── charset-normalizer [required: >=2,<4, installed: 3.4.0]
│   │   ├── idna [required: >=2.5,<4, installed: 3.10]
│   │   └── urllib3 [required: >=1.21.1,<3, installed: 2.2.3]
│   └── voluptuous [required: Any, installed: 0.15.2]
├── dvc-task [required: >=0.3.0,<1, installed: 0.40.2]
│   ├── celery [required: >=5.3.0,<6, installed: 5.4.0]
│   │   ├── billiard [required: >=4.2.0,<5.0, installed: 4.2.1]
│   │   ├── click [required: >=8.1.2,<9.0, installed: 8.1.7]
│   │   ├── click-didyoumean [required: >=0.3.0, installed: 0.3.1]
│   │   │   └── click [required: >=7, installed: 8.1.7]
│   │   ├── click-plugins [required: >=1.1.1, installed: 1.1.1]
│   │   │   └── click [required: >=4.0, installed: 8.1.7]
│   │   ├── click-repl [required: >=0.2.0, installed: 0.3.0]
│   │   │   ├── click [required: >=7.0, installed: 8.1.7]
│   │   │   └── prompt_toolkit [required: >=3.0.36, installed: 3.0.48]
│   │   │       └── wcwidth [required: Any, installed: 0.2.13]
│   │   ├── kombu [required: >=5.3.4,<6.0, installed: 5.4.2]
│   │   │   ├── amqp [required: >=5.1.1,<6.0.0, installed: 5.2.0]
│   │   │   │   └── vine [required: >=5.0.0,<6.0.0, installed: 5.1.0]
│   │   │   ├── tzdata [required: Any, installed: 2024.2]
│   │   │   └── vine [required: ==5.1.0, installed: 5.1.0]
│   │   ├── python-dateutil [required: >=2.8.2, installed: 2.9.0.post0]
│   │   │   └── six [required: >=1.5, installed: 1.16.0]
│   │   ├── tzdata [required: >=2022.7, installed: 2024.2]
│   │   └── vine [required: >=5.1.0,<6.0, installed: 5.1.0]
│   ├── funcy [required: >=1.17, installed: 2.0]
│   ├── kombu [required: >=5.3.0,<6, installed: 5.4.2]
│   │   ├── amqp [required: >=5.1.1,<6.0.0, installed: 5.2.0]
│   │   │   └── vine [required: >=5.0.0,<6.0.0, installed: 5.1.0]
│   │   ├── tzdata [required: Any, installed: 2024.2]
│   │   └── vine [required: ==5.1.0, installed: 5.1.0]
│   └── shortuuid [required: >=1.0.8, installed: 1.0.13]
├── flatten-dict [required: >=0.4.1,<1, installed: 0.4.2]
│   └── six [required: >=1.12,<2.0, installed: 1.16.0]
├── flufl.lock [required: >=8.1.0,<9, installed: 8.1.0]
│   ├── atpublic [required: Any, installed: 5.0]
│   └── psutil [required: Any, installed: 6.1.0]
├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
├── funcy [required: >=1.14, installed: 2.0]
├── grandalf [required: >=0.7,<1, installed: 0.8]
│   └── pyparsing [required: Any, installed: 3.2.0]
├── gto [required: >=1.6.0,<2, installed: 1.7.1]
│   ├── entrypoints [required: Any, installed: 0.4]
│   ├── funcy [required: Any, installed: 2.0]
│   ├── pydantic [required: >=1.9.0,<3,!=2.0.0, installed: 2.9.2]
│   │   ├── annotated-types [required: >=0.6.0, installed: 0.7.0]
│   │   ├── pydantic_core [required: ==2.23.4, installed: 2.23.4]
│   │   │   └── typing_extensions [required: >=4.6.0,!=4.7.0, installed: 4.12.2]
│   │   └── typing_extensions [required: >=4.6.1, installed: 4.12.2]
│   ├── rich [required: Any, installed: 13.9.2]
│   │   ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0]
│   │   │   └── mdurl [required: ~=0.1, installed: 0.1.2]
│   │   └── Pygments [required: >=2.13.0,<3.0.0, installed: 2.18.0]
│   ├── ruamel.yaml [required: Any, installed: 0.18.6]
│   │   └── ruamel.yaml.clib [required: >=0.2.7, installed: 0.2.12]
│   ├── scmrepo [required: >=3,<4, installed: 3.3.8]
│   │   ├── aiohttp-retry [required: >=2.5.0, installed: 2.8.3]
│   │   │   └── aiohttp [required: Any, installed: 3.10.10]
│   │   │       ├── aiohappyeyeballs [required: >=2.3.0, installed: 2.4.3]
│   │   │       ├── aiosignal [required: >=1.1.2, installed: 1.3.1]
│   │   │       │   └── frozenlist [required: >=1.1.0, installed: 1.4.1]
│   │   │       ├── attrs [required: >=17.3.0, installed: 24.2.0]
│   │   │       ├── frozenlist [required: >=1.1.1, installed: 1.4.1]
│   │   │       ├── multidict [required: >=4.5,<7.0, installed: 6.1.0]
│   │   │       └── yarl [required: >=1.12.0,<2.0, installed: 1.16.0]
│   │   │           ├── idna [required: >=2.0, installed: 3.10]
│   │   │           ├── multidict [required: >=4.0, installed: 6.1.0]
│   │   │           └── propcache [required: >=0.2.0, installed: 0.2.0]
│   │   ├── asyncssh [required: >=2.13.1,<3, installed: 2.17.0]
│   │   │   ├── cryptography [required: >=39.0, installed: 43.0.3]
│   │   │   │   └── cffi [required: >=1.12, installed: 1.17.1]
│   │   │   │       └── pycparser [required: Any, installed: 2.22]
│   │   │   └── typing_extensions [required: >=4.0.0, installed: 4.12.2]
│   │   ├── dulwich [required: >=0.22.1, installed: 0.22.3]
│   │   │   └── urllib3 [required: >=1.25, installed: 2.2.3]
│   │   ├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
│   │   ├── funcy [required: >=1.14, installed: 2.0]
│   │   ├── GitPython [required: >3, installed: 3.1.43]
│   │   │   └── gitdb [required: >=4.0.1,<5, installed: 4.0.11]
│   │   │       └── smmap [required: >=3.0.1,<6, installed: 5.0.1]
│   │   ├── pathspec [required: >=0.9.0, installed: 0.12.1]
│   │   ├── pygit2 [required: >=1.14.0, installed: 1.16.0]
│   │   │   └── cffi [required: >=1.17.0, installed: 1.17.1]
│   │   │       └── pycparser [required: Any, installed: 2.22]
│   │   ├── pygtrie [required: >=2.3.2, installed: 2.5.0]
│   │   └── tqdm [required: Any, installed: 4.66.5]
│   ├── semver [required: >=2.13.0, installed: 3.0.2]
│   ├── tabulate [required: >=0.8.10, installed: 0.9.0]
│   └── typer [required: >=0.4.1, installed: 0.12.5]
│       ├── click [required: >=8.0.0, installed: 8.1.7]
│       ├── rich [required: >=10.11.0, installed: 13.9.2]
│       │   ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0]
│       │   │   └── mdurl [required: ~=0.1, installed: 0.1.2]
│       │   └── Pygments [required: >=2.13.0,<3.0.0, installed: 2.18.0]
│       ├── shellingham [required: >=1.3.0, installed: 1.5.4]
│       └── typing_extensions [required: >=3.7.4.3, installed: 4.12.2]
├── hydra-core [required: >=1.1, installed: 1.3.2]
│   ├── antlr4-python3-runtime [required: ==4.9.*, installed: 4.9.3]
│   ├── omegaconf [required: >=2.2,<2.4, installed: 2.3.0]
│   │   ├── antlr4-python3-runtime [required: ==4.9.*, installed: 4.9.3]
│   │   └── PyYAML [required: >=5.1.0, installed: 6.0.2]
│   └── packaging [required: Any, installed: 24.1]
├── iterative-telemetry [required: >=0.0.7, installed: 0.0.9]
│   ├── appdirs [required: Any, installed: 1.4.4]
│   ├── distro [required: Any, installed: 1.9.0]
│   ├── filelock [required: Any, installed: 3.16.1]
│   └── requests [required: Any, installed: 2.32.3]
│       ├── certifi [required: >=2017.4.17, installed: 2024.8.30]
│       ├── charset-normalizer [required: >=2,<4, installed: 3.4.0]
│       ├── idna [required: >=2.5,<4, installed: 3.10]
│       └── urllib3 [required: >=1.21.1,<3, installed: 2.2.3]
├── kombu [required: Any, installed: 5.4.2]
│   ├── amqp [required: >=5.1.1,<6.0.0, installed: 5.2.0]
│   │   └── vine [required: >=5.0.0,<6.0.0, installed: 5.1.0]
│   ├── tzdata [required: Any, installed: 2024.2]
│   └── vine [required: ==5.1.0, installed: 5.1.0]
├── networkx [required: >=2.5, installed: 3.4.2]
├── omegaconf [required: Any, installed: 2.3.0]
│   ├── antlr4-python3-runtime [required: ==4.9.*, installed: 4.9.3]
│   └── PyYAML [required: >=5.1.0, installed: 6.0.2]
├── packaging [required: >=19, installed: 24.1]
├── pathspec [required: >=0.10.3, installed: 0.12.1]
├── platformdirs [required: >=3.1.1,<4, installed: 3.11.0]
├── psutil [required: >=5.8, installed: 6.1.0]
├── pydot [required: >=1.2.4, installed: 3.0.2]
│   └── pyparsing [required: >=3.0.9, installed: 3.2.0]
├── pygtrie [required: >=2.3.2, installed: 2.5.0]
├── pyparsing [required: >=2.4.7, installed: 3.2.0]
├── requests [required: >=2.22, installed: 2.32.3]
│   ├── certifi [required: >=2017.4.17, installed: 2024.8.30]
│   ├── charset-normalizer [required: >=2,<4, installed: 3.4.0]
│   ├── idna [required: >=2.5,<4, installed: 3.10]
│   └── urllib3 [required: >=1.21.1,<3, installed: 2.2.3]
├── rich [required: >=12, installed: 13.9.2]
│   ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0]
│   │   └── mdurl [required: ~=0.1, installed: 0.1.2]
│   └── Pygments [required: >=2.13.0,<3.0.0, installed: 2.18.0]
├── ruamel.yaml [required: >=0.17.11, installed: 0.18.6]
│   └── ruamel.yaml.clib [required: >=0.2.7, installed: 0.2.12]
├── scmrepo [required: >=3.3.8,<4, installed: 3.3.8]
│   ├── aiohttp-retry [required: >=2.5.0, installed: 2.8.3]
│   │   └── aiohttp [required: Any, installed: 3.10.10]
│   │       ├── aiohappyeyeballs [required: >=2.3.0, installed: 2.4.3]
│   │       ├── aiosignal [required: >=1.1.2, installed: 1.3.1]
│   │       │   └── frozenlist [required: >=1.1.0, installed: 1.4.1]
│   │       ├── attrs [required: >=17.3.0, installed: 24.2.0]
│   │       ├── frozenlist [required: >=1.1.1, installed: 1.4.1]
│   │       ├── multidict [required: >=4.5,<7.0, installed: 6.1.0]
│   │       └── yarl [required: >=1.12.0,<2.0, installed: 1.16.0]
│   │           ├── idna [required: >=2.0, installed: 3.10]
│   │           ├── multidict [required: >=4.0, installed: 6.1.0]
│   │           └── propcache [required: >=0.2.0, installed: 0.2.0]
│   ├── asyncssh [required: >=2.13.1,<3, installed: 2.17.0]
│   │   ├── cryptography [required: >=39.0, installed: 43.0.3]
│   │   │   └── cffi [required: >=1.12, installed: 1.17.1]
│   │   │       └── pycparser [required: Any, installed: 2.22]
│   │   └── typing_extensions [required: >=4.0.0, installed: 4.12.2]
│   ├── dulwich [required: >=0.22.1, installed: 0.22.3]
│   │   └── urllib3 [required: >=1.25, installed: 2.2.3]
│   ├── fsspec [required: >=2024.2.0, installed: 2024.10.0]
│   ├── funcy [required: >=1.14, installed: 2.0]
│   ├── GitPython [required: >3, installed: 3.1.43]
│   │   └── gitdb [required: >=4.0.1,<5, installed: 4.0.11]
│   │       └── smmap [required: >=3.0.1,<6, installed: 5.0.1]
│   ├── pathspec [required: >=0.9.0, installed: 0.12.1]
│   ├── pygit2 [required: >=1.14.0, installed: 1.16.0]
│   │   └── cffi [required: >=1.17.0, installed: 1.17.1]
│   │       └── pycparser [required: Any, installed: 2.22]
│   ├── pygtrie [required: >=2.3.2, installed: 2.5.0]
│   └── tqdm [required: Any, installed: 4.66.5]
├── shortuuid [required: >=0.5, installed: 1.0.13]
├── shtab [required: >=1.3.4,<2, installed: 1.7.1]
├── tabulate [required: >=0.8.7, installed: 0.9.0]
├── tomlkit [required: >=0.11.1, installed: 0.13.2]
├── tqdm [required: >=4.63.1,<5, installed: 4.66.5]
├── voluptuous [required: >=0.11.7, installed: 0.15.2]
└── zc.lockfile [required: >=1.2.1, installed: 3.0.post1]
    └── setuptools [required: Any, installed: 75.1.0]

merelcht · 2025-02-17T15:26:15Z

I guess we can close this now the docs have been written right? @astrojuanlu

astrojuanlu · 2025-02-17T17:04:24Z

Yes!

ElenaKhaustova added the Issue: Feature Request New feature or improvement to existing feature label Oct 17, 2024

ElenaKhaustova assigned ElenaKhaustova and ankatiyar Oct 17, 2024

ElenaKhaustova removed the Issue: Feature Request New feature or improvement to existing feature label Oct 17, 2024

merelcht added this to the Dataset Versioning milestone Oct 18, 2024

github-actions bot mentioned this issue Nov 1, 2024

Monthly issue metrics report #4280

Closed

astrojuanlu mentioned this issue Nov 16, 2024

Document usage of Kedro + DVC #2691

Closed

ElenaKhaustova mentioned this issue Nov 18, 2024

[Versioning]: Explore Kedro + Iceberg for versioning #4241

Closed

ankatiyar mentioned this issue Feb 5, 2025

ability to kedro run --version 'label' #4028

Open

astrojuanlu closed this as completed Feb 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Versioning]: Explore Kedro + DVC for versioning #4239

[Versioning]: Explore Kedro + DVC for versioning #4239

ElenaKhaustova commented Oct 17, 2024 •

edited

Loading

aadarshsingh191198 commented Oct 24, 2024 •

edited

Loading

ankatiyar commented Nov 15, 2024 •

edited

Loading

datajoely commented Nov 18, 2024

ElenaKhaustova commented Nov 18, 2024

ankatiyar commented Nov 18, 2024 •

edited by astrojuanlu

Loading

merelcht commented Feb 17, 2025

astrojuanlu commented Feb 17, 2025

[Versioning]: Explore Kedro + DVC for versioning #4239

[Versioning]: Explore Kedro + DVC for versioning #4239

Comments

ElenaKhaustova commented Oct 17, 2024 • edited Loading

Description

Context

aadarshsingh191198 commented Oct 24, 2024 • edited Loading

ankatiyar commented Nov 15, 2024 • edited Loading

Kedro + DVC integration

Versioning data with .dvc files

Scenario 1: I want to go back to a previous version of the data

Scenario 2: I want to store my data remotely

Scenario 3: I want to go back to a previous version of the data, stored remotely

Versioning with DVC Data Pipelines

Define Kedro pipelines as DVC stages

Scenario: One of the datasets is updated

Scenario: I want to track code changes

Scenario: I want to track parameters

Scenario: I want to run the experiment with different parameters

Experiment tracking with DVCLive

datajoely commented Nov 18, 2024

ElenaKhaustova commented Nov 18, 2024

ankatiyar commented Nov 18, 2024 • edited by astrojuanlu Loading

merelcht commented Feb 17, 2025

astrojuanlu commented Feb 17, 2025

ElenaKhaustova commented Oct 17, 2024 •

edited

Loading

aadarshsingh191198 commented Oct 24, 2024 •

edited

Loading

ankatiyar commented Nov 15, 2024 •

edited

Loading

Versioning data with `.dvc` files

ankatiyar commented Nov 18, 2024 •

edited by astrojuanlu

Loading