Catalog to config prototype #4323

ElenaKhaustova · 2024-11-12T16:45:03Z

Description

Made on top of #4347 (review it first)

Implementation of #4329

TODO:

Exclude parameters
Add docstrings and release notes
Open related tickets:
- Discrepancy between setting save_version via catalog constructor and when passing datasets #4327, PR: Validate datasets versions #4347
- Inconsistency when setting version via versioned flag and dataset parameter #4326
- tickets for datasets required extending parent AbstractDataset.to_config()

Development notes

To run pytest tests/io/test_kedro_data_catalog.py::TestKedroDataCatalog::TestKedroDataCatalogFromConfig or see an example from How to test section in #4329

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Elena Khaustova <[email protected]>

merelcht

I see docstrings are still on your todo list, will be great to have these! Otherwise this looks all good 👍

kedro/io/core.py

Signed-off-by: Elena Khaustova <[email protected]>

…catalog-from-to-prototype Signed-off-by: Elena Khaustova <[email protected]>

Signed-off-by: Elena Khaustova <[email protected]>

astrojuanlu · 2024-11-25T06:58:57Z

kedro/io/core.py

+        init_func: Callable = cls.__init__
+
+        def init_decorator(previous_init: Callable) -> Callable:
+            def new_init(self, *args, **kwargs) -> None:  # type: ignore[no-untyped-def]
+                previous_init(self, *args, **kwargs)
+                if type(self) is cls:
+                    call_args = getcallargs(init_func, self, *args, **kwargs)
+                    self.__post_init__(call_args)
+
+            return new_init
+
+        cls.__init__ = init_decorator(cls.__init__)  # type: ignore[method-assign]


Worth adding a comment explaining what this does and why is it here?

This is in TODO 👍

astrojuanlu

Is it worth splitting the version validation PR separately? Or is it tied to the "catalog to config" prototype in any way?

Signed-off-by: Elena Khaustova <[email protected]>

…catalog-from-to-prototype

Signed-off-by: Elena Khaustova <[email protected]>

ElenaKhaustova · 2024-11-27T14:36:44Z

Is it worth splitting the version validation PR separately? Or is it tied to the "catalog to config" prototype in any way?

#4347

Signed-off-by: Elena Khaustova <[email protected]>

ElenaKhaustova added 21 commits November 7, 2024 12:26

Captured init arguments

ae5384a

Signed-off-by: Elena Khaustova <[email protected]>

Implemented unresoloving credentials

fcdf357

Signed-off-by: Elena Khaustova <[email protected]>

Added some comments

b147374

Signed-off-by: Elena Khaustova <[email protected]>

Put type in first place for dataset config

0ed0c1e

Signed-off-by: Elena Khaustova <[email protected]>

Handled version key

3c839a9

Signed-off-by: Elena Khaustova <[email protected]>

Added lazy dataset to_config

29bb714

Signed-off-by: Elena Khaustova <[email protected]>

Removed data key from MemoryDataset

49858b6

Signed-off-by: Elena Khaustova <[email protected]>

Added TODOs

0d0ba91

Signed-off-by: Elena Khaustova <[email protected]>

Saved call args

8413f58

Signed-off-by: Elena Khaustova <[email protected]>

Saved only set credentials

a89db7e

Signed-off-by: Elena Khaustova <[email protected]>

Processed CachedDataset case

b081c65

Signed-off-by: Elena Khaustova <[email protected]>

Updated TODOs

18c0ad6

Signed-off-by: Elena Khaustova <[email protected]>

Tested with PartitionedDataset

2751ea8

Signed-off-by: Elena Khaustova <[email protected]>

Popped metadata

fc576ff

Signed-off-by: Elena Khaustova <[email protected]>

Fixed versioning when load

1d6454c

Signed-off-by: Elena Khaustova <[email protected]>

Fixed linter

8c31237

Signed-off-by: Elena Khaustova <[email protected]>

Tested datasets factories

e035881

Signed-off-by: Elena Khaustova <[email protected]>

Tested transcoding

edcdc38

Signed-off-by: Elena Khaustova <[email protected]>

Removed TODOs

15a1e72

Signed-off-by: Elena Khaustova <[email protected]>

Removed debug output

d4e4534

Signed-off-by: Elena Khaustova <[email protected]>

Removed debug output

e7e8af5

Signed-off-by: Elena Khaustova <[email protected]>

ElenaKhaustova changed the title ~~Catalog from/to dict prototype~~ Catalog to config prototype Nov 13, 2024

ElenaKhaustova mentioned this pull request Nov 13, 2024

[DataCatalog]: Catalog to config #4329

Open

ElenaKhaustova added 7 commits November 14, 2024 11:50

Added logic to set VERSIONED_FLAG_KEY

1b6be8e

Signed-off-by: Elena Khaustova <[email protected]>

Merge branch 'main' into feature/3932-catalog-from-to-prototype

db77436

Updated version set up

c6dc380

Signed-off-by: Elena Khaustova <[email protected]>

Added TODO for versioning

54b0793

Signed-off-by: Elena Khaustova <[email protected]>

Added tests for unresolve_config_credentials

f183e60

Signed-off-by: Elena Khaustova <[email protected]>

Implemented test_to_config

0d9d241

Signed-off-by: Elena Khaustova <[email protected]>

Added test with MemoryDataset

763e635

Signed-off-by: Elena Khaustova <[email protected]>

Renamed catalog config resolver methods

c37c04d

Signed-off-by: Elena Khaustova <[email protected]>

merelcht approved these changes Nov 21, 2024

View reviewed changes

kedro/io/core.py Outdated Show resolved Hide resolved

kedro/io/core.py Outdated Show resolved Hide resolved

ElenaKhaustova added 13 commits November 21, 2024 17:11

Implemented _validate_versions method

591f4a0

Signed-off-by: Elena Khaustova <[email protected]>

Added _validate_versions calls

5aaebe6

Signed-off-by: Elena Khaustova <[email protected]>

Updated error descriptions

bdb7cf6

Signed-off-by: Elena Khaustova <[email protected]>

Added validation to the old catalog

e2ffeaa

Signed-off-by: Elena Khaustova <[email protected]>

Fixed linter

6b1e802

Signed-off-by: Elena Khaustova <[email protected]>

Implemented unit tests for KedroDataCatalog

06e343b

Signed-off-by: Elena Khaustova <[email protected]>

Removed odd comments

5492b9f

Signed-off-by: Elena Khaustova <[email protected]>

Implemented tests for DataCatalog

c96546c

Signed-off-by: Elena Khaustova <[email protected]>

Added docstrings

46f2df6

Signed-off-by: Elena Khaustova <[email protected]>

Added release notes

56a067c

Signed-off-by: Elena Khaustova <[email protected]>

Merge branch 'fix/4327-validate-datasets-versions' into feature/3932-…

cbd1d4a

…catalog-from-to-prototype Signed-off-by: Elena Khaustova <[email protected]>

Updated version logic

e9027b9

Signed-off-by: Elena Khaustova <[email protected]>

Added CachedDataset case

11b148b

Signed-off-by: Elena Khaustova <[email protected]>

astrojuanlu reviewed Nov 25, 2024

View reviewed changes

ElenaKhaustova and others added 10 commits November 27, 2024 10:09

Merge branch 'main' into fix/4327-validate-datasets-versions

163ca17

Updated release notes

ca2ac6c

Signed-off-by: Elena Khaustova <[email protected]>

Added tests for CachedDataset use case

615d135

Signed-off-by: Elena Khaustova <[email protected]>

Merge branch 'fix/4327-validate-datasets-versions' into feature/3932-…

8dd1084

…catalog-from-to-prototype

Updated unit test after version validation is applied

8a01881

Signed-off-by: Elena Khaustova <[email protected]>

Removed MemoryDatasets

eb44a30

Signed-off-by: Elena Khaustova <[email protected]>

Removed _is_parameter

ba3d04e

Signed-off-by: Elena Khaustova <[email protected]>

Pop metadata from cached dataset configuration

35953a9

Signed-off-by: Elena Khaustova <[email protected]>

Fixed lint

d56793b

Signed-off-by: Elena Khaustova <[email protected]>

Fixed unit test

ebf1483

Signed-off-by: Elena Khaustova <[email protected]>

ElenaKhaustova marked this pull request as draft November 27, 2024 15:47

Added docstrings for AbstractDataset.to_config()

f5468c9

Signed-off-by: Elena Khaustova <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Catalog to config prototype #4323

Catalog to config prototype #4323

ElenaKhaustova commented Nov 12, 2024 •

edited

Loading

merelcht left a comment

astrojuanlu Nov 25, 2024

ElenaKhaustova Nov 27, 2024

astrojuanlu left a comment

ElenaKhaustova commented Nov 27, 2024

Catalog to config prototype #4323

Are you sure you want to change the base?

Catalog to config prototype #4323

Conversation

ElenaKhaustova commented Nov 12, 2024 • edited Loading

Description

TODO:

Development notes

Developer Certificate of Origin

Checklist

merelcht left a comment

Choose a reason for hiding this comment

astrojuanlu Nov 25, 2024

Choose a reason for hiding this comment

ElenaKhaustova Nov 27, 2024

Choose a reason for hiding this comment

astrojuanlu left a comment

Choose a reason for hiding this comment

ElenaKhaustova commented Nov 27, 2024

ElenaKhaustova commented Nov 12, 2024 •

edited

Loading