Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove redundant resolve_load_version call #1911

Merged
merged 8 commits into from
Oct 20, 2022
Merged

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Oct 6, 2022

Signed-off-by: Nok Chan [email protected]

Description

Related to #1691
Partially related to #1654 but I want to split this out since this is a separate bug.

Development notes

  • Remove a redundant resolve_load_version call
  • Fix the incorrect test for MatplotlibWriter
  • Similar fix for HoloviewsWriter

Note:
MatplotlibWriter doesn't support load at all. The test is likely copy&paste error, it should be a more specific pattern.

    def _load(self) -> NoReturn:
        raise DataSetError(f"Loading not supported for '{self.__class__.__name__}'")

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes

@noklam noklam self-assigned this Oct 17, 2022
@noklam noklam linked an issue Oct 17, 2022 that may be closed by this pull request
Copy link
Contributor

@SajidAlamQB SajidAlamQB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, LGTM! 👍

Comment on lines -604 to -605
def load(self) -> _DO:
self.resolve_load_version() # Make sure last load version is set
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually quite interesting - pylint complains this is a useless parent delegation since the load method is exactly the same as AbstractDataSet. But let's address this in the refactoring PR instead. It feels like this is the initial design - thus the load method was necessary but later other changes come in, it become obsolete for a long time without anyone noticing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More context of this Refactor PR of versioning in 2+ years ago

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posting the original conversation about this line:
Screenshot 2022-10-17 at 15 31 57

Isn't Lorena's point "The only other thing I can think of is that it acts as an extra protection measure that this gets called/resolved at some point in the load. At the moment it would only be called from _get_load_path, which we encourage users to call in _load but nothing stops them from forgetting to do this or doing something else entirely."
still valid?

Copy link
Contributor Author

@noklam noklam Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some notes here for reference, I have spoken to @lorenabalan to get her perspective. Thanks a lot @lorenabalan for explaining! ⭐️

@lorenabalan point

I personally wouldn’t remove it without having a refactoring plan in place

  • The function itself is not necessary - it signals that it does something more than the AbstractDataSet.
  • Users need to know a lot about the private API - and it's easy to get wrong. So singal this in the public method helps.
  • you can absolutely remove the resolve_load_version from load, but only because we know it’s called further down the chain.

@noklam view
I agree it’s nice to signal that - but I also think we agree that in general, the AbstractVersionDataSet isn’t a nice abstract class - as a result people need to copy a lot of functions. The resolve_load_version alone is not enough - in reality you almost always need this load_path = get_filepath_str(self._get_load_path(), self._protocol), so I am not convinced to keep this.

In terms of implementation, it is not doing the correct thing. The 2 failing test cases are a signal of this, they are not allowed to be loaded so it should throw a DataSetError, but because of the extra resolve_load_version call, it was throwing the VersionNotFoundError instead.
This makes me more lean to remove it now, as this is incorrect and I think it overwhelms the benefit of signaling what's the right thing to do. In reality - people are copy&pasting already

@MerelTheisenQB 's view

  • Lean to remove it now - Especially the tests are a strong sign that this indeed wasn’t the right behavior.
  • I think the refactoring of the load_version logic is a good opportunity to improve the call flow and in that way still make it clear somehow that AbstractVersionDataSet is doing more than AbstractDataSet

@noklam
Copy link
Contributor Author

noklam commented Oct 17, 2022

More context in this Refactor PR of versioning in 2+ years ago

Copy link
Member

@merelcht merelcht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating and explaining what was going on here @noklam ! ⭐ It's a great improvement that the holoviews and matplotlib datasets now return the correct error message.

@@ -176,7 +176,9 @@ def test_http_filesystem_no_versioning(self):

def test_no_versions(self, versioned_hv_writer):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should rename this and the other test to test_load_not_supported instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this will be more accurate - will do

Signed-off-by: Nok Chan <[email protected]>
@noklam noklam changed the title Expose version load version information when load_version=None Remove redundant resolve_load_version call Oct 18, 2022
@noklam noklam merged commit 4869574 into main Oct 20, 2022
@noklam noklam deleted the feat/expose-load-version branch October 20, 2022 10:11
AhdraMeraliQB pushed a commit that referenced this pull request Oct 21, 2022
* remove a redundant function call

Signed-off-by: Nok Chan <[email protected]>

* Remove redundant resolove_load_version & fix test

Signed-off-by: Nok Chan <[email protected]>

* Fix HoloviewWriter tests with more specific error message pattern & Lint

Signed-off-by: Nok Chan <[email protected]>

* Rename tests

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
amaralbf pushed a commit to amaralbf/kedro that referenced this pull request Oct 22, 2022
* remove a redundant function call

Signed-off-by: Nok Chan <[email protected]>

* Remove redundant resolove_load_version & fix test

Signed-off-by: Nok Chan <[email protected]>

* Fix HoloviewWriter tests with more specific error message pattern & Lint

Signed-off-by: Nok Chan <[email protected]>

* Rename tests

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
nickolasrm pushed a commit to ProjetaAi/kedro that referenced this pull request Oct 26, 2022
* remove a redundant function call

Signed-off-by: Nok Chan <[email protected]>

* Remove redundant resolove_load_version & fix test

Signed-off-by: Nok Chan <[email protected]>

* Fix HoloviewWriter tests with more specific error message pattern & Lint

Signed-off-by: Nok Chan <[email protected]>

* Rename tests

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: nickolasrm <[email protected]>
mle-els pushed a commit to mle-els/kedro that referenced this pull request Nov 7, 2022
* remove a redundant function call

Signed-off-by: Nok Chan <[email protected]>

* Remove redundant resolove_load_version & fix test

Signed-off-by: Nok Chan <[email protected]>

* Fix HoloviewWriter tests with more specific error message pattern & Lint

Signed-off-by: Nok Chan <[email protected]>

* Rename tests

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Minh Le <[email protected]>
AhdraMeraliQB added a commit that referenced this pull request Nov 9, 2022
* Release/0.18.3 (#1856)

* Update release version and release notes

Signed-off-by: Nok Chan <[email protected]>

* Update missing release notes

Signed-off-by: Nok Chan <[email protected]>

* update vresion

Signed-off-by: Nok Chan <[email protected]>

* update release notes

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Remove comment from code example

Signed-off-by: Ahdra Merali <[email protected]>

* Remove more comments

Signed-off-by: Ahdra Merali <[email protected]>

* Add YAML formatting

Signed-off-by: Ahdra Merali <[email protected]>

* Add missing import

Signed-off-by: Ahdra Merali <[email protected]>

* Remove even more comments

Signed-off-by: Ahdra Merali <[email protected]>

* Remove more even more comments

Signed-off-by: Ahdra Merali <[email protected]>

* Add pickle requirement to extras_require

Signed-off-by: Ahdra Merali <[email protected]>

* Try fix YAML docs

Signed-off-by: Ahdra Merali <[email protected]>

* Try fix YAML docs pt 2

Signed-off-by: Ahdra Merali <[email protected]>

* Fix code snippets in docs (#1876)

* Fix code snippets

Signed-off-by: Ahdra Merali <[email protected]>

* Separate code blocks

Signed-off-by: Ahdra Merali <[email protected]>

* Lint

Signed-off-by: Ahdra Merali <[email protected]>

Signed-off-by: Ahdra Merali <[email protected]>

* Fix issue with specifying format for SparkHiveDataSet (#1857)

Signed-off-by: jstammers <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update RELEASE.md (#1883)

* Update RELEASE.md

* fix broken link

* Update RELEASE.md

Co-authored-by: Merel Theisen <[email protected]>

Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Deprecate `kedro test` and `kedro lint` (#1873)

* Deprecating `kedro test` and `kedro lint`

Signed-off-by: Nok Chan <[email protected]>

* Deprecate commands

Signed-off-by: Nok Chan <[email protected]>

* Make kedro looks prettier

* Update Linting

Signed-off-by: Nok <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Nok <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Fix micro package pull from PyPI (#1848)

Signed-off-by: Florian Gaudin-Delrieu <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update Error message for `VersionNotFoundError` to handle Permission related issues better (#1881)

* Update message for VersionNotFoundError

Signed-off-by: Ankita Katiyar <[email protected]>

* Add test for VersionNotFoundError for cloud protocols

* Update test_data_catalog.py

Update NoVersionFoundError test

* minor linting update

* update docs link + styling changes

* Revert "update docs link + styling changes"

This reverts commit 6088e00.

* Update test with styling changes

* Update RELEASE.md

Signed-off-by: ankatiyar <[email protected]>

Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: ankatiyar <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update experiment tracking documentation with working examples (#1893)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add NHS AI Lab and ReSpo.Vision to companies list (#1878)

Signed-off-by: Ahdra Merali <[email protected]>

* Document how users can use pytest instead of kedro test (#1879)

* Add best_practices.md with introductory sections

Signed-off-by: Jannic Holzer <[email protected]>

* Add pytest and pytest-cov sections

Signed-off-by: Jannic Holzer <[email protected]>

* Add pytest-cov coverage report

Signed-off-by: Jannic Holzer <[email protected]>

* Add sections on pytest-cov

Signed-off-by: Jannic Holzer <[email protected]>

* Add automated_testing to index.rst

Signed-off-by: Jannic Holzer <[email protected]>

* Reformat third-party library names and clean grammar.

Signed-off-by: Jannic Holzer <[email protected]>

* Add link to virtual environment docs

Signed-off-by: Jannic Holzer <[email protected]>

* Add example of good test naming

Signed-off-by: Jannic Holzer <[email protected]>

* Improve link accessibility

Signed-off-by: Jannic Holzer <[email protected]>

* Improve pytest docs link accessibility

Signed-off-by: Jannic Holzer <[email protected]>

* Add reminder link to virtual environment docs

Signed-off-by: Jannic Holzer <[email protected]>

* Fix formatting in link to coverage docs

Signed-off-by: Jannic Holzer <[email protected]>

* Remove reference to /src under 'Run your tests'

Signed-off-by: Jannic Holzer <[email protected]>

* Modify references to <project_name> to <package_name>

Signed-off-by: Jannic Holzer <[email protected]>

* Fix sentence structure

Signed-off-by: Jannic Holzer <[email protected]>

* Fix broken databricks doc link

Signed-off-by: Jannic Holzer <[email protected]>

Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Capitalise Kedro-Viz in the "Visualize layers" section (#1899)

* Capitalised kedro-viz

Signed-off-by: yash6318 <[email protected]>

* capitalised Kedro viz

Signed-off-by: yash6318 <[email protected]>

* Updated set_up_experiment_tracking.md

Co-authored-by: Deepyaman Datta <[email protected]>
Signed-off-by: yash6318 <[email protected]>

Signed-off-by: yash6318 <[email protected]>
Co-authored-by: Deepyaman Datta <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Fix linting on autmated test page (#1906)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add _SINGLE_PROCESS property to CachedDataSet (#1905)

Signed-off-by: Carla Vieira <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update the tutorial of "Visualise pipelines" (#1913)

* Change a file extention to match the previous article

Signed-off-by: dinotuku <[email protected]>

* Add a missing import

Signed-off-by: dinotuku <[email protected]>

* Change both preprocessed datasets to parquet files

Signed-off-by: dinotuku <[email protected]>

* Change data type to ParquetDataSet for parquet files

Signed-off-by: dinotuku <[email protected]>

* Add a note for installing seaborn if it is not installed

Signed-off-by: dinotuku <[email protected]>

Signed-off-by: dinotuku <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Document how users can use linting tools instead of `kedro lint` (#1904)

* Add documentation for linting tools

Signed-off-by: Ankita Katiyar <[email protected]>

* Revert changes to commands_reference.md

Signed-off-by: Ankita Katiyar <[email protected]>

* Update linting docs with suggestions

Signed-off-by: Ankita Katiyar <[email protected]>

* Update linting doc

Signed-off-by: Ankita Katiyar <[email protected]>

Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Make core config accessible in dict get way  (#1870)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Create dependabot.yml configuration file for version updates (#1862)

* Create dependabot.yml configuration file

* Update dependabot.yml

Signed-off-by: SajidAlamQB <[email protected]>

* add target-branch

Signed-off-by: SajidAlamQB <[email protected]>

* Update dependabot.yml

Signed-off-by: SajidAlamQB <[email protected]>

* limit dependabot to just dependency folder

Signed-off-by: SajidAlamQB <[email protected]>

* Update test_requirements.txt

Signed-off-by: SajidAlamQB <[email protected]>

* Update MANIFEST.in

Signed-off-by: SajidAlamQB <[email protected]>

* fix e2e

Signed-off-by: SajidAlamQB <[email protected]>

* Update continue_config.yml

Signed-off-by: SajidAlamQB <[email protected]>

* Update requirements.txt

Signed-off-by: SajidAlamQB <[email protected]>

* Update requirements.txt

Signed-off-by: SajidAlamQB <[email protected]>

* fix link

Signed-off-by: SajidAlamQB <[email protected]>

* revert

Signed-off-by: SajidAlamQB <[email protected]>

* Delete requirements.txt

Signed-off-by: SajidAlamQB <[email protected]>

Signed-off-by: SajidAlamQB <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update dependabot config (#1928)

Signed-off-by: Ahdra Merali <[email protected]>

* Update robots.txt (#1929)

Signed-off-by: Ahdra Merali <[email protected]>

* fix broken link (#1950)

Signed-off-by: Ahdra Merali <[email protected]>

* Update dependabot.yml config  (#1938)

* Update dependabot.yml

Signed-off-by: SajidAlamQB <[email protected]>

* pin jupyterlab_services to requirments

Signed-off-by: SajidAlamQB <[email protected]>

* lint

Signed-off-by: SajidAlamQB <[email protected]>

Signed-off-by: SajidAlamQB <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Update setup.py Jinja2 dependencies (#1954)

Signed-off-by: Ahdra Merali <[email protected]>

* Update pip-tools requirement from ~=6.5 to ~=6.9 in /dependency (#1957)

Updates the requirements on [pip-tools](https://github.com/jazzband/pip-tools) to permit the latest version.
- [Release notes](https://github.com/jazzband/pip-tools/releases)
- [Changelog](https://github.com/jazzband/pip-tools/blob/master/CHANGELOG.md)
- [Commits](jazzband/pip-tools@6.5.0...6.9.0)

---
updated-dependencies:
- dependency-name: pip-tools
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Ahdra Merali <[email protected]>

* Update toposort requirement from ~=1.5 to ~=1.7 in /dependency (#1956)

Updates the requirements on [toposort]() to permit the latest version.

---
updated-dependencies:
- dependency-name: toposort
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sajid Alam <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Add deprecation warning to package_name argument in session create() (#1953)

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Remove redundant `resolve_load_version` call (#1911)

* remove a redundant function call

Signed-off-by: Nok Chan <[email protected]>

* Remove redundant resolove_load_version & fix test

Signed-off-by: Nok Chan <[email protected]>

* Fix HoloviewWriter tests with more specific error message pattern & Lint

Signed-off-by: Nok Chan <[email protected]>

* Rename tests

Signed-off-by: Nok Chan <[email protected]>

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>

* Make docstring in test starter match real starters (#1916)

Signed-off-by: Ahdra Merali <[email protected]>

* Try to fix formatting error

Signed-off-by: Merel Theisen <[email protected]>

* Specify pickle import

Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Signed-off-by: jstammers <[email protected]>
Signed-off-by: Nok <[email protected]>
Signed-off-by: Florian Gaudin-Delrieu <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: ankatiyar <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: yash6318 <[email protected]>
Signed-off-by: Carla Vieira <[email protected]>
Signed-off-by: dinotuku <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: SajidAlamQB <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Nok <[email protected]>
Co-authored-by: Jimmy Stammers <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: Florian Gaudin-Delrieu <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
Co-authored-by: Yetunde Dada <[email protected]>
Co-authored-by: Jannic <[email protected]>
Co-authored-by: Yash Agrawal <[email protected]>
Co-authored-by: Deepyaman Datta <[email protected]>
Co-authored-by: Carla Vieira <[email protected]>
Co-authored-by: Kuan Tung <[email protected]>
Co-authored-by: Sajid Alam <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor load version logic Exposing load/save version in hooks or included these information in logs?
3 participants