-
Notifications
You must be signed in to change notification settings - Fork 908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove redundant resolve_load_version
call
#1911
Conversation
Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Nok Chan <[email protected]>
…ad-version Signed-off-by: Nok Chan <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch, LGTM! 👍
Signed-off-by: Nok Chan <[email protected]>
def load(self) -> _DO: | ||
self.resolve_load_version() # Make sure last load version is set |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually quite interesting - pylint complains this is a useless parent delegation since the load
method is exactly the same as AbstractDataSet
. But let's address this in the refactoring PR instead. It feels like this is the initial design - thus the load
method was necessary but later other changes come in, it become obsolete for a long time without anyone noticing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More context of this Refactor PR of versioning in 2+ years ago
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Posting the original conversation about this line:
Isn't Lorena's point "The only other thing I can think of is that it acts as an extra protection measure that this gets called/resolved at some point in the load. At the moment it would only be called from _get_load_path, which we encourage users to call in _load but nothing stops them from forgetting to do this or doing something else entirely."
still valid?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some notes here for reference, I have spoken to @lorenabalan to get her perspective. Thanks a lot @lorenabalan for explaining! ⭐️
@lorenabalan point
I personally wouldn’t remove it without having a refactoring plan in place
- The function itself is not necessary - it signals that it does something more than the AbstractDataSet.
- Users need to know a lot about the private API - and it's easy to get wrong. So singal this in the public method helps.
- you can absolutely remove the
resolve_load_version
fromload
, but only because we know it’s called further down the chain.
@noklam view
I agree it’s nice to signal that - but I also think we agree that in general, the AbstractVersionDataSet
isn’t a nice abstract class - as a result people need to copy a lot of functions. The resolve_load_version
alone is not enough - in reality you almost always need this load_path = get_filepath_str(self._get_load_path(), self._protocol)
, so I am not convinced to keep this.
In terms of implementation, it is not doing the correct thing. The 2 failing test cases are a signal of this, they are not allowed to be loaded so it should throw a DataSetError
, but because of the extra resolve_load_version call, it was throwing the VersionNotFoundError
instead.
This makes me more lean to remove it now, as this is incorrect and I think it overwhelms the benefit of signaling what's the right thing to do. In reality - people are copy&pasting already
@MerelTheisenQB 's view
- Lean to remove it now - Especially the tests are a strong sign that this indeed wasn’t the right behavior.
- I think the refactoring of the
load_version
logic is a good opportunity to improve the call flow and in that way still make it clear somehow thatAbstractVersionDataSet
is doing more thanAbstractDataSet
More context in this Refactor PR of versioning in 2+ years ago |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for investigating and explaining what was going on here @noklam ! ⭐ It's a great improvement that the holoviews and matplotlib datasets now return the correct error message.
@@ -176,7 +176,9 @@ def test_http_filesystem_no_versioning(self): | |||
|
|||
def test_no_versions(self, versioned_hv_writer): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should rename this and the other test to test_load_not_supported
instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree, this will be more accurate - will do
Signed-off-by: Nok Chan <[email protected]>
load_version=None
resolve_load_version
call
* remove a redundant function call Signed-off-by: Nok Chan <[email protected]> * Remove redundant resolove_load_version & fix test Signed-off-by: Nok Chan <[email protected]> * Fix HoloviewWriter tests with more specific error message pattern & Lint Signed-off-by: Nok Chan <[email protected]> * Rename tests Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Ahdra Merali <[email protected]>
* remove a redundant function call Signed-off-by: Nok Chan <[email protected]> * Remove redundant resolove_load_version & fix test Signed-off-by: Nok Chan <[email protected]> * Fix HoloviewWriter tests with more specific error message pattern & Lint Signed-off-by: Nok Chan <[email protected]> * Rename tests Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]>
* remove a redundant function call Signed-off-by: Nok Chan <[email protected]> * Remove redundant resolove_load_version & fix test Signed-off-by: Nok Chan <[email protected]> * Fix HoloviewWriter tests with more specific error message pattern & Lint Signed-off-by: Nok Chan <[email protected]> * Rename tests Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: nickolasrm <[email protected]>
* remove a redundant function call Signed-off-by: Nok Chan <[email protected]> * Remove redundant resolove_load_version & fix test Signed-off-by: Nok Chan <[email protected]> * Fix HoloviewWriter tests with more specific error message pattern & Lint Signed-off-by: Nok Chan <[email protected]> * Rename tests Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Minh Le <[email protected]>
* Release/0.18.3 (#1856) * Update release version and release notes Signed-off-by: Nok Chan <[email protected]> * Update missing release notes Signed-off-by: Nok Chan <[email protected]> * update vresion Signed-off-by: Nok Chan <[email protected]> * update release notes Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Remove comment from code example Signed-off-by: Ahdra Merali <[email protected]> * Remove more comments Signed-off-by: Ahdra Merali <[email protected]> * Add YAML formatting Signed-off-by: Ahdra Merali <[email protected]> * Add missing import Signed-off-by: Ahdra Merali <[email protected]> * Remove even more comments Signed-off-by: Ahdra Merali <[email protected]> * Remove more even more comments Signed-off-by: Ahdra Merali <[email protected]> * Add pickle requirement to extras_require Signed-off-by: Ahdra Merali <[email protected]> * Try fix YAML docs Signed-off-by: Ahdra Merali <[email protected]> * Try fix YAML docs pt 2 Signed-off-by: Ahdra Merali <[email protected]> * Fix code snippets in docs (#1876) * Fix code snippets Signed-off-by: Ahdra Merali <[email protected]> * Separate code blocks Signed-off-by: Ahdra Merali <[email protected]> * Lint Signed-off-by: Ahdra Merali <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Fix issue with specifying format for SparkHiveDataSet (#1857) Signed-off-by: jstammers <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update RELEASE.md (#1883) * Update RELEASE.md * fix broken link * Update RELEASE.md Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Deprecate `kedro test` and `kedro lint` (#1873) * Deprecating `kedro test` and `kedro lint` Signed-off-by: Nok Chan <[email protected]> * Deprecate commands Signed-off-by: Nok Chan <[email protected]> * Make kedro looks prettier * Update Linting Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Fix micro package pull from PyPI (#1848) Signed-off-by: Florian Gaudin-Delrieu <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update Error message for `VersionNotFoundError` to handle Permission related issues better (#1881) * Update message for VersionNotFoundError Signed-off-by: Ankita Katiyar <[email protected]> * Add test for VersionNotFoundError for cloud protocols * Update test_data_catalog.py Update NoVersionFoundError test * minor linting update * update docs link + styling changes * Revert "update docs link + styling changes" This reverts commit 6088e00. * Update test with styling changes * Update RELEASE.md Signed-off-by: ankatiyar <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: ankatiyar <[email protected]> Co-authored-by: Ahdra Merali <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update experiment tracking documentation with working examples (#1893) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Add NHS AI Lab and ReSpo.Vision to companies list (#1878) Signed-off-by: Ahdra Merali <[email protected]> * Document how users can use pytest instead of kedro test (#1879) * Add best_practices.md with introductory sections Signed-off-by: Jannic Holzer <[email protected]> * Add pytest and pytest-cov sections Signed-off-by: Jannic Holzer <[email protected]> * Add pytest-cov coverage report Signed-off-by: Jannic Holzer <[email protected]> * Add sections on pytest-cov Signed-off-by: Jannic Holzer <[email protected]> * Add automated_testing to index.rst Signed-off-by: Jannic Holzer <[email protected]> * Reformat third-party library names and clean grammar. Signed-off-by: Jannic Holzer <[email protected]> * Add link to virtual environment docs Signed-off-by: Jannic Holzer <[email protected]> * Add example of good test naming Signed-off-by: Jannic Holzer <[email protected]> * Improve link accessibility Signed-off-by: Jannic Holzer <[email protected]> * Improve pytest docs link accessibility Signed-off-by: Jannic Holzer <[email protected]> * Add reminder link to virtual environment docs Signed-off-by: Jannic Holzer <[email protected]> * Fix formatting in link to coverage docs Signed-off-by: Jannic Holzer <[email protected]> * Remove reference to /src under 'Run your tests' Signed-off-by: Jannic Holzer <[email protected]> * Modify references to <project_name> to <package_name> Signed-off-by: Jannic Holzer <[email protected]> * Fix sentence structure Signed-off-by: Jannic Holzer <[email protected]> * Fix broken databricks doc link Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Capitalise Kedro-Viz in the "Visualize layers" section (#1899) * Capitalised kedro-viz Signed-off-by: yash6318 <[email protected]> * capitalised Kedro viz Signed-off-by: yash6318 <[email protected]> * Updated set_up_experiment_tracking.md Co-authored-by: Deepyaman Datta <[email protected]> Signed-off-by: yash6318 <[email protected]> Signed-off-by: yash6318 <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Fix linting on autmated test page (#1906) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Add _SINGLE_PROCESS property to CachedDataSet (#1905) Signed-off-by: Carla Vieira <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update the tutorial of "Visualise pipelines" (#1913) * Change a file extention to match the previous article Signed-off-by: dinotuku <[email protected]> * Add a missing import Signed-off-by: dinotuku <[email protected]> * Change both preprocessed datasets to parquet files Signed-off-by: dinotuku <[email protected]> * Change data type to ParquetDataSet for parquet files Signed-off-by: dinotuku <[email protected]> * Add a note for installing seaborn if it is not installed Signed-off-by: dinotuku <[email protected]> Signed-off-by: dinotuku <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Document how users can use linting tools instead of `kedro lint` (#1904) * Add documentation for linting tools Signed-off-by: Ankita Katiyar <[email protected]> * Revert changes to commands_reference.md Signed-off-by: Ankita Katiyar <[email protected]> * Update linting docs with suggestions Signed-off-by: Ankita Katiyar <[email protected]> * Update linting doc Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Make core config accessible in dict get way (#1870) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Create dependabot.yml configuration file for version updates (#1862) * Create dependabot.yml configuration file * Update dependabot.yml Signed-off-by: SajidAlamQB <[email protected]> * add target-branch Signed-off-by: SajidAlamQB <[email protected]> * Update dependabot.yml Signed-off-by: SajidAlamQB <[email protected]> * limit dependabot to just dependency folder Signed-off-by: SajidAlamQB <[email protected]> * Update test_requirements.txt Signed-off-by: SajidAlamQB <[email protected]> * Update MANIFEST.in Signed-off-by: SajidAlamQB <[email protected]> * fix e2e Signed-off-by: SajidAlamQB <[email protected]> * Update continue_config.yml Signed-off-by: SajidAlamQB <[email protected]> * Update requirements.txt Signed-off-by: SajidAlamQB <[email protected]> * Update requirements.txt Signed-off-by: SajidAlamQB <[email protected]> * fix link Signed-off-by: SajidAlamQB <[email protected]> * revert Signed-off-by: SajidAlamQB <[email protected]> * Delete requirements.txt Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update dependabot config (#1928) Signed-off-by: Ahdra Merali <[email protected]> * Update robots.txt (#1929) Signed-off-by: Ahdra Merali <[email protected]> * fix broken link (#1950) Signed-off-by: Ahdra Merali <[email protected]> * Update dependabot.yml config (#1938) * Update dependabot.yml Signed-off-by: SajidAlamQB <[email protected]> * pin jupyterlab_services to requirments Signed-off-by: SajidAlamQB <[email protected]> * lint Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Update setup.py Jinja2 dependencies (#1954) Signed-off-by: Ahdra Merali <[email protected]> * Update pip-tools requirement from ~=6.5 to ~=6.9 in /dependency (#1957) Updates the requirements on [pip-tools](https://github.com/jazzband/pip-tools) to permit the latest version. - [Release notes](https://github.com/jazzband/pip-tools/releases) - [Changelog](https://github.com/jazzband/pip-tools/blob/master/CHANGELOG.md) - [Commits](jazzband/pip-tools@6.5.0...6.9.0) --- updated-dependencies: - dependency-name: pip-tools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Ahdra Merali <[email protected]> * Update toposort requirement from ~=1.5 to ~=1.7 in /dependency (#1956) Updates the requirements on [toposort]() to permit the latest version. --- updated-dependencies: - dependency-name: toposort dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sajid Alam <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Add deprecation warning to package_name argument in session create() (#1953) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Remove redundant `resolve_load_version` call (#1911) * remove a redundant function call Signed-off-by: Nok Chan <[email protected]> * Remove redundant resolove_load_version & fix test Signed-off-by: Nok Chan <[email protected]> * Fix HoloviewWriter tests with more specific error message pattern & Lint Signed-off-by: Nok Chan <[email protected]> * Rename tests Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> * Make docstring in test starter match real starters (#1916) Signed-off-by: Ahdra Merali <[email protected]> * Try to fix formatting error Signed-off-by: Merel Theisen <[email protected]> * Specify pickle import Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> Signed-off-by: jstammers <[email protected]> Signed-off-by: Nok <[email protected]> Signed-off-by: Florian Gaudin-Delrieu <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: ankatiyar <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Jannic Holzer <[email protected]> Signed-off-by: yash6318 <[email protected]> Signed-off-by: Carla Vieira <[email protected]> Signed-off-by: dinotuku <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Nok <[email protected]> Co-authored-by: Jimmy Stammers <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Florian Gaudin-Delrieu <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]> Co-authored-by: Yetunde Dada <[email protected]> Co-authored-by: Jannic <[email protected]> Co-authored-by: Yash Agrawal <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Carla Vieira <[email protected]> Co-authored-by: Kuan Tung <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Merel Theisen <[email protected]>
Signed-off-by: Nok Chan [email protected]
Description
Related to #1691
Partially related to #1654 but I want to split this out since this is a separate bug.
Development notes
resolve_load_version
callMatplotlibWriter
HoloviewsWriter
Note:
MatplotlibWriter
doesn't support load at all. The test is likely copy&paste error, it should be a more specific pattern.Checklist
RELEASE.md
file