New data store preload API #1097
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
konstntokas
reviewed
Dec 18, 2024
xcube/core/store/preload.py
Outdated
| for data_id, future in self._futures.items(): | ||
| if f is future: | ||
| break | ||
| if data_id is None: |
There was a problem hiding this comment.
I search for the data_id that belongs to the passed future f.
I'll add a comment.
Co-authored-by: Konstantin Ntokas <[email protected]>
Co-authored-by: Konstantin Ntokas <[email protected]>
konstntokas
approved these changes
Dec 18, 2024
b-yogesh
requested changes
Dec 18, 2024
| @@ -0,0 +1,368 @@ | |||
| { | |||
There was a problem hiding this comment.
We can also add the context manager example in the notebook.
konstntokas
reviewed
Dec 19, 2024
| def __init__( | ||
| self, | ||
| data_ids: tuple[str, ...], | ||
| preload_data: Callable[[PreloadHandle, str], None] | None = None, |
There was a problem hiding this comment.
Maybe it is useful to allow preload_params in the preload_data function.
E.g. I had the case, where I had multiple tiff files within a zip file which could be merged if the user requests it by a keyword boolean.
There was a problem hiding this comment.
I don't understand the use case. Why don't you just add a dedicated opener parameter then?
You could also use functools.partial().
b-yogesh
reviewed
Dec 19, 2024
b-yogesh
approved these changes
Dec 27, 2024
Co-authored-by: b-yogesh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new preload API to xcube data stores:
xcube.core.store.DataStoreclass to optionally supportpreloading of datasets via an API represented by the
new
xcube.core.store.DataPreloaderinterface.NullPreloadHandleandExecutorPreloadHandleto be returned by implementations of the
prepare_data()method of agiven data store.
Closes #1093
Checklist:
New/modified features documented indocs/source/*CHANGES.md