cat with start/stop ranges #744
Conversation
|
The idea is to use the
|
| class KnownPartsOfAFile(BaseCache): | ||
| name = "parts" | ||
|
|
||
| def __init__(self, blocksize, fetcher, size, data={}, **_): |
There was a problem hiding this comment.
It looks like you are assuming that the user has already read the necessary byte ranges from the file and has used them to populate the values of data before the file is open and this cache is initialized. Am I mistaken?
To clarify, I want to make sure that we can specify only the known byte ranges (not the actual data stored at those byte ranges) when we call fs.open. I would expect the necessary byte ranges to be fetched automatically from the file (using cat_ranges) when this cache is initialized. Does that make sense, or am I misunderstanding the API?
There was a problem hiding this comment.
Yes you have it right, the API as I have it here expects the data to be delivered by the caller. I could do as you suggest, and have the class fetch the ranges at instantiation, but fetcher here is not in general (or ever?) asynchronous, and in any case, we expect the reads may come from several files at once. I see your point, though, it would fit better with the current shape of the other cache classes.
|
@rjzamora , as agreed. We can iterate on this yet, since it's not very prominent in the API. |
|
I'll be happy to help add the necessary logic to leverage these new features in Dask once dask#8072 gets in. |
No description provided.