Prefetch page index (#4090) #4216
Conversation
There was a problem hiding this comment.
Thank you @tustvold -- I went through this PR carefully and it looks really nice 👌
I wonder if the same basic pattern could be applied to the Bloom filters as well, or if they suffer from the issue that they don't actually appear in the footer 🤔
cc @thinkharderdev and @Ted-Jiang
| /// the last 8 bytes to determine the footer's precise length, before | ||
| /// issuing a second request to fetch the metadata bytes | ||
| /// | ||
| /// If a `prefetch` is `Some`, this will read the specified number of bytes |
| let mut loader = MetadataLoader::load(f, len, Some(130650)).await.unwrap(); | ||
| assert_eq!(fetch_count.load(Ordering::SeqCst), 1); | ||
| loader.load_page_index(true, true).await.unwrap(); | ||
| assert_eq!(fetch_count.load(Ordering::SeqCst), 1); |
Thanks for ping me, i will review this carefully today. |
Yes, the design of this was with them in mind. Bloom filters can be stored at the end of the file, which would allow prefetching to help, I'm not sure the writer currently does this though |
Which issue does this PR close?
Closes #4090
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
This adds
Sendconstraints tofetch_parquet_metadata, this is unlikely to trip people up in practice