[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570

Ted-Jiang · 2023-09-15T10:05:38Z

Which issue does this PR close?

Closes #7556 .

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

datafusion/core/src/datasource/listing/table.rs

Ted-Jiang · 2023-09-15T10:06:59Z

datafusion/core/src/datasource/listing/table.rs

@@ -1092,6 +1092,95 @@ mod tests {
        Ok(())
    }

+    #[tokio::test]
+    async fn load_table_stats_with_session_level_cache() -> Result<()> {
+        let testdata = crate::test_util::parquet_test_data();


Add test to check cache share in session level

Nice -- thank you. Since this is an end to end test, I recommend moving it to somewhere in core_integration: datafusion/core/tests/core_integration.rs perhaps

datafusion/execution/src/cache/cache_manager.rs

datafusion/execution/src/cache/cache_unit.rs

alamb

Thank you @Ted-Jiang -- this is looking really nice. I left some various clean up comments but I think this PR looks very nice and is well commented and structured 🏆

datafusion/execution/src/cache/mod.rs

alamb · 2023-09-15T16:45:01Z

datafusion/execution/src/cache/mod.rs

+// The cache accessor, users usually working on this interface while manipulating caches
+pub trait CacheAccessor<K, V>: Send + Sync {
+    // Extra info but not part of the cache key or cache value.
+    type Extra: Clone;


Can you explain what the usecase for Extra is? Specifically I wonder why such information could not be added as a field to the Value

Like in default FileStatisticsCache, get func need check last_modified from ObjectMeta which not impl Hash so can not be part of the key, we need put this info in Extra.

I see -- that makes sense -- it might help to document the rationale in statistics

alamb · 2023-09-15T16:46:07Z

datafusion/execution/src/cache/cache_unit.rs

+    }
+
+    /// Get `Statistics` for file location. Returns None if file has changed or not found.
+    fn get_with_extra(&self, k: &Path, e: &Self::Extra) -> Option<Statistics> {


as written this is going to copy the statistics (though I realize that is what this PR did previously) -- maybe we could use something like Arc<Statistics> to store the statistics.

Yes , thanks for point this out

datafusion/core/src/datasource/listing/table.rs

alamb · 2023-09-15T17:18:31Z

datafusion/core/src/datasource/listing/table.rs

@@ -1092,6 +1092,95 @@ mod tests {
        Ok(())
    }

+    #[tokio::test]
+    async fn load_table_stats_with_session_level_cache() -> Result<()> {
+        let testdata = crate::test_util::parquet_test_data();


Nice -- thank you. Since this is an end to end test, I recommend moving it to somewhere in core_integration: datafusion/core/tests/core_integration.rs perhaps

datafusion/execution/src/cache/cache_manager.rs

alamb · 2023-09-15T17:24:34Z

datafusion/execution/src/cache/cache_manager.rs

+}
+
+impl CacheManagerConfig {
+    pub fn enable_table_files_statistics_cache(mut self, cache: FileStaticCache) -> Self {


Can we possibly have the field names match -- here it is called table_files_statistics_cache but on the CacheManager it is called file_statistics_cache -- I think they should be the same in both places (I like file_statistics_cache best as it matches the type name)

ops, this make sense

alamb · 2023-09-15T17:26:29Z

datafusion/execution/src/cache/mod.rs

+pub mod cache_unit;
+
+// The cache accessor, users usually working on this interface while manipulating caches
+pub trait CacheAccessor<K, V>: Send + Sync {


Another classic API to add here would be "clear()" to clear all the values

datafusion/execution/src/cache/mod.rs

alamb · 2023-09-15T17:30:07Z

datafusion/core/src/datasource/listing_table_factory.rs

@@ -229,8 +229,14 @@ impl TableProviderFactory for ListingTableFactory {
        let config = ListingTableConfig::new(table_path)
            .with_listing_options(options)
            .with_schema(resolved_schema);
-        let table =
-            ListingTable::try_new(config)?.with_definition(cmd.definition.clone());
+        let provider;


If you implement the with_cache API, this can look like

let provider = ListingTable::try_new(config)? .with_cache(state.runtime_env().cache_manager.get_file_statistic_cache())

More clear way ! 👍

Co-authored-by: Andrew Lamb <[email protected]>

Ted-Jiang · 2023-09-17T11:11:32Z

@alamb PTAL

alamb

Thanks @Ted-Jiang -- I think this looks great -- I had a suggestion to improve the comments but I think we can do that as a follow on PR as well. Nice work!

datafusion/core/src/datasource/listing/table.rs

alamb · 2023-09-17T13:02:42Z

datafusion/core/tests/parquet/mod.rs

@@ -40,6 +40,7 @@ use std::sync::Arc;
 use tempfile::NamedTempFile;

 mod custom_reader;
+mod file_statistics;


alamb · 2023-09-17T13:05:33Z

cc @Dandandan / @thinkharderdev @liukun4515 and @mateuszkj who appears to have added this feature originally in 85c11c1 / #3649

liukun4515 · 2023-09-17T14:01:39Z

datafusion/execution/src/cache/mod.rs

+/// locking via internal mutability. It can be accessed via multiple concurrent queries
+/// during planning and execution.
+
+pub trait CacheAccessor<K, V>: Send + Sync {


nice abstract/trait

liukun4515

LGTM for file statistics trait api

Co-authored-by: Andrew Lamb <[email protected]>

Ted-Jiang · 2023-09-18T05:29:07Z

@alamb @liukun4515 Thanks for the review.

Ted-Jiang added 3 commits September 15, 2023 15:31

add extensible cache manager in session ctx.

be03887

move FileStatisticCache into cache manager

04a2f06

add config and test

b81230d

github-actions bot added the core Core DataFusion crate label Sep 15, 2023

Ted-Jiang commented Sep 15, 2023

View reviewed changes

Ted-Jiang requested review from alamb, tustvold and liukun4515 September 15, 2023 10:09

Ted-Jiang added 2 commits September 15, 2023 21:25

fix cargo check

382888f

fix cargo clippy

a66ee12

alamb reviewed Sep 15, 2023

View reviewed changes

Ted-Jiang and others added 4 commits September 16, 2023 09:08

Update datafusion/core/src/datasource/listing/table.rs

0160108

Co-authored-by: Andrew Lamb <[email protected]>

Update datafusion/execution/src/cache/mod.rs

8060a76

Co-authored-by: Andrew Lamb <[email protected]>

fix comments

aaa912d

imply debug for CacheManager

c5c162a

Ted-Jiang force-pushed the cache branch from 9e471bd to c5c162a Compare September 16, 2023 03:00

Ted-Jiang added 4 commits September 16, 2023 11:01

imply debug for CacheManager2

83ba66a

fix doc

b09479e

fix clippy

650d419

fix clippy2

784666f

Ted-Jiang requested a review from alamb September 16, 2023 07:45

alamb mentioned this pull request Sep 16, 2023

Parallelize Parquet Serialization #7562

Merged

alamb approved these changes Sep 17, 2023

View reviewed changes

liukun4515 reviewed Sep 17, 2023

View reviewed changes

liukun4515 approved these changes Sep 17, 2023

View reviewed changes

Ted-Jiang and others added 2 commits September 18, 2023 10:33

Update datafusion/core/src/datasource/listing/table.rs

d77ad00

Co-authored-by: Andrew Lamb <[email protected]>

fix fmt

cac0180

Ted-Jiang merged commit 678d27a into apache:main Sep 18, 2023

This was referenced Sep 18, 2023

Should parallel collecting statistics like infer schema? #7573

Closed

Use file statistics in query planning to avoid sorting when unecessary #7490

Closed

alamb mentioned this pull request Apr 5, 2024

[EPIC] Improve the performance of ListingTable #9964

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570

[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570

Ted-Jiang commented Sep 15, 2023

Ted-Jiang Sep 15, 2023

alamb Sep 15, 2023

alamb left a comment

alamb Sep 15, 2023

Ted-Jiang Sep 16, 2023

alamb Sep 17, 2023

alamb Sep 15, 2023

Ted-Jiang Sep 16, 2023

alamb Sep 15, 2023

alamb Sep 15, 2023

Ted-Jiang Sep 16, 2023

alamb Sep 15, 2023

Ted-Jiang Sep 16, 2023

alamb Sep 15, 2023

Ted-Jiang Sep 16, 2023 •

edited

Loading

Ted-Jiang commented Sep 17, 2023

alamb left a comment

alamb Sep 17, 2023

alamb commented Sep 17, 2023 •

edited

Loading

liukun4515 Sep 17, 2023

liukun4515 left a comment

Ted-Jiang commented Sep 18, 2023

[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570

[feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570

Conversation

Ted-Jiang commented Sep 15, 2023

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ted-Jiang Sep 16, 2023 • edited Loading

Choose a reason for hiding this comment

Ted-Jiang commented Sep 17, 2023

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Sep 17, 2023 • edited Loading

Choose a reason for hiding this comment

liukun4515 left a comment

Choose a reason for hiding this comment

Ted-Jiang commented Sep 18, 2023

Ted-Jiang Sep 16, 2023 •

edited

Loading

alamb commented Sep 17, 2023 •

edited

Loading