Get stats from hive metastore only for the necessary columns

[lukasz-stec]

Description

HiveTableHandle contains the list of projected columns. Since only projected columns will be available to the engine, HiveMetadata.getTableStatistics can return statistics limited to those columns.
This also requires CachingHiveMetastore to support caching statistics for a subset of table columns.

Benchmarks

I tested the change on the glue metastore and table with 100 columns (details below) with caching stats disabled (hive.metastore-stats-cache-ttl=0s) and a simple query (select count(c1) from test_col_stats_part) where planning dominates.

There is a difference of about 20% for planning time/elapsed (so visible)

baseline 2.251s on average and only required stats 1.853s

This would have a potentially higher impact on a metastore under heavy load.

The table used:

create table test_col_stats_part (c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, c12, c13, c14, c15, c16, c17, c18, c19, c20, c21, c22, c23, c24, c25, c26, c27, c28, c29, c30, c31, c32, c33, c34, c35, c36, c37, c38, c39, c40, c41, c42, c43, c44, c45, c46, c47, c48, c49, c50, c51, c52, c53, c54, c55, c56, c57, c58, c59, c60, c61, c62, c63, c64, c65, c66, c67, c68, c69, c70, c71, c72, c73, c74, c75, c76, c77, c78, c79, c80, c81, c82, c83, c84, c85, c86, c87, c88, c89, c90, c91, c92, c93, c94, c95, c96, c97, c98, c99, c100, shipdate)
 WITH (                                              
    format = 'ORC',                                  
    partitioned_by = ARRAY['shipdate']               
 )
 as select 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, shipdate from hive.tpch_sf10_dec_orc.lineitem;

I also ran standard tpch/tpcds on scale factor 1000, orc, partitioned. The results show mostly no change (except for some variability)

sf1k-orc-part-240223.pdf

Additional context and related issues

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive
* Improve query planning performance for queries scanning tables with a larger number of columns. ({issue}`16203`)

[lukasz-stec]

CA

[lukasz-stec]

ca

[lukasz-stec]

rebased on the latest master (auto rebase were causing compilation errors)

[lukasz-stec]

ca

[findepi]

we're building fake table object, hoping the callee will trust the table, and not eg use knowledge about column list from some other place

that's an opaque design, something explicit would probably be better.
Why not pass Set<String> requestedColumns (or Optional thereof`) down the call chain?

[lukasz-stec]

Why not pass Set requestedColumns (or Optional thereof`) down the call chain?

That's an excellent idea. It also applies to the getPartitionStatistics, right?

[findepi]

potentially yes