[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737

Merged

yihua merged 27 commits intoapache:masterfrom onehouseinc:ak/spk-ctlg-fix

Jun 7, 2022

+375-257

Commits on Jun 6, 2022

Make sure TestCOWDataSource properly invokes HoodieSparkSessionExtensions
Alexey Kudinkin
committed
Make sure HoodieBaseRelation always prefers schemaSpec over fetching schema from the storage;
Alexey Kudinkin
committed
Fixing HoodieSpark3Analysis rule to properly pass in schema from the catalog
Alexey Kudinkin
committed
Fixed missing conversion
Alexey Kudinkin
committed
Added LogicalPlan caching for Hudi's relations
Alexey Kudinkin
committed
Properly override outputs of the cached LogicalRelation
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Make HoodieCatalog and Spark3DefaultSource return V1Table instead of HoodieInternalV2Table to make sure Spark treats Hudi tables as V1 and not V2
Alexey Kudinkin
committed
Added HoodieV1Table extractor to match Hudi tables;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Fix missing fixture of the DateTimeZone.default
Alexey Kudinkin
committed
Reverting DSv2 APIs
Alexey Kudinkin
committed
Make Hudi resolve into either V1 or V2 relations based on whether Schema Evolution is enabled;
Alexey Kudinkin
committed
Restored resolution-rule to fallback from V2 Relation to V1
Alexey Kudinkin
committed
Rebased Schema Evolution commands resolution rule to use V2 table
Alexey Kudinkin
committed
Fixed schema evoluation rule injection seq
Alexey Kudinkin
committed
Make sure V2 to V1 fallback is properly wired
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Tidying up
Alexey Kudinkin
committed
Fixed V2 to V1 fallback to also apply to Spark 3.1;
Alexey Kudinkin
committed
Fixing compilation
Alexey Kudinkin
committed
Reverting back invalid bifurcation b/w Spark 3.1 and 3.2
Alexey Kudinkin
committed
Make sure we're looking up configs appropriately
Alexey Kudinkin
committed
Moved SCHEMA_EVOLUTION_ENABLED config to HoodieCommonConfig
Alexey Kudinkin
committed

Commits on Jun 7, 2022

Fixed rules evaluation order
Alexey Kudinkin
committed
Inline Alter Table commands resolving rules to control ordering of their application
Alexey Kudinkin
committed
Avoid providing catalog schema in cases when Hive catalog is used
Alexey Kudinkin
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration#5737

Commits on Jun 6, 2022

Make sure `TestCOWDataSource` properly invokes `HoodieSparkSessionExtensions`

Make sure `HoodieBaseRelation` always prefers `schemaSpec` over fetching schema from the storage;

Fixing `HoodieSpark3Analysis` rule to properly pass in schema from the catalog

Fixed missing conversion

Added `LogicalPlan` caching for Hudi's relations

Properly override outputs of the cached `LogicalRelation`

Tidying up

Make `HoodieCatalog` and `Spark3DefaultSource` return `V1Table` instead of `HoodieInternalV2Table` to make sure Spark treats Hudi tables as V1 and not V2

Added `HoodieV1Table` extractor to match Hudi tables;

Fixing compilation

Fix missing fixture of the `DateTimeZone.default`

Reverting DSv2 APIs

Make Hudi resolve into either V1 or V2 relations based on whether Schema Evolution is enabled;

Restored resolution-rule to fallback from V2 Relation to V1

Rebased Schema Evolution commands resolution rule to use V2 table

Fixed schema evoluation rule injection seq

Make sure V2 to V1 fallback is properly wired

Tidying up

Tidying up

Fixed V2 to V1 fallback to also apply to Spark 3.1;

Fixing compilation

Reverting back invalid bifurcation b/w Spark 3.1 and 3.2

Make sure we're looking up configs appropriately

Moved `SCHEMA_EVOLUTION_ENABLED` config to `HoodieCommonConfig`

Commits on Jun 7, 2022

Fixed rules evaluation order

Inline Alter Table commands resolving rules to control ordering of their application

Avoid providing catalog schema in cases when Hive catalog is used