Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove initial splits from delta splits generation #21320

Merged
merged 1 commit into from
Mar 29, 2024

Conversation

raunaqmorarka
Copy link
Member

@raunaqmorarka raunaqmorarka commented Mar 29, 2024

Description

Minimum split sizes in parquet are determined by parquet writer row group size.
The default parquet row group size is 128MB which makes any initial splits smaller than that unncessary.
Concept of initial splits is intended for formats like TEXT

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Delta Lake
* Catalog configuration properties `delta.max-initial-splits` and `delta.max-initial-split-size` are now defunct and must be removed from cluster configuration properties to successfully start the server. The catalog session property `max_initial_split_size` has been removed. ({issue}`21320`)

@cla-bot cla-bot bot added the cla-signed label Mar 29, 2024
@github-actions github-actions bot added the delta-lake Delta Lake connector label Mar 29, 2024
Minimum split sizes in parquet are determined by parquet writer row group size.
The default parquet row group size is 128MB which makes any initial splits smaller
than that unncessary.
Concept of initial splits is intended for formats like TEXT
@sopel39 sopel39 merged commit c8629fc into trinodb:master Mar 29, 2024
28 checks passed
@github-actions github-actions bot added this to the 444 milestone Mar 29, 2024
@raunaqmorarka raunaqmorarka deleted the remove-delta-initial branch July 9, 2024 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

3 participants