Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iceberg-connector: Delete the oldest tracked version metadata files after commit #23766

Closed
wants to merge 3 commits into from

Conversation

7hong
Copy link
Contributor

@7hong 7hong commented Oct 12, 2024

Description

When using the iceberg table, the xxx-metadata.json file is generated each time commit is executed. In the iceberg table, we can automatically clean the previous metadata file through configuration.

write.metadata.delete-after-commit.enabled = true
write.metadata.previous-versions-max = 10
However, the metadata file is not automatically cleaned in trino. A large number of metadata.json files are left on the hdfs.

Additional context and related issues

issue #19582

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x ) Release notes are required, with the following suggested text:

# iceberg
* Fixes #19582. Delete the oldest tracked version metadata files after commit. ({issue} #19582 ). 

@cla-bot cla-bot bot added the cla-signed label Oct 12, 2024
@github-actions github-actions bot added the iceberg Iceberg connector label Oct 12, 2024
@ebyhr
Copy link
Member

ebyhr commented Oct 13, 2024

@7hong Is this PR duplicate of #20863?

@7hong
Copy link
Contributor Author

7hong commented Oct 14, 2024

@7hong Is this PR duplicate of #20863?

They solve the same problem. Differences:
I have followed the configurations in iceberg (write.metadata.delete-after-commit.enabled and write.metadata.previous-version-max ). Instead of adding a new configuration, this keeps it compatible with iceberg.

This PR has not been merged, but we really need this feature, so I submitted it again。Please help review, thank you very much.
@ebyhr

@7hong
Copy link
Contributor Author

7hong commented Oct 15, 2024

@ebyhr @alexjo2144 @grantatspothero
Can you help with the review? Thank you very much,

@7hong 7hong closed this by deleting the head repository Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed iceberg Iceberg connector
Development

Successfully merging this pull request may close these issues.

2 participants