Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a metadata table showing commits in Hudi connector #16181

Merged
merged 2 commits into from
Mar 20, 2023

Conversation

albericgenius
Copy link
Contributor

@albericgenius albericgenius commented Feb 20, 2023

Description

Fix #16149

  • add the Timeline table to show commits in Hudi connector

Release notes

(x) Release notes are required, with the following suggested text:

# Hudi
* Add a `$timeline` system table which can be queried to inspect Hudi table timeline. ({issue}`16149`)

@findinpath
Copy link
Contributor

Please add documentation about the new system table in hudi.rst.

@albericgenius
Copy link
Contributor Author

There is UI CI issue #16179, the failed test case is not relative to this PR.

@ebyhr ebyhr self-requested a review February 22, 2023 23:39
@albericgenius albericgenius force-pushed the snapshots branch 2 times, most recently from 2c606a7 to 58ee60f Compare February 23, 2023 13:19
docs/src/main/sphinx/connector/hudi.rst Outdated Show resolved Hide resolved
.build());
this.types = tableMetadata.getColumns().stream().map(ColumnMetadata::getType).collect(toImmutableList());
HoodieTableMetaClient metaClient = buildTableMetaClient(configuration, hudiTable.getStorage().getLocation());
this.hudiInstants = metaClient.getActiveTimeline().getInstants().sorted(comparing(HoodieInstant::getTimestamp).reversed()).collect(toImmutableList());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will materialize all active timelines in my understanding. It would be nice to avoid it for avoiding high memory usage.

@ebyhr ebyhr merged commit 7927d79 into trinodb:master Mar 20, 2023
@ebyhr ebyhr mentioned this pull request Mar 20, 2023
@github-actions github-actions bot added this to the 411 milestone Mar 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

Add a metadata table showing commits in Hudi connector
3 participants