Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parquet default processor and clean up a bit #480

Merged
merged 3 commits into from
Aug 5, 2024
Merged

Conversation

yuunlimm
Copy link
Contributor

@yuunlimm yuunlimm commented Aug 2, 2024

Description

  • removed table_metadata from parquet, which is a problematic in terms of not being sent to channel [todo: rootcause]
  • added fields to logs for easier debugging
  • moved parquet periodic upload logic before looping structs.
  • made parquet gap detector logic simpler.

Test Plan

  • tested that default processor updating the latest processed version with the correct version.

…s of not being sent to channel [todo: rootcause]

- added fields to logs for easier debugging
- moved parquet periodic upload logic before looping structs.
- made parquet gap detector logic simpler.
@yuunlimm yuunlimm requested review from ying-w and a team August 2, 2024 17:44
@yuunlimm yuunlimm merged commit 89844fb into main Aug 5, 2024
7 checks passed
@yuunlimm yuunlimm deleted the yuunlimm/parquet branch August 5, 2024 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants