Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Parquet Events Processor to SDK #618

Merged
merged 10 commits into from
Dec 5, 2024

Conversation

dermanyang
Copy link
Contributor

@dermanyang dermanyang commented Nov 26, 2024

Purpose

  • Create parquet events processor with SDK.
  • Create new RawEvent model and EventConvertible trait for shared parsing logic. RawEvent.from_raw_eventis used by events and parquet_events.
  • Refactor parquet processing logic to process_transactions_parquet.

Testing

image image

Copy link

codecov bot commented Dec 2, 2024

Codecov Report

Attention: Patch coverage is 23.57143% with 214 lines in your changes missing coverage. Please review.

Project coverage is 49.2%. Comparing base (3c62913) to head (f584bfd).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...src/parquet_processors/parquet_events_processor.rs 0.0% 98 Missing ⚠️
...ors/parquet_processors/parquet_events_processor.rs 0.0% 53 Missing ⚠️
...c/db/parquet/models/event_models/parquet_events.rs 0.0% 24 Missing ⚠️
...rquet_events_processor/parquet_events_extractor.rs 0.0% 23 Missing ⚠️
rust/processor/src/processors/events_processor.rs 64.0% 9 Missing ⚠️
rust/sdk-processor/src/parquet_processors/mod.rs 0.0% 3 Missing ⚠️
...k-processor/src/config/indexer_processor_config.rs 0.0% 2 Missing ⚠️
...or/src/db/common/models/event_models/raw_events.rs 96.6% 1 Missing ⚠️
rust/sdk-processor/src/config/processor_config.rs 66.6% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main    #618     +/-   ##
=======================================
- Coverage   49.4%   49.2%   -0.3%     
=======================================
  Files        192     195      +3     
  Lines      25478   25665    +187     
=======================================
+ Hits       12588   12629     +41     
- Misses     12890   13036    +146     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dermanyang dermanyang changed the title Init Migrate Parquet Events Processor to SDK Dec 2, 2024
@dermanyang dermanyang marked this pull request as ready for review December 3, 2024 20:10
@dermanyang dermanyang requested review from yuunlimm and a team December 3, 2024 20:10
@dermanyang dermanyang requested a review from yuunlimm December 5, 2024 00:12
@aptos-labs aptos-labs deleted a comment from yuunlimm Dec 5, 2024
// More tables
const CURRENT_TABLE_ITEMS = 1 << 24;
const BLOCK_METADATA_TRANSACTIONS = 1 << 25;

// Events
const EVENTS = 1 << 24;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: 24 is used by current table items. maybe this will likely happen again, should we assign specific range to each processor to prevent this conflict? like 20-30 -> default processor 30-40 -> fa. since it will be less than 10 tables

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh. that's a good idea.

@dermanyang dermanyang merged commit b823242 into main Dec 5, 2024
11 checks passed
@dermanyang dermanyang deleted the sdy/migrate_parquet_events_processor branch December 5, 2024 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants