[SDK-parquet] parquet default processor extractor step #601

yuunlimm · 2024-11-12T19:27:48Z

Description

This is to add an extractor step for parquet default processor.

Changes invovled:

added a map to filter parquet structs.
added a new model RawTableItem and a trait TableItemConvertible which is implemented for both Parquet and Postgres TableItem.
updated a (Postgres) TableItem::from_write_table_item to return table_items only

Context on why we added a new RawTableItem model:
tl;dr
This is to reduce the duplicated code as much as possible and reuse the common parsing logic. RawTableItem represents parsed transaction data. This struct will act as the unified output from the parsing function. Parquet TableItem and Postgres TableItem will implement a TableItemConvertible trait to convert RawTableItem into the appropriate format.

With this approach, parsing logic is centralized, and each processor (Postgres or Parquet) can convert RawTableItem to its respective TableItem type, keeping code DRY and maintainable while supporting format-specific needs.

yuunlimm · 2024-11-12T19:28:09Z

[SDK-parquet] add parquet version tracker #609
[SDK-parquet] parquet sized buffer and gcs handler #602
[SDK-parquet] parquet default processor extractor step #601 👈 (View in Graphite)
[parquet-sdk-migration] add a logic to determine the starting version for parquet processor #587 : 1 other dependent PR (#591 )
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

rust/sdk-processor/src/parquet_processors/mod.rs

rust/sdk-processor/src/steps/mod.rs

rust/sdk-processor/src/steps/common/parquet_extractor_helper.rs

rust/sdk-processor/src/steps/parquet_default_processor/parquet_default_extractor.rs

rtso · 2024-11-13T17:17:47Z

rust/sdk-processor/src/steps/parquet_default_processor/parquet_default_extractor.rs

+pub fn process_transactions(
+    transactions: Vec<Transaction>,
+) -> (
+    Vec<MoveResource>,
+    Vec<WriteSetChangeModel>,
+    Vec<TransactionModel>,
+    Vec<TableItem>,
+    Vec<MoveModule>,
+) {
+    // this will be removed in the future.
+    let mut transaction_version_to_struct_count = AHashMap::new();
+    let (txns, _, write_set_changes, wsc_details) = TransactionModel::from_transactions(
+        &transactions,
+        &mut transaction_version_to_struct_count,
+    );
+
+    let mut move_modules = vec![];
+    let mut move_resources = vec![];
+    let mut table_items = vec![];
+
+    for detail in wsc_details {
+        match detail {
+            WriteSetChangeDetail::Module(module) => {
+                move_modules.push(module);
+            },
+            WriteSetChangeDetail::Resource(resource) => {
+                move_resources.push(resource);
+            },
+            WriteSetChangeDetail::Table(item, _, _) => {
+                table_items.push(item);
+            },
+        }
+    }
+
+    (
+        move_resources,
+        write_set_changes,
+        txns,
+        table_items,
+        move_modules,
+    )
+}
+


Is it possible to re-use the process_transactions from the Parquet default processor?

Or better yet, resuse this one from the Postgres default processor:

aptos-indexer-processors/rust/processor/src/processors/default_processor.rs

Line 295 in 7b2873f

pub fn process_transactions(

yeah we can reuse the the process_transactions from the parquet default processor!

just sharing the context why initially I decided not to re-use the process_transactions from the parquet processor because there is that we wouldn't need the struct count map. having new code in this file allows us to simply remove the code later after migrated all. but it already re-uses some of the functions from the existing parquet processor lol.

Or better yet, resuse this one from the Postgres default processor:

I don't think we can reuse the current version of process_transactions from the postgres default processor since they are kind of diverged due to Parquet migration. Keeping them as separate functions could simplify the code by avoiding conditionals for deprecated tables, making each version cleaner and more maintainable.

Alternatively, if there’s significant overlap, we could refactor shared logic into helper functions. What do you think—is it better to keep them separate or refactor shared portions?

Just to understand the context: what part of the Postgres and Parquet diverged during the migration?

unresolving, also curious about the above ^ @yuunlimm

rust/sdk-processor/src/steps/common/parquet_extractor_helper.rs

…out duplicating the parsing code

dermanyang

approving to unblock with some questions

rust/sdk-processor/src/steps/default_processor/default_extractor.rs

dermanyang · 2024-11-14T18:08:33Z

rust/sdk-processor/src/steps/parquet_default_processor/parquet_default_extractor.rs

+                _parquet_table_items,
+                move_modules,
+            ),
+            _transaction_version_to_struct_count,


curious what the convention is in general for var names with leading underscore?

It means the variables are unused, so the compiler doesn't mark this as a lint error

when would you do the leading underscore instead of just the underscore like in

let (txns, _, write_set_changes, wsc_details) = TransactionModel::from_transactions( &transactions, &mut transaction_version_to_struct_count, );

dermanyang · 2024-11-14T18:10:03Z

rust/sdk-processor/src/steps/parquet_default_processor/parquet_default_extractor.rs

+pub fn process_transactions(
+    transactions: Vec<Transaction>,
+) -> (
+    Vec<MoveResource>,
+    Vec<WriteSetChangeModel>,
+    Vec<TransactionModel>,
+    Vec<TableItem>,
+    Vec<MoveModule>,
+) {
+    // this will be removed in the future.
+    let mut transaction_version_to_struct_count = AHashMap::new();
+    let (txns, _, write_set_changes, wsc_details) = TransactionModel::from_transactions(
+        &transactions,
+        &mut transaction_version_to_struct_count,
+    );
+
+    let mut move_modules = vec![];
+    let mut move_resources = vec![];
+    let mut table_items = vec![];
+
+    for detail in wsc_details {
+        match detail {
+            WriteSetChangeDetail::Module(module) => {
+                move_modules.push(module);
+            },
+            WriteSetChangeDetail::Resource(resource) => {
+                move_resources.push(resource);
+            },
+            WriteSetChangeDetail::Table(item, _, _) => {
+                table_items.push(item);
+            },
+        }
+    }
+
+    (
+        move_resources,
+        write_set_changes,
+        txns,
+        table_items,
+        move_modules,
+    )
+}
+


unresolving, also curious about the above ^ @yuunlimm

rust/processor/src/processors/default_processor.rs

rust/processor/src/db/common/models/default_models/move_tables.rs

rust/processor/src/db/common/models/stake_models/delegator_balances.rs

rust/processor/src/db/common/models/default_models/move_tables.rs

rust/processor/src/db/common/models/stake_models/delegator_balances.rs

rust/processor/src/db/common/models/default_models/move_tables.rs

dermanyang

looks good. thanks for the detailed decription.

rtso

lgtm with comments!

rust/processor/src/db/mod.rs

rust/processor/src/db/postgres/models/default_models/move_tables.rs

rust/processor/src/db/postgres/models/coin_models/coin_supply.rs

This was referenced Nov 12, 2024

[parquet-sdk-migration] add a logic to determine the starting version for parquet processor #587

Merged

import step from sdk repo #591

Closed

yuunlimm mentioned this pull request Nov 12, 2024

[SDK-parquet] consolidated step #599

Closed

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch 2 times, most recently from b4f46db to 9bff5f3 Compare November 12, 2024 19:55

yuunlimm mentioned this pull request Nov 12, 2024

[SDK-parquet] parquet sized buffer and gcs handler #602

Merged

yuunlimm marked this pull request as ready for review November 12, 2024 22:03

rtso reviewed Nov 13, 2024

View reviewed changes

rust/sdk-processor/src/parquet_processors/mod.rs Outdated Show resolved Hide resolved

rtso reviewed Nov 13, 2024

View reviewed changes

yuunlimm force-pushed the restored_branch branch 2 times, most recently from 7e0016b to 58ac581 Compare November 13, 2024 19:25

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch from 9bff5f3 to 9c6dc91 Compare November 13, 2024 19:25

dermanyang reviewed Nov 13, 2024

View reviewed changes

rust/sdk-processor/src/steps/common/parquet_extractor_helper.rs Outdated Show resolved Hide resolved

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch from 472b366 to daf3f6e Compare November 14, 2024 00:09

yuunlimm force-pushed the restored_branch branch from 58ac581 to 067e1f2 Compare November 14, 2024 00:10

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch 3 times, most recently from eb396de to 16f0a25 Compare November 14, 2024 00:20

yuunlimm changed the base branch from restored_branch to graphite-base/601 November 14, 2024 00:24

yuunlimm force-pushed the graphite-base/601 branch from 067e1f2 to f828135 Compare November 14, 2024 00:25

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch from 16f0a25 to 2f91ffa Compare November 14, 2024 00:25

yuunlimm changed the base branch from graphite-base/601 to main November 14, 2024 00:25

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch 3 times, most recently from 1bc8672 to ef311b0 Compare November 14, 2024 17:48

yuunlimm added 2 commits November 14, 2024 09:49

[SDK-parquet] parquet default processor extractor step

4cf2e7d

add a raw table item struct to support both parquet and postgres with…

8f43e54

…out duplicating the parsing code

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch from ef311b0 to 8f43e54 Compare November 14, 2024 17:49

yuunlimm requested a review from rtso November 14, 2024 17:52

yuunlimm requested a review from dermanyang November 14, 2024 17:52

yuunlimm mentioned this pull request Nov 14, 2024

[SDK-parquet] add parquet version tracker #609

Merged

lint

7a8aad3

dermanyang approved these changes Nov 14, 2024

View reviewed changes

rtso reviewed Nov 14, 2024

View reviewed changes

rust/processor/src/processors/default_processor.rs Outdated Show resolved Hide resolved

rtso reviewed Nov 14, 2024

View reviewed changes

rust/processor/src/db/common/models/default_models/move_tables.rs Outdated Show resolved Hide resolved

rtso reviewed Nov 14, 2024

View reviewed changes

rust/processor/src/db/common/models/default_models/move_tables.rs Outdated Show resolved Hide resolved

yuunlimm commented Nov 14, 2024

View reviewed changes

rust/processor/src/db/common/models/stake_models/delegator_balances.rs Show resolved Hide resolved

rust/processor/src/db/common/models/default_models/move_tables.rs Outdated Show resolved Hide resolved

rtso reviewed Nov 14, 2024

View reviewed changes

rust/processor/src/db/common/models/stake_models/delegator_balances.rs Outdated Show resolved Hide resolved

rust/processor/src/db/common/models/default_models/move_tables.rs Outdated Show resolved Hide resolved

dermanyang approved these changes Nov 15, 2024

View reviewed changes

rtso approved these changes Nov 18, 2024

View reviewed changes

rust/processor/src/db/mod.rs Show resolved Hide resolved

rust/processor/src/db/postgres/models/default_models/move_tables.rs Outdated Show resolved Hide resolved

rust/processor/src/db/postgres/models/coin_models/coin_supply.rs Show resolved Hide resolved

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch 2 times, most recently from 2198641 to 4455b67 Compare November 18, 2024 21:53

separate parquet and postgres model files

61ecc68

yuunlimm force-pushed the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch from 4455b67 to 61ecc68 Compare November 18, 2024 21:55

yuunlimm merged commit 552ecc7 into main Nov 18, 2024
7 checks passed

yuunlimm deleted the 11-12-_sdk-parquet_parquet_default_processor_extractor_step branch November 18, 2024 23:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SDK-parquet] parquet default processor extractor step #601

[SDK-parquet] parquet default processor extractor step #601

yuunlimm commented Nov 12, 2024 •

edited

Loading

yuunlimm commented Nov 12, 2024 •

edited

Loading

rtso Nov 13, 2024

rtso Nov 13, 2024

yuunlimm Nov 13, 2024 •

edited

Loading

rtso Nov 13, 2024

dermanyang Nov 14, 2024

dermanyang left a comment

dermanyang Nov 14, 2024

rtso Nov 14, 2024

yuunlimm Nov 14, 2024

dermanyang Nov 15, 2024

dermanyang Nov 14, 2024

dermanyang left a comment

rtso left a comment

[SDK-parquet] parquet default processor extractor step #601

[SDK-parquet] parquet default processor extractor step #601

Conversation

yuunlimm commented Nov 12, 2024 • edited Loading

Description

yuunlimm commented Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuunlimm Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dermanyang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dermanyang left a comment

Choose a reason for hiding this comment

rtso left a comment

Choose a reason for hiding this comment

yuunlimm commented Nov 12, 2024 •

edited

Loading

yuunlimm commented Nov 12, 2024 •

edited

Loading

yuunlimm Nov 13, 2024 •

edited

Loading