[index] current unified balances #366

lightmark · 2024-05-06T22:14:36Z

Add a new table current_unified_fungible_asset_balances to reflect the total balance of a coin.

tested locally with devnet data:

Indexer Testing

Just make sure for every step in [Correctness Testing the current_unifed_fungible_asset_balances table shows the correct number at the correct row&col, which matches current_coin_balances and current_fungible_asset_balances.

This is the final state on devnet with running fungible asset processor locally.

Row #1 (Shows as column here for better layout) is the default profile APT balance

Row #2 (Shows as column here for better layout) is the test profile APT balance

Row #3 (Shows as column here for better layout) is the default profile MoonCoin balance

Row#4 (Shows as column here for better layout) is the default profile a Pure FA balance

sent another txn to send 20000 octa to 0x1 who already has 1+ apt, then we get:

larry-aptos · 2024-05-06T22:56:11Z

rust/processor/Cargo.toml

@@ -59,6 +59,8 @@ url = { workspace = true }
 native-tls = { workspace = true }
 postgres-native-tls = { workspace = true }
 tokio-postgres = { workspace = true }
+tiny-keccak = { version = "2.0.2", features = ["sha3"] }


nit: workspace = true

larry-aptos · 2024-05-06T23:04:18Z

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs

+    owner_address: &str,
+    metadata_address: &str,
+) -> anyhow::Result<String> {
+    let mut preimage = hex_to_raw_bytes(owner_address)?;


is the address padded with 0 if not full 64 long.

I follow the existing logic.

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs

grao1991 · 2024-05-10T19:14:38Z

rust/processor/migrations/2024-05-04-025823_current_unified_fungible_asset_balance/up.sql

@@ -0,0 +1,21 @@
+-- current fungible asset balances
+CREATE TABLE IF NOT EXISTS current_unified_fungible_asset_balances (
+    storage_id VARCHAR(66) UNIQUE PRIMARY KEY NOT NULL,


nit: you don't really need UNIQUE here.

I knew, let's remove it.

grao1991 · 2024-05-10T19:15:59Z

rust/processor/migrations/2024-05-04-025823_current_unified_fungible_asset_balance/up.sql

+    last_transaction_timestamp_v1 TIMESTAMP,
+    last_transaction_timestamp_v2 TIMESTAMP,
+    last_transaction_timestamp TIMESTAMP GENERATED ALWAYS AS (GREATEST(last_transaction_timestamp_v1, last_transaction_timestamp_v2)) STORED,
+    inserted_at TIMESTAMP NOT NULL DEFAULT NOW()


@CapCap Do we still need this column (and the corresponding index)? This will likely cause scalability issue later.

when we move everything to parquet writes, we can remove these columns- until then these are used just for ingest :-(

VERY SOON THOUGH!

grao1991 · 2024-05-10T19:18:53Z

rust/processor/migrations/2024-05-04-025823_current_unified_fungible_asset_balance/up.sql

+    last_transaction_timestamp TIMESTAMP GENERATED ALWAYS AS (GREATEST(last_transaction_timestamp_v1, last_transaction_timestamp_v2)) STORED,
+    inserted_at TIMESTAMP NOT NULL DEFAULT NOW()
+);
+CREATE INDEX IF NOT EXISTS cufab_owner_at_index ON current_unified_fungible_asset_balances (owner_address, asset_type);


Waht's the relationship among owner_address, asset_type, and storage_id? Any chance we can remove this index in the future?

grao1991 · 2024-05-10T19:24:59Z

rust/processor/src/processors/fungible_asset_processor.rs

+            .do_update()
+            .set(
+                (
+                    owner_address.eq(excluded(owner_address)),


If there is a conflict between v1 and v2, are these (owner_address, asset_type, is_frozen) fields the same?

is_frozen is not. most of the time it is.

The the order of conflict upserts matters?

I think maybe we should ignore this field for v1.

larry-aptos · 2024-05-15T19:38:10Z

Fix lint, you need to import use std::borrow::Borrow; for borrow() method.

rtso · 2024-05-28T19:44:17Z

Since this is deployed to devnet now, can you add these to the test plan:

test the new table compared with current_fungible_asset_balances to see that the balances match
result of test txn's on devnet

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs

rust/processor/src/processors/fungible_asset_processor.rs

rust/processor/migrations/2024-05-04-025823_current_unified_fungible_asset_balance/up.sql

bowenyang007 · 2024-05-29T20:43:09Z

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs

+    }
+}
+
+fn get_primary_fungible_store_address(


Is this the same as the existing code that you deleted? do we need to test that nothing changed?

your fabulous is_primary_test cover this function.

bowenyang007 · 2024-05-31T17:49:10Z

rust/processor/src/processors/fungible_asset_processor.rs

+    // Process the unified balance
+    let current_unified_fungible_asset_balances = current_fungible_asset_balances
+        .iter()
+        .map(CurrentUnifiedFungibleAssetBalance::from)
+        .collect::<Vec<_>>();


One thing to note is that we can't have partial fill for current_fungible_asset_balances b/c the partials would override current_unified_fungible_asset_balances.

What do you mean by partial fill and override?

Let's say we have wallet A and we transfer some $$ there and the it gets deleted. We'd have partial write for the delete correct? But this logic would miss the delete I think. I guess it won't be overridden automatically since the deletes would be a different map.

lightmark force-pushed the lightmark/coin_fa_balance branch 2 times, most recently from 08c906e to 6b8504e Compare May 6, 2024 22:54

lightmark requested review from bowenyang007, larry-aptos, CapCap and rtso May 6, 2024 22:54

lightmark force-pushed the lightmark/coin_fa_balance branch from 6b8504e to 676bc28 Compare May 6, 2024 23:08

larry-aptos approved these changes May 6, 2024

View reviewed changes

lightmark force-pushed the lightmark/coin_fa_balance branch 2 times, most recently from 9486104 to a03a8c5 Compare May 7, 2024 04:50

rtso requested a review from grao1991 May 7, 2024 17:00

grao1991 reviewed May 10, 2024

View reviewed changes

larry-aptos added the build-rust label May 15, 2024

lightmark force-pushed the lightmark/coin_fa_balance branch 7 times, most recently from ca71635 to 080b6ec Compare May 17, 2024 17:34

lightmark changed the title ~~[index] concurrent unified balances~~ [index] current unified balances May 21, 2024

lightmark force-pushed the lightmark/coin_fa_balance branch from 6c8f121 to d1a230b Compare May 23, 2024 19:28

rtso reviewed May 28, 2024

View reviewed changes

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs Outdated Show resolved Hide resolved

rtso reviewed May 28, 2024

View reviewed changes

rust/processor/src/models/fungible_asset_models/v2_fungible_asset_balances.rs Show resolved Hide resolved

rtso reviewed May 28, 2024

View reviewed changes

rust/processor/src/processors/fungible_asset_processor.rs Show resolved Hide resolved

rtso reviewed May 28, 2024

View reviewed changes

rust/processor/migrations/2024-05-04-025823_current_unified_fungible_asset_balance/up.sql Show resolved Hide resolved

bowenyang007 reviewed May 29, 2024

View reviewed changes

[index] concurrent unified balances

1f53518

fixup! [index] concurrent unified balances

5fe8abe

bowenyang007 force-pushed the lightmark/coin_fa_balance branch from f0fd19e to 5fe8abe Compare May 31, 2024 17:43

bowenyang007 reviewed May 31, 2024

View reviewed changes

bowenyang007 approved these changes May 31, 2024

View reviewed changes

bowenyang007 merged commit f6c39c2 into main May 31, 2024
7 checks passed

bowenyang007 deleted the lightmark/coin_fa_balance branch May 31, 2024 19:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[index] current unified balances #366

[index] current unified balances #366

lightmark commented May 6, 2024 •

edited

Loading

larry-aptos May 6, 2024

larry-aptos May 6, 2024

lightmark May 7, 2024

grao1991 May 10, 2024

lightmark May 15, 2024

grao1991 May 15, 2024

grao1991 May 10, 2024

CapCap May 16, 2024

grao1991 May 10, 2024

lightmark May 15, 2024

grao1991 May 10, 2024

lightmark May 15, 2024

lightmark May 17, 2024

grao1991 May 18, 2024

lightmark May 21, 2024

larry-aptos commented May 15, 2024

rtso commented May 28, 2024

bowenyang007 May 29, 2024 •

edited

Loading

lightmark May 30, 2024

bowenyang007 May 31, 2024 •

edited

Loading

rtso May 31, 2024 •

edited

Loading

bowenyang007 May 31, 2024

[index] current unified balances #366

[index] current unified balances #366

Conversation

lightmark commented May 6, 2024 • edited Loading

Indexer Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larry-aptos commented May 15, 2024

rtso commented May 28, 2024

bowenyang007 May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenyang007 May 31, 2024 • edited Loading

Choose a reason for hiding this comment

rtso May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightmark commented May 6, 2024 •

edited

Loading

bowenyang007 May 29, 2024 •

edited

Loading

bowenyang007 May 31, 2024 •

edited

Loading

rtso May 31, 2024 •

edited

Loading