enable parquet to start processing from the specified starting version in the config

[yuunlimm]

Description

parquet processor was always starting from the min processed version or 0 if not all tables are processed. And we would like the consistency across all processors where it should start from the higher version.

Test Plan

[yuunlimm]

• enable parquet to start processing from the specified starting version in the config #644 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[dermanyang]

if min_processed_version = config_starting_version = 0, then this will return 1 so version 0 will never be processed. Removing the + 1 will fix this!

[yuunlimm]

good catch! will fix tjos

[yuunlimm]

aptos-indexer-processors/rust/sdk-processor/src/utils/starting_version.rs

Lines 26 to 43 in 3dfd2f6

	pub async fn get_starting_version(
	indexer_processor_config: &IndexerProcessorConfig,
	conn_pool: ArcDbPool,
	) -> Result<u64> {
	// Check if there's a checkpoint in the appropriate processor status table.
	let latest_processed_version =
	get_starting_version_from_db(indexer_processor_config, conn_pool)
	.await
	.context("Failed to get latest processed version from DB")?;
	// If nothing checkpointed, return the `starting_version` from the config, or 0 if not set.
	Ok(latest_processed_version.unwrap_or(
	indexer_processor_config
	.transaction_stream_config
	.starting_version
	.unwrap_or(0),
	))
	}

hmm, in that case, would this also return 1?

[yuunlimm]

  // return the higher of the checkpointed version + 1 and `starting_version`.
    Ok(status.map(|status| {
        std::cmp::max(
            status.last_success_version as u64 + 1,
            indexer_processor_config
                .transaction_stream_config
                .starting_version
                .unwrap_or(0),
        )
    }))
    ```

[dermanyang]

yeah lol do you mind changing that too please

[codecov]

Codecov Report

Attention: Patch coverage is 9.09091% with 10 lines in your changes missing coverage. Please review.

Project coverage is 45.2%. Comparing base (22f92fc) to head (7ebce69).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
rust/sdk-processor/src/utils/starting_version.rs	9.0%	10 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #644     +/-   ##
=======================================
- Coverage   47.7%   45.2%   -2.6%     
=======================================
  Files        228     229      +1     
  Lines      27205   27016    -189     
=======================================
- Hits       12983   12216    -767     
- Misses     14222   14800    +578     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yuunlimm]

Merge activity

• Dec 16, 5:52 PM EST: A user merged this pull request with Graphite.