feat: migration("re-indexing"), backfilling and diasgnostics tooling …

…for the `ChainIndexer` (#12450)

* fix conflicts with chain indexer

* feat: chain indexer todos [skip changelog] (#12462)

* feat: finish todos of validation api

* feat: add indexed data verification with chain store

* feat: address comments and finish TODO

* fix: build issue

* address comments

* fix: ci issue

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* changes to Index Validation API based on Rodds first review

* build chain indexer API

* improve error handling

* feat: lotus-shed tooling for chain indexer (#12474)

* feat: add lotus-shed command for backfilling chain indexer

* feat: add lotus-shed command for inspecting the chain indexer

* feat: use single lotus-shed command to inspect and backfill

* fix: remove the unused queries

* small changes

* add change log

* backfilling improvements and fixes

* finish chain index validation and backfill tooling

* user documentation for the

* validate from epoch

* Apply suggestions from code review

Suggestions from Steve's read of the user doc.

Co-authored-by: Steve Loeppky <[email protected]>

* changes to user doc as per review

* Apply suggestions from code review

Co-authored-by: Steve Loeppky <[email protected]>

* changes to user doc as per review

* Apply suggestions from code review

Co-authored-by: Steve Loeppky <[email protected]>

* changes as per review

* feat: add event entries count in validation API (#12506)

* feat: add event entry count in validation API

* address comments

* use sqllite defaults (#12504)

* Apply suggestions from code review

Co-authored-by: Steve Loeppky <[email protected]>

* write chain index to a different dir

* Apply suggestions from code review

Co-authored-by: Steve Loeppky <[email protected]>

* fix conflicts

* UX improvements to backfilling

* feat: tests for the chain indexer (#12521)

* ddl tests

* tests for the chain indexer

* finish unit tests for chain indexer

* fix formatting

* cleanup reverted tipsets to avoid db bloat

* fix logging

* test for filter by address

* test gc cascade delete

* fix db locked error during backfilling

* fix var name

* increase db locked timeout

* fix db locked issue

* reduce db lock timeout

* no lock in gc

* reconcile does not need lock

* improved error handling

* Update chain-indexing-overview-for-rpc-providers.md

Doc updates based on @jennijuju feedack.

* Update chain-indexing-overview-for-rpc-providers.MD

Fixes after reviewing 33c1ca1

* better metrics for backfilling

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* Update chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <[email protected]>

* tests for changes to event addressing

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* changes as per review -> round 1

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* log tipset key cid

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* fix docs

* Apply suggestions from code review

Co-authored-by: Rod Vagg <[email protected]>

* fix tests

* fix tests

* make jen

* fix conflicts

---------

Co-authored-by: Aryan Tikarya <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Steve Loeppky <[email protected]>

CHANGELOG.md

@@ -1,6 +1,13 @@
 # Lotus changelog
 
 # UNRELEASED
+ (`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxIndex` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293).  If you are an RPC provider/node operator, please refer to the [ChainIndexer documentation for RPC providers](TODO: URL) for information on how to enable, configure and use the new Indexer.
+Add `EthGetBlockReceipts` RPC method to retrieve transaction receipts for a specified block. This method allows users to obtain Ethereum format receipts of all transactions included in a given tipset as specified by its Ethereum block equivalent. ([filecoin-project/lotus#12478](https://github.com/filecoin-project/lotus/pull/12478))
+
+
+## ☢️ Upgrade Warnings ☢️
+
+- Minimum go-version been updated to v1.22.7 ([filecoin-project/lotus#12459](https://github.com/filecoin-project/lotus/pull/12459))
 
 ## New features
 - Update `EthGetBlockByNumber` to return a pointer to ethtypes.EthBlock or nil for null rounds. ([filecoin-project/lotus#12529](https://github.com/filecoin-project/lotus/pull/12529))

api/api_full.go

@@ -64,6 +64,37 @@ type FullNode interface {
 	Common
 	Net
 
+	// MethodGroup: ChainIndexer
+	// The ChainIndexer method group contains methods for interacting with the chain indexer.
+
+	// ChainValidateIndex validates the integrity of and optionally backfills
+	// the chain index at a specific epoch.
+	//
+	// It can be used to:
+	//
+	// 1. Validate the chain index at a specific epoch:
+	//   - Ensures consistency between indexed data and actual chain state
+	//   - Reports any errors found during validation (i.e. the indexed data does not match the actual chain state, missing data, etc.)
+	//
+	// 2. Optionally backfill missing data:
+	//   - Backfills data if the index is missing information for the specified epoch
+	//   - Backfilling only occurs when the `backfill` parameter is set to `true`
+	//
+	// 3. Detect "holes" in the index:
+	//   - If `backfill` is `false` and the index lacks data for the specified epoch, the API returns an error indicating missing data
+	//
+	// Parameters:
+	//   - epoch: The specific chain epoch for which to validate/backfill the index.
+	//   - backfill: A boolean flag indicating whether to attempt backfilling of missing data if the index does not have data for the
+	//               specified epoch.
+	//
+	// Returns:
+	//   - *types.IndexValidation: A pointer to an IndexValidation struct containing the results of the validation/backfill.
+	//   - error: An error object if the validation/backfill fails. The error message will contain details about the index
+	//            corruption if the call fails because of an incosistency between indexed data and the actual chain state.
+	//            Note: The API returns an error if the index does not have data for the specified epoch and backfill is set to false.
+	ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error) //perm:write
+
 	// MethodGroup: Chain
 	// The Chain method group contains methods for interacting with the
 	// blockchain, but that do not require any form of state computation.