feat: lotus-shed tooling for chain indexer #12474

akaladarshi · 2024-09-17T12:36:00Z

This PR is a part of: #12450

PR changes:

Adds lotus-shed indexes validate-chainindex to inspect and backfill the chain indexer through ChainValidateIndex API.

akaladarshi · 2024-09-17T12:39:21Z

@aarshkshah1992

inspect-chainindex cmd uses chainValidateIndex API and also fetches manually from the chain indexer to inspect the data by comparing with chain store data.

Should I keep it that way or remove it because ChainIndexValidate API is already verifying the indexed data ?.

aarshkshah1992 · 2024-09-17T12:54:29Z

@akaladarshi Let's remove it because ChainIndexValidate is already validating the indexed data.

akaladarshi · 2024-09-17T12:56:01Z

@akaladarshi Let's remove it because ChainIndexValidate is already validating the indexed data.

Ok sure.

@aarshkshah1992
Also right now, in inspect we are just doing counts mostly, should we check the data as well?

aarshkshah1992 · 2024-09-17T12:58:13Z

@akaladarshi Yeah we can add data validation as well in a follow-up PR. That would be great 👍
Can you raise a PR to validate the data in addition to the counts against #12450 ?

However, please note that that should be done in the API, not in the lotus-shed command here.

aarshkshah1992 · 2024-09-17T12:58:50Z

@akaladarshi But let's do that once we have this lotus-shed command implementation in place.

akaladarshi · 2024-09-17T18:13:03Z

cmd/lotus-shed/indexes.go

@@ -34,6 +35,15 @@ const (
 	insertEntry      = `INSERT OR IGNORE INTO event_entry(event_id, indexed, flags, key, codec, value) VALUES(?, ?, ?, ?, ?, ?)`
 	upsertEventsSeen = `INSERT INTO events_seen(height, tipset_key_cid, reverted) VALUES(?, ?, false) ON CONFLICT(height, tipset_key_cid) DO UPDATE SET reverted=false`
 	tipsetSeen       = `SELECT height,reverted FROM events_seen WHERE tipset_key_cid=?`
+
+	getEthTxHashCountForTipset = `SELECT COUNT(*) FROM eth_tx_hash WHERE message_cid IN (SELECT message_cid FROM tipset_message WHERE tipset_key_cid = ? AND reverted = 0)`


@aarshkshah1992

should we return data related to eth_tx_hash from ChainValidateIndex API in IndexValidation as well ?.

@akaladarshi After thinking this through, I think we should return data for ETH TX hashes, however for now, let's just continue validating counts like we are currently doing. We can validate the actual contents as well as part of a future work stream but I think what we have is good for now..

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

PR title now matches the required format.

aarshkshah1992 · 2024-09-18T11:26:55Z

cmd/lotus-shed/indexes.go

+		},
+		&cli.BoolFlag{
+			Name:  "backfill",
+			Usage: "Backfill missing index entries while validating the chain index. When enabled, the command will perform backfilling for any missing indexes (default: true)",


"will perform backfilling for any missing epochs in the index"

aarshkshah1992 · 2024-09-18T11:33:42Z

cmd/lotus-shed/indexes.go

+		if fromEpoch == 0 {
+			curTs, err := api.ChainHead(ctx)
+			if err != nil {
+				return err


need an informative error message

aarshkshah1992 · 2024-09-18T11:34:46Z

cmd/lotus-shed/indexes.go

+			}
+			fromEpoch = int(curTs.Height()) - 1
+		} else {
+			fromEpoch = fromEpoch - 1


why do we need this ?

Only do this is fromEpoch >= head.Height()

This was done because ChainValidateIndex fetches event of ts from ts+1, So in case of fromEpoch == head.height(), we can't get ts.Height() + 1.

So due to nature of ChainValidateIndex by default fromEpoch will not be included in backfilling, and to maintain consistency I did fromEpoch -1.

aarshkshah1992 · 2024-09-18T11:36:05Z

cmd/lotus-shed/indexes.go

+
+		toEpoch := cctx.Int("to")
+		if toEpoch > fromEpoch {
+			return fmt.Errorf("to epoch must be less than from epoch")


errors.New()

aarshkshah1992 · 2024-09-18T11:38:04Z

cmd/lotus-shed/indexes.go

+			fromEpoch, toEpoch)
+
+		// starting from the FromEpoch-1 and going down to the ToEpoch
+		// this is because `ChainValidateIndex` fetches the tipset.height()+1, which might not be available in case of chain head


this comment is not clear. Wdym by "ChainValidateIndex fetches the tipset.height()+1" ?

aarshkshah1992 · 2024-09-18T11:41:09Z

cmd/lotus-shed/indexes.go

+				if failfast {
+					return fmt.Errorf("failed to validate index for epoch %d: %w", epoch, err)
+				}
+				log.Warnf("Error validating index for epoch %d: %v", epoch, err)


use %w for errors.

aarshkshah1992 · 2024-09-18T11:43:15Z

cmd/lotus-shed/indexes.go

+				continue
+			}
+
+			if !indexValidateResp.TipSetKey.IsEmpty() || indexValidateResp.Height != uint64(epoch) && indexValidateResp.Backfilled == backfill {


I dont think we need this validation block here at all. The ChainIndexValidationAPI guruantees that it will return a valid indexValidateResp if error is nil.

aarshkshah1992 · 2024-09-18T11:45:58Z

@akaladarshi This looks mostly correct. Thanks for all the great work here 👍

Let me merge this, fill in the missing gaps and test it out e2e on a calibnet node. I will tag you for review on the changes I make.

@jennijuju

…for the `ChainIndexer` (#12450) * fix conflicts with chain indexer * feat: chain indexer todos [skip changelog] (#12462) * feat: finish todos of validation api * feat: add indexed data verification with chain store * feat: address comments and finish TODO * fix: build issue * address comments * fix: ci issue * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * changes to Index Validation API based on Rodds first review * build chain indexer API * improve error handling * feat: lotus-shed tooling for chain indexer (#12474) * feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log * backfilling improvements and fixes * finish chain index validation and backfill tooling * user documentation for the * validate from epoch * Apply suggestions from code review Suggestions from Steve's read of the user doc. Co-authored-by: Steve Loeppky <[email protected]> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * changes as per review * feat: add event entries count in validation API (#12506) * feat: add event entry count in validation API * address comments * use sqllite defaults (#12504) * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * write chain index to a different dir * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * fix conflicts * UX improvements to backfilling * feat: tests for the chain indexer (#12521) * ddl tests * tests for the chain indexer * finish unit tests for chain indexer * fix formatting * cleanup reverted tipsets to avoid db bloat * fix logging * test for filter by address * test gc cascade delete * fix db locked error during backfilling * fix var name * increase db locked timeout * fix db locked issue * reduce db lock timeout * no lock in gc * reconcile does not need lock * improved error handling * Update chain-indexing-overview-for-rpc-providers.md Doc updates based on @jennijuju feedack. * Update chain-indexing-overview-for-rpc-providers.MD Fixes after reviewing 33c1ca1 * better metrics for backfilling * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * tests for changes to event addressing * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * changes as per review -> round 1 * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * log tipset key cid * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix docs * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix tests * fix tests * make jen * fix conflicts --------- Co-authored-by: Aryan Tikarya <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steve Loeppky <[email protected]>

@jennijuju

* chain index complete for msgs and txns * dont need observer changes for now * changes * fix tests * fix tests * use th right context * index empty tipsets correctly * implement automated backfilling * add event indexing and remove all old indices * fix test * revert deployment test changes * revert test changes and better error handling for eth tx index lookups * fix sql statments naming convention * address review for Index GC * more changes as per review * changes as per review * fix config * mark events as reverted during reconciliation * better reconciliation; pens down and code complete; also reconcile events * fix tests * improve config and docs * improve docs and error handling * improve read logic * improve docs * better logging and handle ennable event storage * improve logs and index init proc * better logging * fix bugs based on calibnet testing * create sqliite Indices * gc should be based on epochs * fix event query * foreign keys should be enabled on the DB * reverted tipsets should be removed as part of GC * release read lock * make it easy to backfill an empty index using reconciliation * better docs for reconciliation * fix conflicts with master * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix go mod * fix formatting * revert config changes * address changes in observer * remove top level chainindex package * changes as per review * changes as per review * changes as per review * handle index with reverted tipsets during reconciliation * changes as per review * fix type of max reconcile epoch * changes to reconciliation as per review * log ipld error * better logging of progress * disable chain indexer hydrate from snapshot based on config * always populate index * make config easy to reason about * fix config * fix messaging * revert config changes * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * changes as per review * make error messages homogenous * fix indentation * changes as per review * feat: recompute tipset to generate missing events if event indexing is enabled (#12463) * auto repair events * make jen * fix leaky abstraction * better docs for gc retention epoch * imrpove DB handling (#12485) * fix conflict * fix lite node config for indexer * exclude reverted events from eth get logs if client queries by epoch * Simply addressing for event lookups in the index. simply addressing for event lookups * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix tests * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` (#12450) * fix conflicts with chain indexer * feat: chain indexer todos [skip changelog] (#12462) * feat: finish todos of validation api * feat: add indexed data verification with chain store * feat: address comments and finish TODO * fix: build issue * address comments * fix: ci issue * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * changes to Index Validation API based on Rodds first review * build chain indexer API * improve error handling * feat: lotus-shed tooling for chain indexer (#12474) * feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log * backfilling improvements and fixes * finish chain index validation and backfill tooling * user documentation for the * validate from epoch * Apply suggestions from code review Suggestions from Steve's read of the user doc. Co-authored-by: Steve Loeppky <[email protected]> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * changes as per review * feat: add event entries count in validation API (#12506) * feat: add event entry count in validation API * address comments * use sqllite defaults (#12504) * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * write chain index to a different dir * Apply suggestions from code review Co-authored-by: Steve Loeppky <[email protected]> * fix conflicts * UX improvements to backfilling * feat: tests for the chain indexer (#12521) * ddl tests * tests for the chain indexer * finish unit tests for chain indexer * fix formatting * cleanup reverted tipsets to avoid db bloat * fix logging * test for filter by address * test gc cascade delete * fix db locked error during backfilling * fix var name * increase db locked timeout * fix db locked issue * reduce db lock timeout * no lock in gc * reconcile does not need lock * improved error handling * Update chain-indexing-overview-for-rpc-providers.md Doc updates based on @jennijuju feedack. * Update chain-indexing-overview-for-rpc-providers.MD Fixes after reviewing 33c1ca1 * better metrics for backfilling * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <[email protected]> * tests for changes to event addressing * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * changes as per review -> round 1 * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * log tipset key cid * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix docs * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix tests * fix tests * make jen * fix conflicts --------- Co-authored-by: Aryan Tikarya <[email protected]> Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Steve Loeppky <[email protected]> * fix lint * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * remove reverted flag from RPC * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix testing of events and dummy chain store * remove lotus shed commands for old Indices * change type of event counts to uint64 * only recompute events if theyre not found * short-circuit empty events path for older tipsets * chain indexer must be enabled if ETH RPC is enabled * change name of message_id column to id in tipset_message table * only expose SetRecomputeTipSetStateFunc * dont block on head indexing for reading messages * document why we're only checking for missing events for a single tipset * document when we query for reverted events * simplify event collection * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * fix test * change event_id to id in the event table * change head indexed timeout * remove deprecated config options * fail ETH RPC calls if ChainIndexer is disabled * fix docs * remove the tipset key cid func from lotus shed * address review comments * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * chore(events): remove unnecessary DisableRealTimeFilterAPI (#12610) * feat(cli): add --quiet to chainindex validate-backfill + cleanups (#12611) * fix tests * Apply suggestions from code review Co-authored-by: Rod Vagg <[email protected]> * error type for disabled chainindexer * fix(chainindex): recompute tipset when we find no receipts * fix(chainindexer): backfilling should halt when chain state data is missing and not backfill parents (#12619) * fix backfilling UX * Update chain/index/api.go Co-authored-by: Rod Vagg <[email protected]> * address review --------- Co-authored-by: Rod Vagg <[email protected]> * reduce log noise * make jen * make jen * docs: finishing chain-indexer-overview-for-operators.md (#12600) * Followup to PR #12450 for doc updates This is being used to resolve the unresolved items in #12450 since that PR is unwieldly at this point. * Incorporated some items and added TODOs based on unresolved items from #12450 * Incorporating more feedback * Pointing to issue to learn about benefits * Formatting fixes * Apply most of the suggestions from @rvagg code review Co-authored-by: Rod Vagg <[email protected]> * Incorporating feedback from #12600 (comment) * Addressing #12600 (comment) and more * Moved chain-indexer docs to documentation Renamed Added ToC We can move to lotus-docs later * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <[email protected]> * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <[email protected]> * Added upgrade path when importing chain state from a snapshot. * Typo fixes * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <[email protected]> * chore(doc): "regular checks" section for chainindexer docs (#12612) * Apply suggestions from @rvagg code review Co-authored-by: Rod Vagg <[email protected]> * Incorporating @aarshkshah1992 feedback * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <[email protected]> --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Aarsh Shah <[email protected]> * remove go mod replace * remove unnecessary changes from CHANGELOG * fix test * compare events AMT root (#12632) * fix(chainindex): retry transaction if database connection is lost (#12657) * retry database lost connection * log context cancellation * address review * fix gateway itest: no chainindexer for lite nodes * fix changelog --------- Co-authored-by: Rod Vagg <[email protected]> Co-authored-by: Aryan Tikarya <[email protected]> Co-authored-by: Steve Loeppky <[email protected]>

aarshkshah1992 self-requested a review September 17, 2024 16:05

aarshkshah1992 assigned akaladarshi Sep 17, 2024

akaladarshi added 3 commits September 17, 2024 23:28

feat: add lotus-shed command for backfilling chain indexer

8908e07

feat: add lotus-shed command for inspecting the chain indexer

b37291e

feat: use single lotus-shed command to inspect and backfill

0dc4dba

akaladarshi force-pushed the akaladarshi/lotus-shed-chainindexer-tooling branch from fa127ed to 0dc4dba Compare September 17, 2024 18:09

fix: remove the unused queries

2777272

akaladarshi commented Sep 17, 2024

View reviewed changes

akaladarshi marked this pull request as ready for review September 17, 2024 18:14

github-actions bot previously requested changes Sep 17, 2024

View reviewed changes

akaladarshi added 2 commits September 18, 2024 10:57

small changes

fa30f23

add change log

cb141d7

github-actions bot requested changes Sep 18, 2024

View reviewed changes

akaladarshi changed the title ~~feat(lotus-shed): lotus-shed tooling for chain indexer~~ feat: lotus-shed tooling for chain indexer Sep 18, 2024

aarshkshah1992 reviewed Sep 18, 2024

View reviewed changes

aarshkshah1992 merged commit e7506cc into filecoin-project:feat/implement-index-validation-api Sep 18, 2024
77 of 79 checks passed

akaladarshi deleted the akaladarshi/lotus-shed-chainindexer-tooling branch September 19, 2024 05:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: lotus-shed tooling for chain indexer #12474

feat: lotus-shed tooling for chain indexer #12474

akaladarshi commented Sep 17, 2024 •

edited

Loading

akaladarshi commented Sep 17, 2024 •

edited

Loading

aarshkshah1992 commented Sep 17, 2024

akaladarshi commented Sep 17, 2024 •

edited

Loading

aarshkshah1992 commented Sep 17, 2024

aarshkshah1992 commented Sep 17, 2024

akaladarshi Sep 17, 2024

aarshkshah1992 Sep 18, 2024

github-actions bot left a comment

github-actions bot left a comment

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

akaladarshi Sep 18, 2024 •

edited

Loading

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

aarshkshah1992 Sep 18, 2024

aarshkshah1992 commented Sep 18, 2024

feat: lotus-shed tooling for chain indexer #12474

feat: lotus-shed tooling for chain indexer #12474

Conversation

akaladarshi commented Sep 17, 2024 • edited Loading

akaladarshi commented Sep 17, 2024 • edited Loading

aarshkshah1992 commented Sep 17, 2024

akaladarshi commented Sep 17, 2024 • edited Loading

aarshkshah1992 commented Sep 17, 2024

aarshkshah1992 commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akaladarshi Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarshkshah1992 commented Sep 18, 2024

akaladarshi commented Sep 17, 2024 •

edited

Loading

akaladarshi commented Sep 17, 2024 •

edited

Loading

akaladarshi commented Sep 17, 2024 •

edited

Loading

akaladarshi Sep 18, 2024 •

edited

Loading