Incorporated some items and added TODOs based on unresolved items from …

…#12450

CHANGELOG.md

@@ -1,7 +1,7 @@
 # Lotus changelog
 
 # UNRELEASED
- (`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxIndex` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293).  If you are an RPC provider/node operator, please refer to the [ChainIndexer documentation for RPC providers](TODO: URL) for information on how to enable, configure and use the new Indexer.
+(`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxHashLookup` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293).  If you are an RPC provider/node operator, please refer to the [ChainIndexer documentation for RPC providers](TODO: URL) for information on how to enable, configure and use the new Indexer.  While there is no migration and one can upgrade and downgrade without backups, there are manual steps that need to be taken to backfill data for the new Indexer.
 Add `EthGetBlockReceipts` RPC method to retrieve transaction receipts for a specified block. This method allows users to obtain Ethereum format receipts of all transactions included in a given tipset as specified by its Ethereum block equivalent. ([filecoin-project/lotus#12478](https://github.com/filecoin-project/lotus/pull/12478))
 
 

chain/index/chain-indexing-overview-for-rpc-providers.MD

@@ -1,7 +1,22 @@
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1772125029
+move this to lotus-docs
+
 # ChainIndexer Documentation for RPC Providers
 
 ## Introduction
 
+---
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792731823
+use different language than "RPC providers"
+Some nuance is needed here - everyone running a Lotus node serves RPC requests, SPs use lotus as an RPC endpoint for their miner (lotus-miner or curio).
+
+Something like "externally-available RPC" or "public API", or "RPC other than for internal consumption", or even better, target this document at those requiring: "high-performance RPC", which may include some SPs.
+
+As we move forward, SPs are likely going to need some of this functionality anyway—consider PDP which is smart-contract mediated, they're going to need eth_getLogs and friends, so turning this on might start to make sense.
+
+So, limiting this to "RPC providers" is probably not the best because (a) that term isn't precise enough and (b) there's grey area about who it might make sense to use this. You could even introduce a section that describes the functionality so it helps people decide for themselves.
+---
+
 This document is for RPC providers and node operators who serve RPC requests walking through the configuration changes, migration flow and operations/maintenance work needed to enable, backfill and maintain the [`ChainIndexer`](#chainindexer-indexing-system).
 
 **Note: If you are a Storage Provider or node operator who does not serve RPC requests (i.e, if `Fevm.EnableEthRPC = false`), you can skip this document as the `ChainIndexer` is already disabled by default**. 
@@ -25,11 +40,18 @@ The following must be enabled on an Lotus node before starting as they are disab
   EnableActorEventsAPI = true
 ```
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1784677150
+document when EnableEthRC == true && EnableActorEventsAPI == true but EnableIndexer == false
+
 You can learn more about these configuration options and other configuration options available for the `ChainIndexer` [here](https://github.com/filecoin-project/lotus/blob/master/documentation/en/default-lotus-config.toml).
 
 
 ### Garbage Collection
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792736417
+Can we give a very rough estimate of growth to help with informed decisions here? e.g.:
+"At the time of writing, ChainIndexer will accumulate approximately 120MiB per day of data".
+
 The `ChainIndexer` includes a garbage collection (GC) mechanism to manage the amount of historical data retained. By default, GC is disabled to preserve all indexed data.
 
 To configure GC, use the `GCRetentionEpochs` parameter in the `ChainIndexer` section of your config.
@@ -52,6 +74,9 @@ The ChainIndexer [periodically runs](https://github.com/filecoin-project/lotus/b
 (*Example:* if your node is configured to retain 2 days of Filecoin chain state with the Splitstore, set `GCRetentionEpochs` to `retentionDays * epochsPerDay = 2 * 2880 = 5760`).
 **Warning:** Setting this value below the chain state retention period will degrade RPC performance and reliability because the Index will lack data for epochs still present in the chain state.  
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792740160
+> "if your node is configured to retain 2 days of Filecoin chain state with the Splitstore" isn't helpful because nobody configures it like this, so you need to say something here to align with how splitstore works
+
 ### Removed Options
 
 **Note: The following config options no longer exist in Lotus and have been removed in favor of the ChainIndexer config options explained above. They can be removed when upgrading to Lotus v1.31.0.**
@@ -60,6 +85,9 @@ The ChainIndexer [periodically runs](https://github.com/filecoin-project/lotus/b
 [Fevm]
 EthTxHashMappingLifetimeDays = 0
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792743250
+> see note in #12421, this one is duplicated with a moved redirect
+
 [Events]
 DisableHistoricFilterAPI = false
 DatabasePath = ""
@@ -69,8 +97,14 @@ DatabasePath = ""
 
 > **Note:** One can upgrade/downgrade between [pre-ChainIndexer](#previous-indexing-system) and [with-ChainIndexer](#chainindexer-indexing-system) Lotus versions without conflict because they persist state to different directories and don't rely on each other. No backup is necessary (but extra backups don't hurt).  
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792744673
+> yeah .. but it's more complicated than this, because if you go back to a pre-ChainIndexer version then you'd need to backfill to fill in the holes you created in the period you were running the with-ChainIndexer version. I think you need to document this somewhere because you can't just hand-wave at this problem for people.
+
 Upgrading to the new `ChainIndexer` involves these steps:
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1771226124
+Time estimates for the migration for Snapshot synced and archival would also be good to add at the top level here.
+
 1. **Stop the Lotus Node**
    - Stop your Lotus node before starting the upgrade and migration process.
 2. **Update Configuration**
@@ -80,15 +114,24 @@ Upgrading to the new `ChainIndexer` involves these steps:
    - The `ChainIndexer` will begin indexing **real-time chain state changes** immediately in the `${LOTUS_PATH}/chainindex` directory.
    - **However, it will not automatically index any historical chain state (i.e., any previously existing chain state prior to the upgrade). To perform backfilling, please see the [`Backfill` section below](#backfill).**
 
+TODO: from https://github.com/filecoin-project/lotus/pull/12450#discussion_r1771220740
+Could be good to add some migration time estimates here when moving to the ChainIndexer?
+
 > **Note:** It's recommended to keep the [pre-ChainIndexer](#previous-indexing-system) indexing database directory (`${LOTUS_PATH}/sqlite`) around until you've confirmed you don't need to [downgrade](#downgrade).  After sustained successful operations after the upgrade, the [pre-ChainIndexer](#previous-indexing-system) indexing database directory can be removed to reclaim disk space.  
 
 ## Backfill
 There is no automated migration from [pre-ChainIndexer indices](#previous-indexing-system) to the [ChainIndex](#chainindexer-indexing-system).  Instead one needs to index historical chain state (i.e., backfill) using one of the following tools.
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792801381
+Add more infor here
+
 > **Note:** The ChainIndex will consume ~10GB of storage per month of tipsets (e.g., ~86400 epochs).  Ensure you have sufficient disk space before proceeding.
 
 ### `ChainValidateIndex` JSON RPC API
 
+TODO: https://github.com/filecoin-project/lotus/pull/12450#discussion_r1792818109
+Move this out
+
 Please refer to the [Lotus API documentation](https://github.com/filecoin-project/lotus/blob/master/documentation/en/api-v1-unstable-methods.md) for detailed documentation of the `ChainValidateIndex` JSON RPC API.
 
 The `ChainValidateIndex` JSON RPC API serves a dual purpose: it validates/diagnoses the integrity of the index at a specific epoch (i.e., it ensures consistency between indexed data and actual chain state), while also providing the option to backfill the `ChainIndexer` if it does not have data for the specified epoch. 
@@ -214,7 +257,7 @@ In case you need to downgrade to the [previous indexing system](#previous-indexi
 ### Previous Indexing System
 * This corresponds to the indexing system used in Lotus versions before v1.31.0.  
 * It has been replaced by the [ChainIndex](#chainindexer-indexing-system).
-* It was composed of three indexers using three separate databases: [`EthTxHashLookup`](https://github.com/filecoin-project/lotus/blob/master/chain/ethhashlookup/eth_transaction_hash_lookup.go), [`MsgIndex`](https://github.com/filecoin-project/lotus/blob/master/chain/index/msgindex.go), and [`EventIndex`](https://github.com/filecoin-project/lotus/blob/master/chain/events/filter/index.go).
+* It was composed of three indexers using three separate databases: [`EthTxHashLookup`](https://github.com/filecoin-project/lotus/blob/v1.31.0/chain/ethhashlookup/eth_transaction_hash_lookup.go), [`MsgIndex`](https://github.com/filecoin-project/lotus/blob/v1.31.0/chain/index/msgindex.go), and [`EventIndex`](https://github.com/filecoin-project/lotus/blob/v1.31.0/chain/events/filter/index.go).
 * It persisted state to the [removed option](#removed-options) for `Events.DatabasePath`, which defaulted to `${LOTUS_PATH}/sqlite`.
 * It had CLI backfill tooling: `lotus-shed index backfill-*`