feat: markets: Integrate index ingest protocol and retrieve by any CID

[dirkmc]

Related Issues

The motivation for this work is to solve content routing in Filecoin network, and facilitate retrieval of content by any CID.

Fixes:

• API support for libp2p protection #8015

Proposed Changes

The work here integrates the indexing ingestion protocol into lotus using the default implementation.

As a result, the multihashes contained in successful deals are announced onto indexer ingest gossipsub topic. Similarly, multihashes that belonged to slashed or expired deals are advertised as no longer available. The work exposes an endpoint on the markets process to allow indexer nodes to sync the list of multihashes for processing. The indexers use this information to then provider a service where a user can look up which SP provides a given CID or Multihash.

The work also introduces a peer protection API that allows dynamic change to the list of peer IDs the connection to which is protected by the daemon process. This is introduced so that: 1) there is no need to restart the daemon in order to make changes to this config parameter, and 2) automatically protect connections from markets to daemon process to allow indexing gossip messages to propagate through the network via Damon's connection to Filecoin bootstrap peers.

The lotus CLI is expanded to provide a set of utility commands that allow bulk announcements of all deals to indexers, along with functionality to list and modify peer IDs protected by the daemon process.

Additional Info

Discussion:

• SPX Index Provider Testing #8087

Integrates indexing integration work done in go-fil-markets:

• retrieve by any CID (not just root CID) and reference provider integration go-fil-markets#629
• Index publishing work go-fil-markets#673

Additionally, a range of work is done in filecoin-project/dagstore to accommodate the indexing integration as well as rolling out a new CARv2 index, that stores the multihash code as well as the multihash digest in the CARv2 indices.

The core index provider functionality integrated is implemented in filecoin-project/index-provider. The Indexing protocol and indexer node can be found in filecoin-project/storetheindex.

Checklist

Before you mark the PR ready for review, please make sure that:

• All commits have a clear commit message.
• The PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, INTERFACE BREAKING CHANGE, CONSENSUS BREAKING, build, chore, ci, docs,perf, refactor, revert, style, test
  • area: api, chain, state, vm, data transfer, market, mempool, message, block production, multisig, networking, paychan, proving, sealing, wallet, deps
• This PR has tests for new functionality or change in behaviour
• If new user-facing features are introduced, clear usage guidelines and / or documentation updates should be included in https://lotus.filecoin.io or Discussion Tutorials.
• CI is green

[codecov]

Codecov Report

Merging #7313 (4c8015c) into master (6b06c1b) will decrease coverage by 0.20%.
The diff coverage is 32.24%.

@@            Coverage Diff             @@
##           master    #7313      +/-   ##
==========================================
- Coverage   40.67%   40.46%   -0.21%     
==========================================
  Files         666      675       +9     
  Lines       72932    73500     +568     
==========================================
+ Hits        29662    29741      +79     
- Misses      38071    38527     +456     
- Partials     5199     5232      +33     

Impacted Files	Coverage Δ
api/api_storage.go	0.00% <ø> (ø)
cli/net.go	8.48% <0.00%> (-1.52%)	⬇️
cmd/lotus-miner/index_provider.go	0.00% <0.00%> (ø)
cmd/lotus-miner/main.go	4.20% <0.00%> (-0.04%)	⬇️
lib/oldpath/oldresolver/resolver.go	0.00% <0.00%> (ø)
lib/oldpath/path.go	0.00% <0.00%> (ø)
markets/idxprov/mesh.go	0.00% <0.00%> (ø)
metrics/metrics.go	100.00% <ø> (ø)
node/builder.go	76.47% <ø> (ø)
node/builder_chain.go	100.00% <ø> (ø)
... and 50 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b06c1b...4c8015c. Read the comment docs.

[magik6k]

Are there any setup docs you could link in this PR?

[willscott]

#8087 has a lot of docs around it, currently

[magik6k]

Feels very close to landing. CI is failing.

Looks like there is an unintentional ffi update pushed.