Spark DDO support for Curio

[bajtos]

The initial DDO support (#218) left out Curio, only Boost and Venus are supported.

Let's improve Spark to be able to test FIL+ DDO deals created by Curio software too.

Known tasks:

Piece Indexer side

• Support http-path in index provider address piece-indexer#117

• Support DAG-CBOR encoded entries piece-indexer#119

• Improve piece-indexer to check if the advertisement's ContextID is formatted in the new way described in FRC: Retrieval Checking Requirements filecoin-project/FIPs#1089. If it is, extract PieceCID from there.
  → converted to Extract PieceCID from ContextID piece-indexer#118

• Verify that piece-indexer is adding PieceCID->PayloadCID mappings for Curio SPs.

  • Change the data in our production Redis instance and delete the ingestion status for provider peer ID 12D3KooWJ91c6xQshrNe7QAXPFAaeRrHWq2UrgXGPf8UmMZMwyZ5 to trigger re-scan of all advertisements. (This provider is running Curio, see Support http-path in index provider address piece-indexer#117).
  • Wait until we process enough advertisements. You can check progress here: https://pix.filspark.com/ingestion-status/12D3KooWJ91c6xQshrNe7QAXPFAaeRrHWq2UrgXGPf8UmMZMwyZ5
  • Verify that piece-indexer could extract PieceCID->PayloadCID mappings from the advertisements, e.g. by inspecting piecesIndexed in the ingestion-status response.

spark-deal-observer

• Modify deal-observer and spark (checker) to support the new way for linking MinerID and IndexProviderID, see the discussion in FRC: Retrieval Checking Requirements filecoin-project/FIPs#1089 (comment) and feat: spark contract filecoin-project/curio#377

Spark Checker

• Modify the Spark checker node to support the new linking mechanism too

• Verify that Spark can retrieve content announced by Curio to IPNI.

  • TODO: How to find a payload ID to test? Inspect piece-indexer Redis state?
  • See also Support retrievals from multiaddrs with http-path segment #239
  • bafkreih74ljilat42tptlywlnuejfkyigun6ahor5gmkxemqkvvdftasg4 could be a good one to use. However, at the time of writing this comment, the SP was advertising an incorrect retrieval address. (cid.contact query, correct retrieval address).

Finale

• Announce Spark's support for Curio (both f05 and DDO deals)
  • Onboarding rate and gas optimizations filecoin-project/FIPs#1092 (comment)
  • Slack channels