Implementation plan for direct data onboarding proposal' in Lotus-Miner

[rjan90]

Description
Work plan for the Lotus-Miner implementation of implementing the direct data onboarding proposal in lotus-miner. The proposal aims to simplify data onboarding to sectors for Storage Providers (SPs) and reduce data onboarding cost.

LM Tasks

Beta Give feedback

1. Expand SectorAddPieceToAny to accept new allocation info (~1 hour)
2. Make the sealing pipeline input code be aware of new allocation fields for sector duration calculations (~1 day)
3. Lotus miner needs to use new prove_replica_updates2 #11192
   kind/feature +1
   
4. Inspect other accesses to api/SectorPiece.DealInfo in the pipeline, and extend those to support new allocation fields (~1-2 days)
5. Support new Precommit/Commit methods - switch to using new precommit_batch2
6. Support new Precommit/Commit methods - Add codepaths for new Commit
7. Integration testing (~3-5 days)

[magik6k]

What I see here:

• Lotus-Miner
  • Expand SectorAddPieceToAny to accept new allocation info (~1 hour)
    • This can be done by adding new optional fields to api/PieceDealInfo for the Allocation, no new method needed
  • Make the sealing pipeline input code be aware of new allocation fields for sector duration calculations (~1 day)
    • This should boil down to replacing instances of accesses to piece.DealInfo in input.go with a helper which can handle the new allocation fields
  • Inspect other accesses to api/SectorPiece.DealInfo in the pipeline, and extend those to support new allocation fields (~1-2 days)
    • This seems to be mostly just a handful of simple sanity-checks, and error-recovery functions
  • Support new Precommit/Commit methods
    • Switch to using the new precommit_batch2 (~1 day)
    • Add codepaths for new Commit
      • Should be a fairly straightforward new bit of code deciding on what to do based on the Precommit message (~0.5 day)
• Integration testing
  • If we can write some ITests which just interact with SectorAddPieceToAny without the markets subsystem, and exercise the new logic this way, without changes to legacy markets.
    • If we want to fully deprecate legacy markets, we will need a new framework for making deals in integration tests anyways, so this work will be quite useful even outside supporting this proposal
    • Initially this should just be adding a number of test-cases adding coverage to the new codepaths, plus the new deal test harness
    • ~3-5 days
• Lotus
  • Unclear if the proposal will depend on clients/SPs consuming events from native actors, if yes, this may mean some new APIs

[jennijuju]

Unclear if the proposal will depend on clients/SPs consuming events from native actor

so far no - https://github.com/FILCAT/fil-pm/issues/4 i think taking the async approach is better

[jennijuju]

@magik6k
skimming this through:

• there will be a new PRU method as well to add
• need some design discussion: should lotus miner database to have a field to capture the market that the deal is participated in?
• the estimation seems a bit too optimistic to me, especially for the new commit code path. im interested in what kind of SP ux do we plan to offer here

[rjan90]

• If we can write some ITests which just interact with SectorAddPieceToAny without the markets subsystem, and exercise the new logic this way, without changes to legacy markets.
  • If we want to fully deprecate legacy markets, we will need a new framework for making deals in integration tests anyways, so this work will be quite useful even outside supporting this proposal

If writing a new framework I think we should also consider doing the sealing/proving through lotus-workers and not the built-in lotus-miner workers, such that #10870 will not break the tests, and an eventual path for depreciation of the built-in worker is more straight forward.

[anorth]

Support new Precommit/Commit methods

• Switch to using the new precommit_batch2 (~1 day)

There will be new PRU methods too, and similarly there's an old legacy one to be deprecated. Can Lotus miner switch to prove_replica_updates2? The difference, as for pre-commit, is that the new unsealed CID is required as a parameter.

I filed #11192 to track this, please track appropriately.

[rjan90]

Closing as #11226 has been merged into the feat/nv22 branch