Scheduler improvement ideas - Lotus/Boost - 100% hardware utilisation 24x7x365

[LexLuthr]

This Discussion will be used to brainstorm the ideas to improve scheduling mechanism to maximise the efficiency of the sealing pipeline

Historical Context:
In the Boost AMA, we received feedback from larger SPs that they want to be able to utilise their current pipeline to 100% efficiency with no gaps. Any idle CPU/GPU is money lost. On further clarification, this resulted in 2 asks:

1. Boost should be scalable to avoid bottlenecks in data onboarding - In progress (Boost)
2. Sealing pipeline should never sit ideal for any amount of time

The second point was further discussed with evergreen team and a more detailed feedback emerged from there. SPs would like the ability to chose when data enters the pipeline so they can they schedule jobs to maximise the efficiency. But, this request is not really focused on the ability to chose alone but ability to chose in a way that hardware is occupied 100% of the time. Another reason for this request was to avoid accepting so many deal that total data received > sealing capacity per day.

After internal discussion within the Boost team, we came up with following ideas.

1. Allowing the ability to chose when data enters pipeline can be done on the market side. But, this will create a new problem where SPs need to monitor their dashboard all the time and they might miss deal start epochs. Not a good approach
2. We can create a very complicated scheduler on market side for addpiece() but that might just contradict with lotus internal scheduling logic. It would also be painful to maintain and will pull focus from market development. Not a good approach
3. Boost can allow a daily cap in the deal acceptance within the config file so total data received per day <= sealing capacity per day. This is a doable ask without any major drawbacks.

Current State:
#8700 should fix the efficiency problem upto a certain degree. @magik6k Can probably clarify more on this.

** Goal:**
This discussion will be used a unified platform to discuss ideas to achieve the 100% hardware utilisation 24x7x365.
Boost and Lotus team will collaborate here to find the best possible solution and then we can open up this discussion for feedback from SPs once we have a clear idea on how to solve the problem.

CC: @dirkmc @nonsense @brendalee @jennijuju @magik6k @arajasek

Please feel free to tag other people for inputs.

[magik6k]

We need scheduling/throttling on the market side, and the reason for this is that as soon as a deal proposal is signed, due to the start epoch there's a limit on how long we have to get that deal into a sector (which includes data transfer + sealing)

Because both boost and sealing pipeline sides have limits on how much data they can handle in a given amount of time (and accepting deals (committing to process data) is very cheap, we can accept 100 000s of proposals with compute resources it takes to process one), both of those need scheduling logic.

Even right now lotus has back pressure on the SectorAddPieceToAny call used by market processes to add deal data into sectors - the call will block until the piece was added to a sector - if the pipeline is full, the call will block. This works well enough to prevent the sealing pipeline from being overloaded, but is a bit lagging for markets/boost which can commit to/transfer a lot of data before it gets blocked in the sealing pipeline.

On the lotus side we can add a new API method which would return usage stats of the sealing pipeline, with things like:

• Amount of data (sector count) which can be added without blocking
• (when cc is preferred over snapdeals with higher limit for snapdeals - 'surge snapdeals mode') Amount of data (sector count) which can be added without creating snapdeals sectors
• Amount of data which is about to leave the sealing pipeline
  • List of sectors so it's possible to tag/reserve this storage as those sectors move out of the sealing pipeline
  • Likely sectors in Commit2 and later states, which are in most cases 30min (+-10min) away from leaving the pipeline
  • This would allow boost to decide to start fetching data sooner
• (maybe) Stats on projected time-to-seal (expected, worst-case, best-case)

[nonsense]

Boost has back-pressure in the form of a staging area, and we start rejecting deals when the staging area is full (i.e. Boost has accepted and transferred 100GiB worth of deals, that it hasn't been able to pass to the sealer for some reason).

The issue as I understand it is that the back-pressure from Lotus in the form of SectorAddPieceToAny blocking is not good enough, because when it is not-blocking, it means that the sealing pipeline is available/free, and can process a new piece. The problem is that if we keep pushing pieces from Boost to Lotus when SectorAddPieceToAny allows, we are not getting 100% hardware utilisation, and SPs are not happy. SPs therefore are requesting work-arounds in the form of "manually triggering SectorAddPieceToAny when they decide that this will result in 100% hw utilisation".

Is it possible to update the scheduler behind SectorAddPieceToAny to accept pieces when this will result in 100% utilisation?

My understanding of the issue is that SPs have to carefully orchestrate the sealer, and want to delay forwarding pieces from Boost to Lotus via SectorAddPieceToAny, even if the call was not to block, so that they maintain the most efficient pipeline. I am not sure that this is the best solution to their problem.

[LexLuthr]

Use case from one of the big miner:

1. I have sealing workers of different types, but the bottleneck is the PC1 sealing workers, say I have 4 workers and each of them can seal 9 PC1 at the same time, my script will make sure to have 36 PC1 sealing at the same time (to the best effort). I never use MaxSealingSectors or MaxSealingSectorsForDeal because the bottleneck is not the total number of jobs but the PC1 jobs, at least for me.

2. When one sealing worker needs to be shutdown, I will change my scheduling configuration to reduce the sealing workers by 1, so that the script will now only allow 27 PC1 running at the same time - it happens in real time

3. I use some special math to handle deal publishing - I want to accumulate a few deals in one publish message while also waiting for the gas to drop over the next 30minutes to 1 hour. If I have too many deals in one message, that may overflow my sealing pipeline since one published, they goes to add piece directly. Here I think having a trigger to start adding piece for staged deals would be helpful

4. I accept deals from multiple sources - there are deals made manually from other clients which I need to import manually, there are deals from bidbot, there are deals from evergreen. So I define precedence in my scheduling configuration.

5. I also made changes in lotus source code to do a symbolic link when importing car file and also skip commp verification if I'm 100% sure the CAR file matches by looking at the CAR header. This helps increase the throughput of deal importing