[New Feature]: Improve performance for Rebuild pipeline

[denelon]

Description of the new feature/enhancement

The latest service interruption involved an attempt to run the rebuild pipeline to completely restore the index and build all manifests as a part of the BCDR (Business Continuity Disaster Recovery) process. This run took approximately 21 hours.

This goal of this feature is to reduce the amount of time required to rebuild the WinGet PreIndexed source cache and the package manifests.

Proposed technical implementation details (optional)

No response

[stephengillie]

On the spectrum between BC and DR, it sounds like we're way too far over on the DR side, and this work should push us back towards BC. It's not that data was lost or corrupted. But that the recovery process takes too long, possibly because it requires too much rebuilding or reconfiguration. One way to look at this is that more intermediate steps are needed, usually created by decoupling the existing process steps into more steps. So that recovery can be just re-running the last step or few in the process, or only running the essential steps and skipping non-essential steps.