fix(prover): Parallelize circuit metadata uploading for BWG

[EmilLuta]

Basic Witness Generation (BWG) creates circuits for base layer to be proven. BWG runs out-of-circuit VM and provides circuits for the instructions in the execution (note that those are ordered, messing with the order would break proving).

Additionally, harness will do heavy computation -> provide circuits -> heavy computation -> provide circuits. As such, providing circuits is rather fast. Once a circuit is produced, it's sent to BWG, which in turn sends it to GCS and saves it to DB. The queue between BWG runner and harness is of size 1. It was picked in this way to ensure that harness won't overwhelm bwg runner with circuits (if the queue was big, the amount of RAM used would be size of queue * size of circuit). The problem is that the harness will be throttled on circuit submission. It needs to blockingly wait for bwg runner to upload data (GCS + PG).

There are 2 alternatives here:

• make the buffer bigger -- this would work, but it would start consuming more memory. In theory, if you can match the size of the buffer to fit in between CPU cycles, harness wouldn't need to wait for upload and bwg runner would catch up by the time harness has a new set of circuits. The problem with this approach is that it relies on deterministic upload time (nothing can be deterministic in distributed systems) and fixed circuit size (this is not true, because batches can have varying number of circuits, of varying types).
• upload everything async -- this work wonders, but there will be multiple circuits in flight being uploaded at any time. This will also increase RAM. In my testing, I've seen increases smaller than 100MB (acceptable tradeoff). The edge cases are when GCS is down or not working as intended (for instance, refusing connection). In such scenarios the async tasks will grow, adding to a lot more RAM usage (up to 50GB). Whilst this is a real problem, I consider it more of an edge case. Furthermore, there are a few ways to make GCS more reliable.

The second option was picked here. MPSC channel (the buffer) is still of size 1 to have a single point of failure when RAM goes up (we don't want to figure out -- do we have too many things in the buffer, or do we have problems with GCS? -- in the current implementation, it will always be GCS). Furthermore, MPSC(1) doesn't cause much overhead.

The implementation has been tested locally, using mainnet and testnet batches. Whilst the number are super encouraging (00:03:58 time, with ~40GB RAM), the CPUs (and their speed) is slightly lower in GCP.

[0xVolosnikov]

Maybe we should add a limit on the number of simultaneous uploads? And throttle the harness until there is space in that "sending queue".
I am concerned about non-deterministic and almost unlimited data accumulation in memory

[0xVolosnikov]

lgtm