fix(prover): speed up LWG and NWG

[0xVolosnikov]

What ❔

• Loading of proofs for each recursive circuit is now done asynchronously
• Each recursive circuit is now processed in a blocking thread, which speeds up serialization and other CPU-sensitive processing.

Locally, with additional artificial network delays (500 ms), a x5 speed up is observed

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• Code has been formatted via zk fmt and zk lint.

[EmilLuta]

Async job loading looks like the right thing. I fail to understand how moving to blocking thread helps with serialization/deserialization. Did you run tests both ways (sync/async)?

[0xVolosnikov]

Did you run tests both ways (sync/async)?

Yes, the difference is noticeable. In different cases, the strategy for using threads by tokio may be different depending on blocking/non-blocking context.

tldr: afaik, async is single-threaded, but blocking tasks aren’t
https://docs.rs/tokio/latest/tokio/task/fn.spawn_blocking.html

[EmilLuta]

tldr: afaik, async is single-threaded, but blocking tasks aren’t

So, my understanding is that when you start a blocking process you get a thread only for that blocking process (and more might be added, if there're more blocking asks or be blocked until the runtime can get one). I totally fail to understand what is going on behind the hood that could make it faster. @popzxc -- maybe you understand?

[0xVolosnikov]

The spawned task may execute on the current thread, or it may be sent to a different thread to be executed. The specifics depend on the current Runtime configuration.

https://docs.rs/tokio/latest/tokio/task/fn.spawn.html

[popzxc]

tldr: afaik, async is single-threaded, but blocking tasks aren’t

tokio::spawn will run your task on one of the runtime threads (tokio uses the same threadpool to schedule and run stuff). tokio::spawn_blocking will run your task on a dedicated blocking threadpool, which is much larger.

handle.block_on(..) will block the thread you run on, but it won't block the runtime itself, e.g. runtime will only be utilized for async stuff.

[EmilLuta]

Discussed in DMs. I had the misconception that the runtime scheduler runs in its own thread. It runs as a regular task, therefore all blocking tasks can slow it down. We have some ser/deser in here (maybe 1ms each) and also create_leaf_witness which I haven't measured. Disregard previous comments.