remoteActionBuildingSemaphore slows down highly parallel remote builds

[brentleyjones]

Description of the bug:

Highly parallel RBE builds are slower than they need to be because of remoteActionBuildingSemaphore.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Have a build that can do 300+ actions at once, use --jobs=300, and a host machine with 24 or fewer threads. Build clean once with Bazel 7.0.0. Build clean once with remoteActionBuildingSemaphore usage commented out. Compare and see that the second build is faster.

Which operating system are you running Bazel on?

Ubuntu 22.04

What is the output of bazel info release?

development version

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

Built the release-7.0.0 branch and modified it per above.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

#17120 / 4069a87 added the semaphore

Any other information, logs, or outputs that you want to share?

No response

[meisterT]

cc @coeuvre @EdSchouten

@brentleyjones for your build, did you see any impact on peak memory from your change?

[brentleyjones]

@meisterT Yes, it's higher, but not massively so. Similar to @werkt's comment (#19924 (comment)), it depends on the build at hand. To quote him:

remoteActionBuildingSemaphore is being used to regulate CPU and RAM pressure - merkle or generalized RAM estimation should be used to regulate the latter, distinct from the former, in low overhead merkle tree situations (bazel-stress uses minimal inputs and only measures action throughput, so it is a pathological representation of the lowest possible memory overhead).

[meisterT]

Compare and see that the second build is faster.

How much faster is this for you when you comment out the code?

[brentleyjones]

The remote portion of the build is 40% faster, so quite a lot.

[fmeum]

@coeuvre Is there a plan for improving the default situation? If the Merkle tree cache is the main motivation for this limit, I could prioritize working on that.

[coeuvre]

I don't have the capacity to improve it at the moment, but I am happy to review PRs!