-
-
Notifications
You must be signed in to change notification settings - Fork 250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use and pass jobserver instead of limiting cores directly #1338
Use and pass jobserver instead of limiting cores directly #1338
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I only scanned this and am not very familiar with jobserver, but this looks like a good improvement.
I know it’s WIP but looks like CI wants a “cargo fmt” and I’d want the readme updated before we merge. This is neat. |
Conflicted with #1348 |
I'll fix up the conflicts on this and my other PR later today |
No rush! |
This one's fun: because we now have the jobserver controlling concurrency, rayon is actually hindering us, because it's spawning num-cores threads, and goes from there, when what we want to do is spawn one thread per pgconfig, and let them fight (block) for the job token pool.
7b9f635
to
7a48ed9
Compare
Rebased |
Thanks! |
Currently cargo-pgrx uses both rayon and some manual "max(1, cores / 3)" logic to use multiple cores without overwhelming the system.
This PR ditches these two mechanisms in favour of the jobserver concept, implemented by the jobslot crate. That is an implementation of the
make
jobserver protocol.Basically a jobserver controls access to a limited pool of job slots: to do some compute, you grab a slot, blocking until one becomes available if necessary. A process in the jobserver context passes down the jobserver handle to subprocesses, who, if they're aware (like
make
is), all thus cooperate together.In this PR, cargo-pgrx:
make
pg_config
resolution taskThis naturally and granularly distributes tasks across all available cores, without restricting each
make
to a fixed amount of jobs. Notably, if eg building pg12 and pg13, and pg12 finishes completely, themake
for pg13 may scale up to use the freed-up cores.Before (4 cores, notice how it doesn't even start downloading/untarring pg16 until (off-screen) one of the compiles is fully done):
After (4 cores, notice how there's first 5 downloads and then it restricts itself to 4 compute tasks at a time, interleaving as necessary):