- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent OpenBLAS from modifying the scheduler affinity #3097
Conversation
Julia already manages the scheduler affinity by itself, which is subsequently undone by OpenBLAS pinning to the first core available. Because of this, external commands spawned from Julia would be restricted to a single core, which in case of multithreaded applications is very suboptimal.
This is what
|
How does this explain multiple calls? This |
I just realized my error. This only calls it once. |
Prevent OpenBLAS from modifying the scheduler affinity
@sebastien-villemot This is something that may affect the debian package too. |
Debian should really be compiling OpenBLAS with Using processor affinity by default (as opposed to manually by Processor affinity is an ugly hack to artificially inflate benchmark numbers; it has no place in production code. Julia shouldn't be using processor affinity by default either. Users who want it can use |
Debian already compiles with |
Git blame reveals the affinity setting code had been added because for some users Julia only used a single core (bug #1070); probably that was caused by OpenBLAS as well? If so, the affinity setting code still in place seems pretty useless. It would also explain why the main Julia process still had an affinity of 1 (@JeffBezanson's comment at the end). |
do not set CPU affinity on Linux (see #3097)
Most of the information is already in the commit:
I ran into this by
spawn()
ing a MATLAB session from Julia; it always seemed to be using a single core while running the very same command from a terminal made use of all available ones. Tracing all system calls, it seemed that the main julia process calledsched_setaffinity
multiple times, once from Julia'sinit.c
(allowing use of all cores) and another time from OpenBLAS'driver/others/init.c
(pinning to the first core). The latter seems to be an OpenBLAS optimization, as per http://comments.gmane.org/gmane.comp.lib.openblas.general/128That thread also mentions the R guys running into the same problem, and proposes using the
NO_AFFINITY
compilation flag to avoid OpenBLAS from touching the affinity. This pull request adds that very build flag to the OpenBLAS Makefile rule, fixing the aforementioned issue with spawning multithreaded programs.