Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent OpenBLAS from modifying the scheduler affinity #3097

Merged
merged 1 commit into from
May 14, 2013
Merged

Prevent OpenBLAS from modifying the scheduler affinity #3097

merged 1 commit into from
May 14, 2013

Conversation

maleadt
Copy link
Member

@maleadt maleadt commented May 14, 2013

Most of the information is already in the commit:

Julia already manages the scheduler affinity by itself, which is subsequently
undone by OpenBLAS pinning to the first core available. Because of this, external
commands spawned from Julia would be restricted to a single core, which in case of
multithreaded applications is very suboptimal.

I ran into this by spawn()ing a MATLAB session from Julia; it always seemed to be using a single core while running the very same command from a terminal made use of all available ones. Tracing all system calls, it seemed that the main julia process called sched_setaffinity multiple times, once from Julia's init.c (allowing use of all cores) and another time from OpenBLAS' driver/others/init.c (pinning to the first core). The latter seems to be an OpenBLAS optimization, as per http://comments.gmane.org/gmane.comp.lib.openblas.general/128

That thread also mentions the R guys running into the same problem, and proposes using the NO_AFFINITY compilation flag to avoid OpenBLAS from touching the affinity. This pull request adds that very build flag to the OpenBLAS Makefile rule, fixing the aforementioned issue with spawning multithreaded programs.

Julia already manages the scheduler affinity by itself, which is subsequently
undone by OpenBLAS pinning to the first core available. Because of this, external
commands spawned from Julia would be restricted to a single core, which in case of
multithreaded applications is very suboptimal.
@ViralBShah
Copy link
Member

This is what src/init.c has, which explains the multiple calls to sched_setaffinity you see:

#if defined(__linux__)
    int ncores = jl_cpu_cores();
    if (ncores > 1) {
        cpu_set_t cpumask;
        CPU_ZERO(&cpumask);
        for(int i=0; i < ncores; i++) {
            CPU_SET(i, &cpumask);
        }
        sched_setaffinity(0, sizeof(cpu_set_t), &cpumask);
    }
#endif

@maleadt
Copy link
Member Author

maleadt commented May 14, 2013

How does this explain multiple calls? This sched_setaffinity makes for the first, correct call, setting the mask so Julia can use all processors. A second call, way later, restricts this again to a single core. I traced that second call back to the OpenBLAS library (and verified by commenting it out and recompiling Julia & OpenBLAS and stracing again).

@ViralBShah
Copy link
Member

I just realized my error. This only calls it once.

ViralBShah added a commit that referenced this pull request May 14, 2013
Prevent OpenBLAS from modifying the scheduler affinity
@ViralBShah ViralBShah merged commit 1a10c54 into JuliaLang:master May 14, 2013
@ViralBShah
Copy link
Member

@sebastien-villemot This is something that may affect the debian package too.

@stevengj
Copy link
Member

Debian should really be compiling OpenBLAS with NO_AFFINITY=1.

Using processor affinity by default (as opposed to manually by taskset invocation) is insanity on general-purpose systems. Once more than one program tries to do it, you are screwed, and on a general-purpose system (as opposed to a dedicated single-user compute node) you have to assume that other programs are running in general.

Processor affinity is an ugly hack to artificially inflate benchmark numbers; it has no place in production code.

Julia shouldn't be using processor affinity by default either. Users who want it can use taskset.

@staticfloat
Copy link
Member

Debian already compiles with NO_AFFINITY=1, so both the packages Sebastien merged into debian and my PPA are good to go.

@maleadt
Copy link
Member Author

maleadt commented May 14, 2013

Git blame reveals the affinity setting code had been added because for some users Julia only used a single core (bug #1070); probably that was caused by OpenBLAS as well? If so, the affinity setting code still in place seems pretty useless. It would also explain why the main Julia process still had an affinity of 1 (@JeffBezanson's comment at the end).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants