[0.4] Grave overall performance regression on Linux x86 with LLVM ≥ 3.4

[petercolberg]

While packaging Julia 0.4.1 for Debian, we experienced a build failure on x86 caused by a timeout.

I performed a series of test builds in a Debian unstable i386 chroot, building the latest release-0.4 using various LLVM versions from 3.3 to 3.7. Please inspect the complete build and test suite logs with

git clone https://gist.github.com/d9f0ddab9658f57c762e julia-llvm-linux-x86-tests

These are the run-times of the test suite obtained with time:

arch	julia	llvm	elapsed
x86	eb31eef	3.3	5:59.55
x86	eb31eef	3.4	segfault
x86	eb31eef	3.5	segfault
x86	eb31eef	3.6	segfault
x86	eb31eef	3.7	1:04:29

On x86 the Julia test suite is an order of magnitude slower using LLVM 3.7, compared to LLVM 3.3.

(Using LLVM 3.4, 3.5, or 3.6 the test suite aborts with a segmentation fault.)

For comparison I performed the same series of test builds in a Debian unstable amd64 chroot on the same machine. The test suite takes longer with LLVM versions > 3.3, but the increase is still tolerable:

arch	julia	llvm	elapsed
x86_64	eb31eef	3.3	5:22.86
x86_64	eb31eef	3.4	8:41.82
x86_64	eb31eef	3.5	10:01.84
x86_64	eb31eef	3.6	10:11.50
x86_64	eb31eef	3.7	9:56.67

Unfortunately I cannot provide more detail regarding the x86 performance regression, but it appears to affect all tests across the board. This issue is blocking Julia 0.4 from entering Debian. The Julia version currently in Debian unstable, 0.3.12, does not exhibit this issue when using LLVM ≥ 3.4.

I wish I had better news to report so shortly before the 0.4.2 release (#14186).

[petercolberg]

The x86 performance regression using LLVM > 3.3 equally affects master.

[Keno]

Duplicate of #9336. We currently only support LLVM 3.3. Most of the performance regressions have been fixed on 3.8 and we will be switching soon.

[petercolberg]

LLVM 3.8 is better than 3.7, but still four times slower than 3.3:

arch	llvm	elapsed
x86	3.8~svn254193	20:01.38

[nalimilan]

I think many fixes are only on master, not on 0.4.x. For Fedora, I've eventually managed to ship a parallel-installable LLVM 3.3 package only for Julia.

[tkelman]

This is why distributions removing old versions of llvm from the archive is terrible for downstream projects and users. LLVM changes substantially from version to version and it's not a trivial drop in replacement to upgrade - major pieces of functionality get removed that take literally years of work with upstream to get back to performance parity. Keno (and Jameson and others) has done a great job so far but we're not there yet on master, much less release-0.4.

[petercolberg]

LLVM 3.3 has not been updated since two years, and we cannot embed a copy in the julia package. It would mean a major relief for distribution packaging if Julia 0.4 (and master) were fixed to achieve the same performance on x86 as on x86_64 with LLVM > 3.3.

Julia 0.3.12 is working fine with LLVM > 3.3 on Linux x86.

What was changed in Julia 0.4 (and master) to degrade the performance on x86 so significantly?

[petercolberg]

Regarding Julia’s dependency on specific LLVM versions: Have you thought about working with upstream to create long-time stable (LTS) versions that are maintained by downstream users over the span of years instead of months?

[nalimilan]

This discussion has already happened before: #11761 I had the same position as you, but in the end I was allowed to ship LLVM 3.3 in Fedora (which has a very strict policy against bundling).

[tkelman]

It's a good idea, I have no idea how it would be received though. I know they had a "surviving downstream without drowning" birds-of-a-feather session at the recent LLVM dev meeting, @Keno did you go to that?

[StefanKarpinski]

Closing since this is a dup of a very well-trodden issue.