Port to ARM

[ViralBShah]

This is the build log for the julia build on ARM:

https://buildd.debian.org/status/fetch.php?pkg=julia&arch=armhf&ver=0.1.2%2Bdfsg-3&stamp=1368675598

The julia system image build fails with:
LLVM ERROR: Not supported instr: BMOVPCRX_CALL %R3, , %LR<imp-def,dead>, %SP, %R0<imp-use,kill>, %R1<imp-use,kill>, %R2<imp-use,kill>, %SP, %R0; dbg:no file:0

[JeffBezanson]

That is probably an LLVM issue, but to make sure we should run with LLVM assertions enabled, and with verifyFunction in codegen.cpp enabled.

[ViralBShah]

#3128 will help make this job and any future ports easier too.

[ihnorton]

The runtime builds successfully on a Cortex-A15 (Chromebook) with a few small Makefile changes. Unfortunately, type inference seems to go haywire somewhere in sysimg.jl, and eventually crashes.

[Keno]

That MCJIT issue may not be ARM specific. I've been hunting some bugs in the MCJIT code which could cause that.

[ihnorton]

What kind of bugs? It seems like this is a loop in inference, leading to a stack overflow.

I should also mention that I commented out constants.jl in sysimg.jl because some Big... asserts were failing (due to ARM differences?).

[tkelman]

For kicks I just tried building Julia master on a Linaro-Ubuntu ARM using system everything including LLVM 3.4 (Make.user contents and same patch as ihnorton here https://gist.github.com/anonymous/9177166), and had these compilation errors in codegen.cpp: https://gist.github.com/anonymous/9176765

Hope this info is possibly useful in some way.

[dmbates]

@tkelman I think this problem is related to using llvm-3.4 with the code in the master branch of Julia, which switches to using MCJIT when 3.4 is detected. oI have encountered a similar problem on an amd_64 system. In src/codegen.cpp you will see

#if defined(LLVM_VERSION_MAJOR) && LLVM_VERSION_MAJOR == 3 && LLVM_VERSION_MINOR >= 4
#define LLVM34 1
#define USE_MCJIT 1
#include "llvm/ExecutionEngine/MCJIT.h"
#include "llvm/ExecutionEngine/SectionMemoryManager.h"
#include "llvm/ADT/DenseMapInfo.h"
#endif

[ihnorton]

Yes - either use 3.3, or the LLVM 3.5 svn head (that is what I compiled with).

[ihnorton]

@tkelman the base system compiled fine for me on an A15 chromebook, the problem was in bootstrap (but some things have been fixed since the last time I tried).

[Keno]

Sorry, it's sometimes very difficult to know what APIs are from what version of LLVM, I'll go through and make sure everything works on 3.4, but for now 3.5svn should work fine.

[tkelman]

If the likelihood of success is going to be higher with 3.5, then don't worry too much about it. From what I've pieced together it sounds like legacy JIT is expected to be broken on ARM, and the new JIT is only turned on with LLVM 3.4+, so that's why I tried the combination I did.

Ooh, LLVM has apt repos for 3.5. Aww, not for ARM :(

[tkelman]

Slightly different set of errors when I set LLVM_VER = svn in deps/Versions.make, and comment out anything LLVM-related from Make.user: https://gist.github.com/9226335
The errors this time are all related to raw_fd_ostream. This was built at Julia commit 0dbbaf7

[ihnorton]

Here's the backtrace I get on the current master during bootstrap:

https://gist.github.com/ihnorton/1b52f4f034005306f167

I'm not sure why the arguments are not printed, but apparently backtrace is harder on ARM. In the first one I've put jl_(ex) just below the expression check in toplevel. There's no immediately obvious reason why a variable would be undefined in that expression.

[Keno]

Out stack unwinder probably doesn't work on ARM

[ihnorton]

[Keno]

🍰

[aviks]

👏

[tkelman]

Whoa, nice work! Details, details, where's the PR? Also: 1398635418 days?

[ihnorton]

Thanks, but this is less impressive than it looks! I basically just turned a few broken things off: rec_backtrace, cpuid, and some bignum asserts in constants.jl. But that appears to be it, at least for build-to-bootstrap and basic functionality. It's a very tractable list of issues, which is a good sign.

[timholy]

Really fantastic! This opens quite a new world of possibilities.

[JeffBezanson]

It's a shop!

[ihnorton]

Patch set:

• Make.user: https://gist.github.com/ihnorton/11357352 (ignore the JCXXFLAGS - oldish gcc)
• Julia changes so far: https://github.com/ihnorton/julia/compare/arm-make2

@JeffBezanson lol :) you can have an account if you want to check! (chromebook wireless is kind of flaky though, so I have to go reset it manually every 1-2 hours; need to order usb<->ethernet)

[StefanKarpinski]

Bravo! This is really exciting.

[gregzuro]

Hope not.. Yay!
On Apr 27, 2014 3:59 PM, "Jeff Bezanson" [email protected] wrote:

It's a shop!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3134#issuecomment-41512275
.

[jakebolewski]

👍

[ViralBShah]

This is fabulous, and will open a new set of possibilities for Julia!

[ViralBShah]

It would be great to release 0.3 with initial arm support!

[nalimilan]

If you manage to do it, I'll add ARM support to the coming Fedora package.

[ViralBShah]

If we can make master build on arm, we can make sure that Debian builds with arm support for the julia 0.3 release. This will get reflected in the next Ubuntu release then.

Cc: @sebastien-villemot

[2jcjohnson]

Attempted install on Rasberry Pi under rasberrian. Make ran make for 30 minutes before failure. Followed /ihnorton/Julia/compare/arm-make2 instructions. Very excited about the progress. Wish I could make it to Julia Con

[2jcjohnson]

screenshot of progress

[ViralBShah]

Is this using llvm 3.4?

[tkelman]

The problem there was openlibm, you'll need USE_SYSTEM_LIBM=1 - check the Make.user posted above

[ihnorton]

Yes, that will help. I have not tested with LLVM3.4, only with the SVN trunk (as also specified in Make.user)

[2jcjohnson]

I'm a new at this but Ill keep testing while you guys supply the direction.

1. I have exactly the Make.user file specified by ihnorton above.

JCXXFLAGS=-std=c++11

FORCE_ARMV7=1
LLVM_ASSERTIONS=1

override USE_SUSTEM_LIBM=1
override OPENBLAS_VER = v0.2.9.rcl
override LLVM_VER=svn

2. how do I check which version of LLVM was used? I took the main Julia
   source download off of the server on May 10 as 0.3.0-prerelease. I presume
   it - or the override command here selected the LLVM version to grab. (told
   you I was new at this)

3. I did not specify any options with make. (dont know if that makes a
   difference) should I be using -d or -k ? would using those help you guys
   understand the failure points? If I specify -d is it automatically sending
   to a file or do I specify a file / pipe it to a file. I did not see an
   option for capturing debug info directly to a file.
   Thanks to all for your assistance.

On Sun, May 18, 2014 at 2:38 AM, Isaiah [email protected] wrote:

Yes, that will help. I have not tested with LLVM3.4, only with the SVN
trunk (as also specified in Make.user)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3134#issuecomment-43429791
.

[tkelman]

override USE_SUSTEM_LIBM=1
override OPENBLAS_VER = v0.2.9.rcl

Are these typos? It should be USE_SYSTEM_LIBM and v0.2.9.rc1

There should be a directory inside deps called llvm-__ where __ will tell you the version number.

Don't think you should need any make options just yet.

[2jcjohnson]

Much further along: I had to install gfortran, update to g++-4.7, (and
update to gcc-4.7?). I dont know if the gcc update was necessary.
I cant tell if this is an llvm question or makefile question at this point.
Any suggestions?

llvm[4]: Compiling DiagnosticPrinter.cpp for Release+Asserts build
llvm[4]: Compiling Dominators.cpp for Release+Asserts build
llvm[4]: Compiling Function.cpp for Release+Asserts build
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See file:///usr/share/doc/gcc-4.7/README.Bugs for instructions.
/bin/rm: cannot remove
`/home/pi/julia/deps/llvm-svn/build_Release+Asserts/lib/IR/Release+Asserts/Function.d.tmp':
No such file or directory
make[4]: ***
[/home/pi/julia/deps/llvm-svn/build_Release+Asserts/lib/IR/Release+Asserts/Function.o]
Error 1
make[3]: *** [all] Error 1
make[2]: ***
[llvm-svn/build_Release+Asserts/Release+Asserts/lib/libLLVMJIT.a] Error 2
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2
pi@raspberrypi ~/julia $

PS: yes those were typographical errors in my Make.user file

On Sun, May 18, 2014 at 8:20 PM, Tony Kelman [email protected]:

override USE_SUSTEM_LIBM=1
override OPENBLAS_VER = v0.2.9.rcl

Are these typos? It should be USE_SYSTEM_LIBM and v0.2.9.rc1

There should be a directory inside deps called llvm-__ where __ will tell
you the version number.

Don't think you should need any make options just yet.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3134#issuecomment-43450427
.

[Keno]

Seems like the compiler ran out of RAM while linking. Are there prepackaged llvm packages for your distribution?

[2jcjohnson]

On the raspberry pi I modified the memory split between the CPU and GPU. Giving all 512 MB to the CPU. The compile went much further through the llvm compile ~30hrs. But failed in a similar manner later in the process.

I checked the llvm repository and the raspberry pi repository to see if they already have an arm distribution that fits. Unfortunately they do not (the pi is a System on a chip using ARMv6). The most recent llvm release included a compiled ARMv7 package (I assume this is not compatible with an ARMv6 processor)

I am not familiar with cross compiling and am not sure I am up for this challenge. Am I stuck? Is my next step begging for an llvm compile to be generated compatible with the raspberry pi?

I'd love to test out Julia on this little platform.

[ViralBShah]

@ihnorton Do you think we should have a README.arm where we just document the steps to build on ARM, and provide a Make.user.ARM?

[2jcjohnson]

http://llvm.org/docs/HowToBuildOnARM.html

This page gives instructions for compiling llvm on armv6 and armv7
Including how to reduce the memory footprint.

Can someone help me figure out where to place these instructions in the make file set for Julia
I think it would be valuable info to review what is there and include it in the ARM.readme
Again I am trying to build on a rasberry pi. ARMv6 with 512mb ram

I don't know where to specify the
--enable-optimized
or what they mean by
please use gold rather then GNU ld.

[ihnorton]

Can someone help me figure out where to place these instructions in the make file set for Julia

see https://github.com/JuliaLang/julia/blob/master/deps/Makefile#L379

please use gold rather then GNU ld.

part of binutils. I haven't used it, but see here

@2jcjohnson cross-compiling is going to be much less frustrating.

[ihnorton]

@ViralBShah sure, will do. Will need to be labeled as "experimental" (and make clear that codegen is quite slow, even on a "high-end" Cortex A15).

[ViralBShah]

@ihnorton Could you create a PR with your ARM changes? I just got an arm Chromebook, and am essentially running into the same issues.

[sjkelly]

+1 I have an application which would be really nice to run on a BBB.

[ViralBShah]

Closing this to avoid further discussion here. Please follow progress and discuss in #7662.

[ViralBShah]

@sjkelly I see you got it running on the BBB.

@ihnorton Do you think that the CpuInfo stuff you did will throw off the LLVM codegen for armv6?

[ihnorton]

That is only used to make sure sys.so is not run on an incompatible cpu,
and it is not used by LLVM. So it doesn't matter as long as the build is
not transferred to another system.

On Sun, Sep 14, 2014 at 1:08 PM, Viral B. Shah [email protected]
wrote:

@sjkelly https://github.com/sjkelly I see you got it running on the BBB.

@ihnorton https://github.com/ihnorton Do you think that the CpuInfo
stuff you did will throw off the LLVM codegen for armv6?

—
Reply to this email directly or view it on GitHub
#3134 (comment).