Adding ARM builds

[CRImier]

Hello!

Sorry for the delay. Here are the proposed changes to build.xml file for ARM architecture support, as discussed at #3334. It works on Raspbian Wheezy&Jessie if compiled on Raspbian Wheezy, and on Jessie if compiled with Jessie.
It all works, but there's a need either to natively compile on ARM or cross-compile. I could help with both. Same goes about libastyle and toolchain-avr - they either need to be cross-compiled or compiled on an ARM machine. I did all the compiling on ARM machine, and it worked great.

[ffissore]

Thank you @CRImier
I think you should first provide proper instructions for packing an ARM version of the toolchain by submitting a PR to https://github.com/arduino/toolchain-avr
I'm reviewing arduino/astyle#3 right now
Only AFTER we start building the ARM version on our own and publish the result on our server, we can merge this PR

[ffissore]

Argh, no, I'll review arduino/astyle#3 on September

[NicoHood]

Raspberry now has a jessie build with 4.1 kernel and arm6/7 and gtk3
https://www.raspberrypi.org/blog/raspbian-jessie-is-here/

If the new arduino builder would support all the 4 cores (which i think will be added regarding the opened issue) this could really give us a reasonable environment to develop some code on raspberry.

[ffissore]

I'm sorry but everything that has do with supporting another operating system is going down in my priority list. I don't expect to work on this for the coming months.

[NicoHood]

@CRImier could you please update the PR? Debian Jessie is out now (for the raspi) and I think wheezy support can be dropped if it adds more complexibility?

@ffissore As far as I understood arduino/astyle#3 is required to build this. Why dont you just add it? If it doesnt work, then it wont hurt anyways since arm is just an optional community addition. But since the build system relies on hardcoded download links (as far as I understood) this makes it complicated to patch this?

[NicoHood]

Compiling and uploading now fully works for me on RaspberryPi2 Debian Jessie arm7hf.

I resolved the merging conflicts:
NicoHood@576e882

Cross compiling currently only works if you use the precompiled (by arm) binaries to assemble the ide with those parts. An astyle patch was added, but the toolchain and arduino builder seem to not work yet.

For downloading precompiled binaries skip to step "Setup Java8".

Instructions

You can compile the IDE with the Raspberry Pi itself or with an x64 linux system. On the Raspberry it takes way longer of course and the checksums of the binaries might differ to the x64 build (due to older compiler versions). So the cross compiled version might(!) be smaller and faster.

Download Arduino git

# Download the ARM PR branch. You might want to remove depth for further developing
git clone  --depth 1 -b ARM https://github.com/NicoHood/Arduino.git
cd Arduino/build/

Compile libastyle

First compile the arm version of libastyle 2.05.1:

git clone --depth 1 https://github.com/arduino/astyle.git
cd astyle/

# Subversion is required to download astyle, java8 to compile and execute
sudo apt-get install subversion
sudo apt-get install openjdk-8-jdk openjdk-8-jre

# arm or x64, might differ on your system
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-armhf
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
./setup.bash 

# The arm compiler is only required for cross compiling
sudo apt-get install gcc-arm-linux-gnueabi g++-arm-linux-gnueabi gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf

# For cross compiling change the CXX compiler flag to:
nano ./astyle-code/AStyle/build/gcc/Makefile
CXX = arm-linux-gnueabihf-g++

# Compile astyle
./arm.bash

# Copy the libastyle binary manually into the folder where it should be downloaded
cd ../
mkdir -p ./linux/libastylej-2.05.1/
cp ./astyle/libastylej_arm.so ./linux/libastylej-2.05.1/

Compile the avr-toolchain

I recompiled the avr-gcc and avrdude from:
https://github.com/arduino/toolchain-avr

# Download toolchain
git clone  --depth 1 https://github.com/arduino/toolchain-avr.git
cd toolchain-avr/

# Install some tools to build the toolchain
sudo apt-get install build-essential gperf bison subversion texinfo zip automake flex libusb-dev libusb-1.0-0-dev libtinfo-dev pkg-config

# Build the toolchain. This takes (very) long, even on an x64 system
./arch.arm.build.bash

# Copy binaries and create checksums
cd ../
cp ./toolchain-avr/avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2 ./linux/
cp ./toolchain-avr/avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2 ./linux/
shasum ./linux/avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2 | awk '{ print $1 }' > ./linux/avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2.sha 
shasum ./linux/avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2 | awk '{ print $1 }' > ./linux/avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2.sha 

Download and pack arduino-builder and ctag arm binaries

Current attempt with a hint from this comment arduino/arduino-builder#76 (comment) shows, that Arduino has pre build arm binaries. I do not know how up to date they are. I found this simpler, since I had problems to compile arduino-builder and ctags myself.

mkdir -p arduino-builder-arm
cd arduino-builder-arm
wget http://downloads.arduino.cc/arduino-builder-arm/arduino-builder
mkdir -p tools/5.8-arduino5/
mkdir -p hardware/
wget http://downloads.arduino.cc/arduino-builder-arm/ctags
chmod +x ctags
mv ctags tools/5.8-arduino5/
wget https://raw.githubusercontent.com/arduino/arduino-builder/master/src/arduino.cc/builder/hardware/platform.keys.rewrite.txt
mv platform.keys.rewrite.txt hardware/
wget https://raw.githubusercontent.com/arduino/arduino-builder/master/src/arduino.cc/builder/hardware/platform.txt
mv platform.txt hardware/

# Pack the downloaded files, you might need to change the number in future builds
cd ../
tar -cvjSf ./arduino-builder-arm-1.3.6.tar.bz2 -C ./arduino-builder-arm/ ./
shasum ./arduino-builder-arm-1.3.6.tar.bz2 | awk '{ print $1 }' > ./arduino-builder-arm-1.3.6.tar.bz2.sha

Setup Java8

Before you start the IDE you need to select java 8 as default java version. This cannot be done via ssh, you need to do it via a terminal in the X session on the raspberry pi. If you crosscompile you should do this on your x64 linux as well.

Otherwise you will get this error: java.lang.UnsupportedClassVersionError: processing/app/Base : Unsupported major.minor version 52.0

# Select "java-8-openjdk[...]"
sudo update-alternatives --config javac
java -version

Compile the Arduino IDE

# 1st option for cross compiling 2nd option for direct compiling
# For cross compiling you need to enter a version number. Enter "1.6.7".
sudo apt-get install ant
ant clean build dist -Dplatform=arm
ant clean build

Install the IDE

Precompiled files download:
https://github.com/NicoHood/Arduino-IDE-for-Raspberry/releases/tag/1.6.7

Only required for cross compiling/downloading a pre compilation.
Make sure you read the part "Setup Java8" in any case first!

Download/copy the file arduino-1.6.7-arm.tar.xz and place it on an usb stick in the root directory.
https://github.com/NicoHood/Arduino-IDE-for-Raspberry/releases/tag/1.6.7
https://github.com/NicoHood/Arduino-IDE-for-Raspberry/blob/4316e2f374a76806510293fe8dcd58cc44045613/arduino-1.6.7-arm.tar.xz

# Mount usb stick where the pre compiled IDE is placed and copy it
sudo mount /dev/sda1 /mnt
cp /mnt/arduino-1.6.7-arm.tar.xz .
tar -xf arduino-1.6.7-arm.tar.xz 
cd arduino-1.6.7/

Install the IDE

# You need to start the IDE via terminal on the pi itself, not ssh (both options work).
# If you downloaded the binary or used cross compiling use the first option
./arduino
./linux/work/arduino
ant start

# Create a desktop icon and .ino links if desired (execute on the raspberry):
./install.sh
./linux/work/install.sh

Fix Ctags

The ctags variable does not get unrolled, so you get a compile error. I though I solved this issue, removed the instructions, but its still there. You need to manually edit the path in the hardware/platform.txt and hardcode the path /home/pi/Documents/Arduino/Arduino/build/linux/work/tools-builder/ctags/5.8-arduino5. No reason why.

The error is: fork/exec {runtime.tools.ctags.path}/ctags: no such file or directory

I guess the error happens in this file, in those lines. This means the arduino-builder I downloaded from arduino.cc is probably old. Looking at the arduino builder it sais version 1.3.3 which is 4 days old. Switching back my x64 ide to 1.3.3 does not make the error occur.
https://github.com/arduino/Arduino/blob/master/arduino-core/src/cc/arduino/Compiler.java#L253-L262

Veryfied:
If I execute the arduino builder commands manually hey also fail. Its an arduino builder issue. I tried to recompile arduino builder 1.3.6 (parallel beta version, got it laying around) with golang 1.3.3 (newer version in the readme recommended, used older one! x64 with 1.3.3 go compiler works though) and replaced the binary. Still the same issue. This means the bug needs to be fixed there.
https://gist.github.com/NicoHood/18ee8c4567927bf8857c

It shows that this line returns:
https://github.com/arduino/arduino-builder/blob/master/src/arduino.cc/builder/ctags_runner.go#L68

It was also not caused by a linefeed problem of the wget function. I tested it on x64 too.

Save pre compiled binaries

After a working compilation I saved the binaries of the compiled dependencies. I will attach the binaries to this post, so you can save this time and only recompile the IDE itself which takes a few minutes on the pi and a few seconds via x64 crosscompile.

https://github.com/NicoHood/Arduino-IDE-for-Raspberry/releases/tag/1.6.7
https://github.com/NicoHood/Arduino-IDE-for-Raspberry/blob/4316e2f374a76806510293fe8dcd58cc44045613/arduino_binaries.tar.bz2

mkdir binaries
cp ./arduino-builder-arm-1.3.6.tar.bz2 ./arduino-builder-arm-1.3.6.tar.bz2.sha ./linux/avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2 ./linux/avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2.sha ./linux/avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2 ./linux/avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2.sha ./linux/libastylej-2.05.1/libastylej_arm.so ./binaries/
tar -cjf arduino_binaries.tar.bz2 ./binaries/

Additional notes

The compiling and uploading is slow and laggy. I dont know how the devs could solve it, maybe its just a limitation we have to take. A multicore arduino-builder could help us. I did not test it, since I dont know how to recompile it myself properly (used arduino arm binaries this time).
#4240

Avrdude crashed for me the first time after uploading. This kept the CPU at 100%. This is a known bug on linux, Just want to mention it here, because CPU power matters more. killall avrdude_bin solves the issue. The issue was caused by my hyperion daemon which tried to open the device, the cause was me, not the IDE. I'll leave it as not here anyways to show that those avrdude bugs should be solved.
#3569
#3771

We could solve most issues if arduino.cc provides a precompiled version of avr-gcc, avrdude, astyle and arduino-builder. This is the most annoying part for manually patching the files in (because they cannot be downloaded, so you need to fake them with the correct name). It should be simple to compile those binaries for arm as well. Then we could merge this PR (after the checksums are updated).

@ffissore @facchinm are the arduino-builder arm binaries updated frequently? If you could change the name similar to the x64 builds, that would make things a lot more simple. This requires to move ctags and arduino-builder into a single folder like the x64 builds.

[NicoHood]

Okay I got it running fully. Full instructions above. Precompiled binaries for the dependencies and IDE itself included.

It might look a lot more complicated than it is. Basically you just need to compile and organize 5 binaries first (avrdude, avr-gcc, arduino-builder, ctags, astyle). Then you just hit ant build as usual.

If Arduino.cc could provide those arm binaries it would make the build process as easy as for x64 and others. No special changes, except the commits (for build.xml) I mentioned above are required.

Any chance to get arm support?

[TXCodeDancer]

@NicoHood I thought you'd be interested to know that I was able to follow your detailed instruction and build/install Arduino 1.6.7 on my Raspberry Pi running Ubuntu MATE 15.10. Thank you so much for documenting and sharing your efforts. Now my RPi will actually be useful as a development platform.

[facchinm]

Hi @NicoHood and @TXCodeDancer ,
we decided not to provide an ARM build directly but if you could package up a fully self contained IDE (with Java, tools and all relative paths) we'll be glad to host it on the website (under a new Contributed section).
Nico, I'm sorry for the delay but I completely missed your post, great work by the way! Thank you very much! I'm setting up the Jenkins job for arduino-builder so it will always be in sync with the latest master

[NicoHood]

@TXCodeDancer Do you also have this ctags bug? I opened a bug on arduino builder, that this get fixed first. I guess @ffissore has to fix it first so we can pack up a useful IDE. Edit: hm someone else need to do it now!?

So we have raspbian and ubuntu mate tested. Great!

@facchinm Thanks, that'd be great. Could you please also add the same name schema as for the other platforms? Could you also host libastyle and the avr toolchain like this? This would make things a lot of faster since cross compiling for those tools did not work for me yet. Then it would be easier for other people to follow this instructions. Is it also possible to compile arm builds yourself? I dont know how the Arduino-Builder Builds are generated, but if we can do that with the other libs (asystle etc) too (officially) that'd be great. (please also add the dev branches like the parallel builds).

So @facchinm

Adding Arduino Builder

[get] Error opening connection java.io.IOException: Server returned HTTP response code: 403 for URL: http://downloads.arduino.cc/tools/arduino-builder-arm-1.3.9.tar.bz2

You need to add proper arduino-builder names to the online sources.

Integrating libastyle into the current zip:

From [this binary package}(https://github.com/NicoHood/Arduino-IDE-for-Raspberry/blob/master/arduino_binaries.tar.bz2) we need to include /binaries/libastylej_arm.so to the libastylej-2.05.1 zip and update the shasum.

Adding the avr toolchain

avrdude-6.0.1-arduino5-armhf-pc-linux-gnu.tar.bz2 and avr-gcc-4.8.1-arduino5-armhf-pc-linux-gnu.tar.bz2 should also be officially uploaded.

I added them to my fork as binaries, but you should upload them somewhere properly.
NicoHood@1ff0c4e

[ffissore]

@NicoHood please see this https://groups.google.com/a/arduino.cc/forum/#!topic/developers/YJLX6AZHem4

[NicoHood]

@ffissore thanks for this note, I would have missed it. That is really sad, who else could do such a great job now, if not you?

[NicoHood]

class HelloWorld
{ 
        public static void main(String args[])
        {
           System.out.println("Hello World!");
           System.out.println(System.getProperty("os.name"));
           System.out.println(System.getProperty("os.arch"));
        }
}

nano HelloWorld.java
javac HelloWorld.java 
java HelloWorld 

Hello World!
Linux
arm

Tested on the latest raspbian image. Is there anything else I could/should do?

Thx for having a look into that issue! I hope we can easily fix this.

[cmaglie]

Is there anything else I could/should do?

I forget the gcc target, please run a gcc -v 2>&1 | grep Target on my system is:

$ gcc -v 2>&1 | grep Target
Target: x86_64-linux-gnu

With those info I'm going to update this file:

https://github.com/arduino/Arduino/blob/master/arduino-core/src/cc/arduino/contributions/packages/HostDependentDownloadableContribution.java

This should be enough to provide downloads for arm when available.

[ShorTie8]

Been working on a script that lives in the Arduino world to build the IDE.
When done, it should work on any system not just ARM.
It even checks for go, and installs if needed.
After the 1st run, if nothing has changed you should just get a bunch of
"Your branch is up-to-date with 'origin/master'"
https://www.dropbox.com/s/eu69r7xcxlnjus3/Arduino_IDE_builder.bash?dl=0

and yes @NicoHood I made some stuff optional for you
It's almost done, but between the Chickens and Taxes, it may not get done till the weekend.

@CRImier "gcc -v 2>&1 | grep Target" on my pi gives "arm-linux-gnueabihf"

@CRImier @facchinm Don't know if it would work or have any ill affects, but have you ever thought of file branding when configuring with --with-pkgversion='Arduino $PKGVERSION' ??
Then you could add the logging of PKGVERSION in error reports so you know if it is a Arduino package/build ??

Have A Great Day

[ShorTie8]

can't do it in CONFARGS, but on the config, like ../gcc-4.8.1/configure --with-pkgversion="Arduino" $CONFARGS

to give

./avr-c++ --version
avr-c++ (Arduino ) 4.8.1
./g++ --version
g++ (Arduino ) 4.8.1

[NicoHood]

arch linux (pi2) gives:

Target: armv7l-unknown-linux-gnueabihf

I will check raspbian later and edit here. I think we should focus on raspbian and ubuntu mate, wanted to add arch anyways.

Also it might depend if you boot the image on a pi1 or pi2. I'd also vote to focus on pi2, as its faster and the ide itself (that i compiled before) is already quite slow.

Rapbian on pi2 gives:

arm-linux-gnueabihf

[Testato]

I'm on a BBB (BeagleBone), is it useful for testing ?

[TXCodeDancer]

@cmaglie Thanks for taking on this effort!

Here are my results with Ubuntu MATE on a pi2 (following Nico's code example):
$ java HelloWorld
Hello World!
Linux
arm

$ gcc -v 2>&1 | grep Target
Target: arm-linux-gnueabihf

$ uname -a
Linux xxxx-RPi-00 4.1.15-v7+ #830 SMP Tue Dec 15 17:02:45 GMT 2015 armv7l armv7l armv7l GNU/Linux

Please let me know if I can provide any more useful information.

[NicoHood]

@Testato, please. If its the same name, we can use the build on BBB too (and you can test it if you like to).

Looking forward to see arm builds finally =)

[Testato]

Board: BBB (BeagleBoneBlack)
OS: Debian GNU/Linux 7.9 (wheezy)

~ # uname -a
Linux beaglebone 3.8.13-bone70 #1 SMP Fri Jan 23 02:15:42 UTC 2015 armv7l GNU/Linux

~ # java HelloWorld
Hello World!
Linux
arm

~ # gcc -v 2>&1 | grep Target
Target: arm-linux-gnueabihf

[NicoHood]

Raspbian on a Pi Zero (the old one on arm6) gives the same results. So except arch-linux we could really build an arm build that works on all OS. I mean the current one from me should still work, with this workaround. But the builder should be patched and the binaries should be uploaded.

[ShorTie8]

@NicoHood on the ctags path issue, maybe try putting it in
Arduino/build/linux/work/tools-builder/ctags/5.8-arduino5
Also, and see if that fixes the path issue
it may not really need to be in
arduino-builder-arm/tools/5.8-arduino5/
it maybe only be needed there if your gonna test run arduino-builder on cmd line

[NicoHood]

I just looked into my files and it is already there: RaspiIDE/Arduino/build/linux/work/tools-builder/ctags/5.8-arduino5/

Edit: it might be for x64, i do not remember what i did those month ago. Have to check it again.

Edit2: they seem to be copied at runtime and they are arm builds. The builder just cannot resolve the path for some reason.

RaspiIDE/Arduino/build$ sha1sum ./arduino-builder-arm/tools/5.8-arduino5/ctags 
77aeb15afc2d4b994e4fad6115d27e0b80b16b0b  ./arduino-builder-arm/tools/5.8-arduino5/ctags
RaspiIDE/Arduino/build$ sha1sum ./linux/work/tools-builder/5.8-arduino5/ctags 
77aeb15afc2d4b994e4fad6115d27e0b80b16b0b  ./linux/work/tools-builder/5.8-arduino5/ctags

[ShorTie8]

Got me, your just not putting it into the right place I do believe
Here is my script
wget https://raw.githubusercontent.com/ShorTie8/Arduino_IDE/master/Arduino_IDE_builder.bash

It seems to work fine on Raspberry and Odroid, other Linux's should work too...
It brand the files so you know there mine, lol.
Or Our's I guess .. :/~

Have Fun

[NicoHood]

I still do not understand what is wrong then. I will run your script and try out then.

I already added some comments to you commit:
ShorTie8/Arduino_IDE@d181b65

[NicoHood]

For some reason your script exists after building go.
The last print on the bash is:

? unsafe [no test files]

Not sure why this happens. I used sudo su first and then run the script.

Afterwards it fails at autoconf. It says M4 is required. Line 206 seems to fail.

sudo apt-get install m4 -y

#other requirements i have not installed yet (will try to run without for now)
sudo apt-get install build-essential gperf bison subversion texinfo zip automake flex libusb-dev libusb-1.0-0-dev libtinfo-dev pkg-config

Also ant and java might be required. You should add this information somewhere with and apt-get line to pull in all of those at the top. If installed, just skip them.

[ShorTie8]

I run it as root. I live on the wild side and always play as root.
Tried to warn that is the only way it seemed to work for me .. :/~

[ShorTie8]

Here is a better write-up of the script maybe, http://forum.arduino.cc/index.php?topic=372830.0

[NicoHood]

The toolchain script fails to download this file:
http://www.multiprecision.org/mpc/download/mpc-0.9.tar.gz

arduino/toolchain-avr#30

[NicoHood]

I works! I will search for the error I made. Now it should be easier to provide official arduino arm builds.

I will update this post after i find out new things. I guess it was an old go version that causes the builder to compile wrong.

I updated this issue:
arduino/arduino-builder#79

So I got the old 1.6.7 working. I pulled from upstream now and the IDE crashes because no liblistserials is available. I have no idea where to get this from, but I will find out. We should be able to recompile it for arm as well.

For 1.6.8 we have to also compile a new library. I've compiled it on arm with those patches myself:
arduino/listSerialPortsC#4

[ShorTie8]

Added listSerialPortsC and fixed up some stuff
Think i got the naming and stuff right

[NicoHood]

You also noticed that I updated my ARM branch to the latest arduino? Do you also get this error when you try to start the IDE?

[ShorTie8]

no I haven't noticed it, but maybe may update github thing is not working right.
Are you pushing that a head of the current release ??
your gonna mess peeps up, as I did post that script around so others could use it.
I'm still around on irc

Edit: Ya Update_git wasn't working right
But it still seems to be building 1.6.7, you do something to get it to build 1.6.8 ??

[facchinm]

Closing this one because all the discussion moved to #4457