Adding ARM builds (2nd round)

[NicoHood]

Originally based on #3549

See the link above for additional recompile instructions of the different components and also the work of @ShorTie8 https://github.com/ShorTie8/Arduino_IDE

Take me to Download!

Just see the latest readme here which should point to the current downloads:
https://github.com/NicoHood/Arduino

Attention

This PR does include precompiled binaries! You should remove those before you merge this PR. I could do this on a different branch, if you are willing to merge this PR. But for now we keep this PR as visual reference of the changes I made to far to build.xml.

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

Build instructions

This PR can be build with an arm device or any x64 linux:

#x64
ant clean build dist -Dplatform=arm

#arm
ant clean build

To create new checksums use this:

shasum ./arduino-builder-arm-1.3.6.tar.bz2 | awk '{ print $1 }' > ./arduino-builder-arm-1.3.6.tar.bz2.sha

Compiling on arm with the binaries in place takes about 5~10min on a raspi2 which is acceptable since you do this once a day as dev.

Please note that crosscompiling the IDE is possible, since the binaries are precompiled on arm. The arm binaries are not yet all crosscompile able. Also it is not worth to add this option, since we only need to do that once (except for arduino-builder).

Bugs

Edit: Fixed! See below

The current build will not start! This happens with crosscompile and normal compilation.

arduino-1.6.8 $ ./arduino
Picked up JAVA_TOOL_OPTIONS: 
Tool name : avrdude 6.0.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
Tool name : avr-gcc 4.8.1-arduino5
     sys : i386-apple-darwin11 
     sys : x86_64-linux-gnu 
     sys : i686-linux-gnu 
     sys : i686-mingw32 
 seems to have no downloadable contributions for your operating system, but it is installed in
/home/pi/hackallthethings/arduino-1.6.8/hardware/tools/avr
java.lang.NullPointerException
    at processing.app.BaseNoGui.createToolPreferences(BaseNoGui.java:851)
    at processing.app.BaseNoGui.initPackages(BaseNoGui.java:609)
    at processing.app.Base.<init>(Base.java:295)
    at processing.app.Base.guardedMain(Base.java:222)
    at processing.app.Base.main(Base.java:136)

This bug happens here:
https://github.com/NicoHood/Arduino/blob/master/arduino-core/src/cc/arduino/contributions/packages/ContributionsIndexer.java#L273

And might or might not be related to this:
6398cde

Which looks to me like it is a bug from the commit above:

Arduino/arduino-core/src/cc/arduino/contributions/packages/ContributedTool.java

Line 45 in 6398cde

public DownloadableContribution getDownloadableContribution(Platform platform) {

This commit caused the error, I reverted the commit:
6398cde

cc @cmaglie @facchinm @CRImier @imihalkovic @TXCodeDancer @Testato

[ShorTie8]

Ya, you broke it when you updated to 1.6.8, lol.
Maybe try syncing your git ??

[NicoHood]

It has to do something with this commit:
6398cde

I think we can easily fix this one. I just dont know how, but @cmaglie knows how this works. Then I get update the git again and it should compile then without any additional steps.

[ShorTie8]

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

They even make a hidden directory to hide the package_index.json and sig file

 locate package_index.json
/home/pi/.arduino15/package_index.json
/home/pi/.arduino15/package_index.json.sig
/usr/local/src/Arduino_IDE/package_index.json.diff

ps: @NicoHood your github could still use updating, there are no 'git tags' after 1.6.5-r5 .. :/~

[Testato]

+1
Il 26/gen/2016 08:52, "ShorTie8" [email protected] ha scritto:

Yup, If you undo that patch, all is well.
I tried adding some stuff into package_index.json, but then it fails the
sig check.
Which is some sort of funky file, so I can't make a new sig file.
It seems like there new motto is, DEATH to open source !!
ffissore was/is a big advocate of open source, the way the IDE is getting
locked down with all those stupid exceptions and stuff, might be a cause
for his moving on, but that is just MHO !!!!

—
Reply to this email directly or view it on GitHub
#4457 (comment).

[NicoHood]

IDE is now fixed and a download is provided. Please test and report if it works on your device and which device you used.

The current master is broken, due to the lack of a new arduino-builder arm version. But I tagged the working version, so you can always go back to this and recompile it yourself.
https://github.com/NicoHood/Arduino

[cmaglie]

The patch on the HostDownloadable class has been made to allow updates of the cores from the Boards Manager. As you have noticed the packages_index.json still misses the tools section with the arm flavour of the tools, you can add it by yourself and rename package_index.json to test_package_index.json for testing (adding the test_ prefix will skip the signature check).

Without this patch the next time you try to update the AVR core, or install a non-AVR core, through the Board Manager the IDE will download the tools for x86 and consequently fail: that's the reason why I added the patch, not to lock someone out.

ffissore was/is a big advocate of open source, the way the IDE is getting locked down with all those stupid exceptions and stuff, might be a cause for his moving on, but that is just MHO !!!!

@ShorTie8
I can understand your frustration but, as you can see from the git history, the signature check has been added by ffissore himself, a long time ago, and for a very good reason. I can assure you that this is not the cause of his moving and I would like to ask you (and anyone else) to not make FUD out of personal choices that someone may have made for a number of different reasons that are out of your/our knowledge.

Thanks.

[cmaglie]

Adding precompiled binaries can be avoided if the arduino company uploads those files on their folders and also recompile new arm builds, such as the arduino-builder arm build (which needs a fix for now).

@NicoHood
We are going to setup an ARM builder on our CI system, we'll provide an ARM build shortly (at least for the arduino-builder it should be easy).

[NicoHood]

Oh I see now. Thx for clarifying @cmaglie

So what about arduino hosting the binaries? Then I can add the download link instead and remove the binaries from the git. This would make things simpler.

If the arm arduino-builder (with the path fix) is setup and the other files are hosted it should be easy to maintain this. It would be almost ready to merge. We just have to keep track of the changes and apply those additions for arm too. But most of the work is done already.

[cmaglie]

So what about arduino hosting the binaries?

Sure, but we can't host untrusted binaries, we need to reproduce the build here. Let's start with arduino-builder, I'll give it a spin tomorrow, now I should go AFK for a while.

[NicoHood]

Sure, thats true.
You can use the steps I used above (except for arduino builder, you know this one better). Then the checksum should be the same. Keep in mind that most arm stuff has to be compile on an arm device. RaspberryPi2 works perfect for me and in reasonable time (toolchain takes long and has a missing download mirror atm for all systems).

[ShorTie8]

fyi: My script always builds the latest git versions, but may tar them up as a different version so the build.xml is satisfied. And I don't hide what it is doing, I'm nice and it tells ya in the push to be.

Thankz @NicoHood for updating and fixing github

ps @cmaglie it's my computer not yours, why do write files that I can not normally see ??
Like your trying to hide something that peeps can't see it !!!!!!!!
Thought this was open source and not find me if you can !!!!!
And if you look at what ffissore said a couple of months ago, it sure seems to me that some thing turn his stomach quickly to make him move on, but that is just MF`nHO.

So how do you make the sig file ???

[facchinm]

@ShorTie8, to make the sig file you only need echo shasum $filename > $filename.sig.
But, please, stop acting like we hid something and stop talking about people you don't know. Everything in the build system is crystal clear, the utility to generate the signature is open source and the code to check if the downloaded file is valid is open source too.
If you are worried about your security you only need to read the code 😉

[ShorTie8]

@facchinm oh really ??

pi Arduino_IDE # cat /home/pi/.arduino15/package_index.json.sig
▒V▒Y▒
        $▒k)B▒Yc▒H2▒▒8▒bEv▒=▒s▒▒W▒▒m▒zA▒y▒J*▒K▒▒▒:▒Ou,kT▒▒(LP▒▒▒K00i ▒▒Tn[▒?▒▒)▒[u▒}▒▒▒:]▒▒▒▒▒G▒k▒K▒t▒▒
▒▒      c▒#▒ǯI
              ▒Dhv%▒e▒▒*ߵu▒2 ▒▒/▒▒H▒a▒▒▒▒
㰑0P▒▒3▒▒▒?▒
            ▒P2▒}+▒▒▒▒.j▒▒Y\▒P▒▒▒)▒ů▒O▒▒:&▒▒z▒▒▒i▒▒▒▒o▒\▒▒l▒▒
@▒+6▒▒M▒▒
         ▒0▒▒j▒▒I▒▒▒▒▒▒w▒▒▒6▒(▒▒V▒el▒▒[▒▒Rn▒▒▒▒▒tn'▒▒▒(z▒▒K0▒▒▒▒▒▒XՃB▒Z▒M▒Ta▒▒P˿m▒)▒▒▒˥jr▒PMj▒▒T▒_}▒▒▒▒c(▒d▒J▒▒6▒▒g4▒(▒7ܢ▒▒▒▒N▒.6▒t٘B▒▒▒2▒▒▒▒▒-QfV▒R▒▒}d▒▒▒X▒M▒▒▒;▒▒▒▒▒H▒4q▒C▒▒Z▒▒▒▒▒w   &▒▒$Ǐq▒Q▒▒▒▒▒▒s▒@▒k▒r ▒▒▒▒t▒
▒/▒▒֎▒▒h▒▒~ڀX▒\▒r▒PuTTYpi Arduino_IDE # PuTTY

[ShorTie8]

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

But truely not sure, and not worth my time to figure further, I'll just hack the build.xml to include it if peeps want it and see how it goes .. :/~

So I got the IDE and toolchain-avr of course, arduino-builder, astyle, bossac, Coan, ctags, OpenOCD and listSerialPortsC all being compiled and/or options to. Some of witch I don't know what they are specifically for or where they might go .. :/~

What other os specific things need compiling ??

[cmaglie]

@ShorTie8 do you want to express your disappontment? that's fine, I'm just asking to not talk on behalf of someone else, especially if that someone else is unaware of that and what are you saying is just your opinion.

to make the sig file you only need echo shasum $filename > $filename.sig`

shasum is needed to make the .sha files needed on build.xml.

To make the .sig file we use gpg and a private key to sign the package_index.json file. This is a common way to use strong crypto to validate downloaded package and protect against tampering.
Just for the note:

• the same approach is adopted by the major linux distributions (and the reason is clear, literature is full of cases when a single file compromised in an FTP server lead to thousand of user's PC infected).
• we do the check above on our package index, but anyone is free to use his own package_xxxxx_index.json, like a lot of 3rd party producer, unaffiliated with arduino, does (see https://github.com/arduino/Arduino/wiki/Unofficial-list-of-3rd-party-boards-support-urls)

Hope this clarifies the reasons behind the .sig file.

[cmaglie]

@cmaglie Actually it seems I think Board Manager is smart enough not to offer any platform specific downloads for unknown os types. Or so it seems because I was trying to add the sam stuff for yucks and couldn't find it.

Yes that's the reason why I added the extra case for ARM, once the builds are in place everything should go smooth ;-)

[PaulStoffregen]

I'm making good progress on the 2014q1 toolchain build. Details here: arduino/arduino-builder#105 (comment)

[NicoHood]

Thx for uploading the binaries. However the checksums did not match the checksums that I compiled myself. Maybe you had a different compiler version.

However I've updated the checksums, reverted the patch (for the json file) and updated the latest git. I also remove the binaries now.

@cmaglie The package_index.json does not seem to work for me. I still get this error, that no proper versions were found:

   83  cd arduino-1.6.8/
   85  cd dist/
   86  wget http://downloads.arduino.cc/packages/test_package_arm_index.json
   89  mv test_package_arm_index.json test_package_index.json 
   90  rm package_index.json

It seems to totally ignore the file (even if I delete it), so I do not know where it is searching. Or do I have to add this before compiling the IDE? Also could you please add the .sig file too (without the arm in the name, if that matters)?

Edit: I had to remove the file from .arduino15/package_index.json. However the test prefix does not skip the signing.

It also looks like this file totally differs from the current git: http://downloads.arduino.cc/packages/test_package_arm_index.json Also I could not apply the patch.

Then we need to merge the commit together (to remove the binaries from the history), test the IDE (on different systems) and also remove the additional readme that I added.

[cmaglie]

@NicoHood
I've rebased and squashed your commits in this branch:
https://github.com/arduino/Arduino/compare/master...cmaglie:arm-2?expand=1

This lead to a very clean history as you described, and now I'm able to build the IDE on the raspberry and run it successfully.

The package_index.json does not seem to work for me.

Yes, sorry, I forget to tell you that you can set the java property PACKAGE_INDEX_URL to override the default URL used by the IDE to download the package index, BTW you're right this won't bypass the signature check so I added a this possibility here 0e471bc

After that you can overrides the package_index.json in this way: c35250d (note that this commit will be removed before merging the branch upstream).

Now I'm looking into ArduinoBot to allow it build also the ARM version. We are very close!

[NicoHood]

Great work done!
So I guess after ArduinoBot can also build ARM builds we are done and also dont need to maintain 3rd party arm builds? This would be totally fantastic!

I am going to test this later. We then could add a list of tested boards and check if all of them work properly. Also the diff itself is quite simple. It was a lot of work to get there, but actually its impressive that its not much to change. ;)

[PaulStoffregen]

About 10 hours are left until the (hopefully) final build of the toolchain for Due & Zero is complete.
arduino/arduino-builder#105

[PaulStoffregen]

Finished building and checking the 2014q1 toolchain. :)
arduino/arduino-builder#105 (comment)

[theanch92]

Hi, I am trying to install the Arduino Two board on Raspberry but at compile time I getting the message "arm-none-eabi-g ++ exec format error" even with the 2014q1 toolchain by PaulStoffregen. Can someone help me?

[PaulStoffregen]

Can someone help me?

Maybe, but what steps would I need to take to recreate this error? I don't even know which RPi you're using. I can't see your screen, or what things you've click or what commands you've typed.

I do know that toolchain runs fine on my Raspberry Pi. I'm absolutely confident if I try it again the way I know how, it'll work perfectly, as it did a few weeks ago. Can you understand my reluctance to spend time on this without clear info about exactly the steps you took?

[theanch92]

Thanks for the reply. If you say it works then definitely something I may be wrong. Meanwhile, I have a Raspberry pi 3, and I'm using the build of NicoHood 1.6.8-rc. Everything ok for Arduino Uno / Mega. I have the need to program for an Arduino Due, in the Arduino IDE. I go to Tools-> Board Manager and add "Sam Arduino Board 1.6.7". Once completed I select the Arduino card Two and I try to compile I get error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.
Now I try to replace "arm-none-eabi-gcc" in /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc with your build for ARM.
But I have always the same error:
fork / exec /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++: exec format error.

Definitely wrong in something .. and I apologize for my stupidity

[PaulStoffregen]

On your RPi in a terminal, can you try these command, and post the result?

md5sum /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++
file /home/pi/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++

When I run them here on my Raspberry Pi, with the pathname to that file, I see this:

paul@raspberrypi ~ $ md5sum gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
bfd196cbc3e85ec922d8d5d7fa8d5449  gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
paul@raspberrypi ~ $ file gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++
gcc-arm-none-eabi-4_8-2014q1/bin/arm-none-eabi-g++: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.26, BuildID[sha1]=0x539864a83fc520418c0686b9d66e9da1a2f5fd44, stripped

[theanch92]

I do not know what strange reason, but does not copy the files .. they do not matched the md5! Now it works beautifully! Thank you so much for your patience, I owe you a drink

[cmaglie]

Supersed by #4517