Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration with @PlatformIO Build System #3321

Merged
merged 1 commit into from
Aug 7, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"name": "framework-arduinoespressif8266",
"description": "Arduino Wiring-based Framework (ESP8266 Core)",
"url": "https://github.com/esp8266/Arduino",
"version": "2.4.0-rc.1"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to keep this in sync with platform.txt?
I would probably go with 2.4.0-dev or something along these lines for git version, because -rc1 is a quite specific tagged version...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2.4.0-dev is good, but need to update it to the final release if you make release tag.

Currently, we have own forked and patched ESP826 Core for Arduino. Our community does not like it and asks us to use your repository. That is why I added package.json manifest. It's required for PIO Package Manager. After merge/release, I'll switch our development platform to this repo here.

}
56 changes: 56 additions & 0 deletions tests/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,53 @@ function build_package()
./build_boards_manager_package.sh
}


function install_platformio()
{
pip install -U https://github.com/platformio/platformio/archive/develop.zip
platformio platform install https://github.com/platformio/platform-espressif8266.git#feature/stage
sed -i 's/https:\/\/github\.com\/esp8266\/Arduino\.git/*/' ~/.platformio/platforms/espressif8266_stage/platform.json
ln -s $TRAVIS_BUILD_DIR ~/.platformio/packages/framework-arduinoespressif8266
# Install dependencies:
# - esp8266/examples/ConfigFile
pio lib install ArduinoJson
}

function build_sketches_with_platformio()
{
set +e
local srcpath=$1
local build_arg=$2
local sketches=$(find $srcpath -name *.ino)
for sketch in $sketches; do
local sketchdir=$(dirname $sketch)
local sketchdirname=$(basename $sketchdir)
local sketchname=$(basename $sketch)
if [[ "${sketchdirname}.ino" != "$sketchname" ]]; then
echo "Skipping $sketch, beacause it is not the main sketch file";
continue
fi;
if [[ -f "$sketchdir/.test.skip" ]]; then
echo -e "\n ------------ Skipping $sketch ------------ \n";
continue
fi
local build_cmd="pio ci $sketchdir $build_arg"
echo -e "\n ------------ Building $sketch ------------ \n";
echo "$build_cmd"
time ($build_cmd >build.log)
local result=$?
if [ $result -ne 0 ]; then
echo "Build failed ($1)"
echo "Build log:"
cat build.log
set -e
return $result
fi
rm build.log
done
set -e
}

function run_travis_ci_build()
{
# Build documentation using Sphinx
Expand Down Expand Up @@ -165,6 +212,15 @@ function run_travis_ci_build()
echo -e "travis_fold:start:size_report"
cat size.log
echo -e "travis_fold:end:size_report"

# PlatformIO
echo -e "travis_fold:start:install_platformio"
install_platformio
echo -e "travis_fold:end:install_platformio"

echo -e "travis_fold:start:build_sketches_with_platformio"
build_sketches_with_platformio $TRAVIS_BUILD_DIR/libraries "--board nodemcuv2 --verbose"
echo -e "travis_fold:end:build_sketches_with_platformio"
}

set -e
Expand Down
95 changes: 95 additions & 0 deletions tools/platformio-build.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Copyright (c) 2014-present PlatformIO <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Arduino

Arduino Wiring-based Framework allows writing cross-platform software to
control devices attached to a wide range of Arduino boards to create all
kinds of creative coding, interactive objects, spaces or physical experiences.

http://arduino.cc/en/Reference/HomePage
"""

# Extends: https://github.com/platformio/platform-espressif8266/blob/develop/builder/main.py

from os.path import isdir, join

from SCons.Script import DefaultEnvironment

env = DefaultEnvironment()
platform = env.PioPlatform()

FRAMEWORK_DIR = platform.get_package_dir("framework-arduinoespressif8266")
assert isdir(FRAMEWORK_DIR)


env.Prepend(
CPPDEFINES=[
("ARDUINO", 10600),
"LWIP_OPEN_SRC"
],
CPPPATH=[
join(FRAMEWORK_DIR, "tools", "sdk", "include"),
join(FRAMEWORK_DIR, "tools", "sdk", "lwip", "include"),
join(FRAMEWORK_DIR, "tools", "sdk", "libc",
"xtensa-lx106-elf", "include"),
join(FRAMEWORK_DIR, "cores", env.BoardConfig().get("build.core"))
],
LIBPATH=[
join(FRAMEWORK_DIR, "tools", "sdk", "lib"),
join(FRAMEWORK_DIR, "tools", "sdk", "ld"),
join(FRAMEWORK_DIR, "tools", "sdk", "libc", "xtensa-lx106-elf", "lib")
],
LIBS=[
"mesh", "wpa2", "smartconfig", "espnow", "pp", "main", "wpa", "lwip_gcc",
"net80211", "wps", "crypto", "phy", "hal", "axtls", "gcc",
"m", "c", "stdc++"
]
)

env.Append(
LIBSOURCE_DIRS=[
join(FRAMEWORK_DIR, "libraries")
],
LINKFLAGS=[
"-Wl,-wrap,system_restart_local",
"-Wl,-wrap,spi_flash_read"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to pull these flags from platform.txt? As far as i can tell, platformio-build.py is normal Python program, so opening platform.txt and extracting flags from it seems to be quite doable. Otherwise the flags would need to be updated in two files...

Something along the lines of what makeEspArduino does, but in Python.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arduino Build System is a self-written bunch of files where they play with common problems which were resolved by make and other build tools a lot of years ago. We don't depend on Arduino IDE, their build system or even Java. For us, Arduino is just a framework with some "build logic".

We didn't invent the new wheel and our PIO Build System is based on SCons construction tool. It's 100% pure Python implementation with Zero-dependencies to other packages or even to the OS software. We applied a lot of patches to SCons to make it better for embedded using and now it's super-fast and reliable.

In platform.txt all flags are located in ONE LINE, where in PlatformIO we separate them into multiple scopes. Later, a developer is able to "unflag" some of them directly from project configuration file or add new. Here is very important that each flag/option is located in correct scope.

We would be thankful if you support this build script. I see that it doesn't require a lot of efforts. Take a look at platformio-build.py for ESP32 by @me-no-dev . I don't see here a hundred commits.

In any case, you can ping me or @valeros any time and we will help as soon as possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is very important that each flag/option is located in correct scope.

Can you point me to the description which flag option should be located in which scope? If there is such a description, it should be more or less straightforward to write the code which will parse flags from platform.txt and put them into the correct scopes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the changes I have done to the file are because libs change often on ESP32, other than that I let PlatformIO be as it is. I have asked @ivankravets a couple of times when I implement a new core feature (like partitions), so it kinda helps keep both in sync. He would not need to read through every commit to learn that a new command/feature is required :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is base build script for ESP8266. Build script from this PR extends that script.

Also, full list with SCons construction variables

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, i'm reading these and given that there doesn't seem to be a strict rule which flag has to go where, i'm trying to find the pattern. Why do -fno-inline-functions, -ffunction-sections, -fdata-sections go into the ESP8266-specific script? Shouldn't that be a framework-level choice? None of these flags are specific to the ESP8266 or Xtensa.F_CPU flag is also an Arduino-specific thing, should it not be in the framework level flags? Sorry for a lot of questions folks, this thing is new to me, once i understand it i can probably clean this up to some extent.

Another way of approaching this is generating both platform.txt and the platformio script from a single configuration file. That would likely be even more straightforward. Would reading these flags from some .json or .yml file work for you?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is base configuration. You can "override" all scopes with own flags/data. Use env.Replace(SCOPE1=value, SCOPE2=value..) instead of env.Append(SCOPE1=value, SCOPE2=value..).

To dump current build environment, use print env.Dump() in any place

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that this is the base configuration, and i understand that it can be overridden...

My question above was why these flags are in the ESP8266-specific configuration, when logically they should not be in the chip-dependent part.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another way of approaching this is generating both platform.txt and the platformio script from a single configuration file.

This is a good idea. I'm not sure but @projectgus works on the same for ESP-IDF.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My question above was why these flags are in the ESP8266-specific configuration, when logically they should not be in the chip-dependent part.

Good question :) This is a historical problem. We had 1 script for ESP8266 in PlO 2.0 and kept inside PIO Core (2 years ago). 1 year ago we released PIO 3.0 with decentralized platforms where each platform using PIO Build API decides how to build own frameworks, SDKs, HALs, etc.

Later, we added support for ESP8266 SDK and Simba framework. After that, we don't touch that part of code, need to check if Simba uses our default environment.

In any case, we are open for discussion and PR. Feel free to make contribution to https://github.com/platformio/platform-espressif8266

]
)

#
# Target: Build Core Library
#

libs = []

if "build.variant" in env.BoardConfig():
env.Append(
CPPPATH=[
join(FRAMEWORK_DIR, "variants",
env.BoardConfig().get("build.variant"))
]
)
libs.append(env.BuildLibrary(
join("$BUILD_DIR", "FrameworkArduinoVariant"),
join(FRAMEWORK_DIR, "variants", env.BoardConfig().get("build.variant"))
))

libs.append(env.BuildLibrary(
join("$BUILD_DIR", "FrameworkArduino"),
join(FRAMEWORK_DIR, "cores", env.BoardConfig().get("build.core"))
))

env.Prepend(LIBS=libs)
18 changes: 18 additions & 0 deletions tools/sdk/ld/eagle.app.v6.common.ld
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,18 @@ SECTIONS
*.c.o( EXCLUDE_FILE (umm_malloc.c.o) .literal*, \
EXCLUDE_FILE (umm_malloc.c.o) .text*)
*.cpp.o(.literal*, .text*)
*.pioenvs\\*\\lib*.a:(EXCLUDE_FILE (umm_malloc.o) .literal*, \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to tweak object file names to make them same as in Arduino?

Copy link
Collaborator Author

@ivankravets ivankravets Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same story. In PlatformIO (platformio.ini), you can have multiple build environment per a single project. For example:

[env:release]
platform = espressif8266
framework = arduino
board = myboard
build_flags = -DRELEASE ... other flags
lib_deps =
    StableLib@~1.2.3
    otherLib@>2.0

[env:debug]
platform = espressif8266
framework = arduino
board = myboard
build_flags = -DDEBUG... other flags
lib_deps =
    https://git.repo/stable.git
    otherLib@~3.0

You will have 2 different build environments and the final objects will be located %project_root%/.pioenvs/%env_name%/.... All dependent libraries will be built to static libraries and placed to .pioenvs folder per each build environment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the case, the build product may be different, because Arduino doesn't link libraries into archive files, an archive file is only produced for the "core". All object files from libraries get passed to the linker. This may have implications for libraries which override weak functions defined in the core. If the function within the library is placed into a separate object file, and this object file doesn't contain any other unresolved functions, the linker will not include this object file from the library into the final output. We don't happen to have this kind of situation now, but once we have it eventually, working around this may prove interesting :)

But we have diverged from the original question... If you don't mind, i'll set up some code generation to create these extra LD script lines.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may have implications for libraries which override weak functions defined in the core.

PlatformIO Build System is very flexible. You can even override the WHOLE build process for a single library with a custom extra script. According to your question, library archiving is the default behavior. You are able to turn if off for specific library. See http://docs.platformio.org/en/latest/librarymanager/config.html#libarchive

If you don't mind, i'll set up some code generation to create these extra LD script lines.

No problem, thanks! For us would be good if you support this script. Users will be happy because will not be a delay "When we update script for new changes?". We have this problem for other 20 development platforms. Need to maintain all of them :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are able to turn if off for specific library. See http://docs.platformio.org/en/latest/librarymanager/config.html#libarchive

That seems to be a library-level switch. Can we apply this switch at framework level?

Also, back to my original question, is it possible to tweak object file names at framework level to make them the same as in Arduino? I.e. .c files get compiled into .c.o, .cpp into .cpp.o and so on? Directory layout doesn't matter much in this case, as the LD script currently has filename-dependent rules only.

Need to maintain all of them

Well, that's the part of platformio's value proposition, so there's probably no way around that.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems to be a library-level switch. Can we apply this switch at framework level?

Currently, is not possible. Need to create library.json manifest and make configuration. However, I can open issue for that if you need.

I.e. .c files get compiled into .c.o, .cpp into .cpp.o and so on?

It's possible if build each file separately, see docs. We don't play with env.Object() and use env.VariantDir() where the rest work for us SCons does automatically. Is this critical for you?

EXCLUDE_FILE (umm_malloc.o) .text*)
*.pioenvs/*/lib*.a:(EXCLUDE_FILE (umm_malloc.o) .literal*, \
EXCLUDE_FILE (umm_malloc.o) .text*)
*.pioenvs\\*\\lib\*.a:(EXCLUDE_FILE (umm_malloc.o) .literal*, \
EXCLUDE_FILE (umm_malloc.o) .text*)
*.pioenvs/*/lib/*.a:(EXCLUDE_FILE (umm_malloc.o) .literal*, \
EXCLUDE_FILE (umm_malloc.o) .text*)
*.pioenvs\\*\\src\\*.o(EXCLUDE_FILE (umm_malloc.o) .literal*, \
EXCLUDE_FILE (umm_malloc.o) .text*)
*.pioenvs/*/src/*.o(EXCLUDE_FILE (umm_malloc.o) .literal*, \
EXCLUDE_FILE (umm_malloc.o) .text*)
*libc.a:(.literal .text .literal.* .text.*)
*libm.a:(.literal .text .literal.* .text.*)
*libgcc.a:_umoddi3.o(.literal .text)
Expand Down Expand Up @@ -205,6 +217,12 @@ SECTIONS
*(.literal .text .literal.* .text.* .stub .gnu.warning .gnu.linkonce.literal.* .gnu.linkonce.t.*.literal .gnu.linkonce.t.*)
*.cpp.o(.iram.text)
*.c.o(.iram.text)
*.pioenvs\\*\\lib*.a:(.iram.text)
*.pioenvs/*/lib*.a:(.iram.text)
*.pioenvs\\*\\lib\\*.a:(.iram.text)
*.pioenvs/*/lib/*.a:(.iram.text)
*.pioenvs\\*\\src\\*.o(.iram.text)
*.pioenvs/*/src/*.o(.iram.text)
*(.fini.literal)
*(.fini)
*(.gnu.version)
Expand Down