Successfully built arduino-esp32 as component on esp-idf v4.2 release

[summivox]

After a whole afternoon googling around, I was finally able to build an esp-idf project with arduino-esp32 as a component.

I have committed my changes to my fork of this repo. This fork can be built without errors (see readme in the fork), in case someone else wants to try this approach.

I understand that v4.2 is not HEAD, and the component use case is not as popular as directly using the Arduino IDE. However, I do believe there is value in making this work for at least one known v4.x release, since HEAD of this repo does not work (as of now) with v4.3 release.

I'd love to hear feedback, specifically whether this works or not for you, and suggestions on how I can contribute my changes upstream.

[Jason2866]

I dont understand what the problem is when using the branch https://github.com/espressif/arduino-esp32/tree/idf-release/v4.2
Compile does work (in general) Tried with a modified Tasmota version https://github.com/Jason2866/Tasmota/tree/idf_42
Does compile successful. The Linking is failing...

Linking .pio/build/tasmota32/firmware.elf
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:30: warning: redeclaration of memory region `iram0_0_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:33: warning: redeclaration of memory region `iram0_2_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:50: warning: redeclaration of memory region `dram0_0_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:53: warning: redeclaration of memory region `drom0_0_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:57: warning: redeclaration of memory region `rtc_iram_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:59: warning: redeclaration of memory region `rtc_data_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:64: warning: redeclaration of memory region `rtc_slow_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:67: warning: redeclaration of memory region `extern_ram_seg'
/workspace/Tasmota/.platformio/packages/toolchain-xtensa32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld:esp32_out.ld:77: error: redefinition of memory region alias `rtc_data_location'
collect2: error: ld returned 1 exit status
*** [.pio/build/tasmota32/firmware.elf] Error 1

[app-js]

Your adaptation works well for me. Well done.

[lbernstone]

The idf/release-v4.2 branch is currently tied to idf master. That is nominally 4.2, although not the release version (a rose by any other name). It is unlikely that esp32s2 + arduino will work with release 4.2- definitely not the USB.
Your changes may be of interest to the platformio guys, but will be very quickly out of date as me-no-dev and the idf make changes. Once 1.0.5 gets out the door, more focus will go to 4.2/4.3 and it should be more consistently buildable.

[joshwapohlmann]

Thanks for your work. Working right now on the same problem.
Want to use the (great) IDF framework but also Arduino libraries without having to port them to IDF.

Just got started with this task. Got any hints for me what to look out for? Any good reads?

[lbernstone]

If you are not familiar with building esp-idf, I would recommend using lib-builder or my docker container

[joshwapohlmann]

@lbernstone i couldn't find adequate documentation for this process. the tool and it's README do not explain at all what it actually does.

[lbernstone]

https://github.com/espressif/arduino-esp32/blob/master/docs/lib_builder.md

[joshwapohlmann]

@lbernstone I saw that. But what does it do? Does it convert a specific Arduino library?
There is no workflow described how to combine a given arduino library with this too, where to put the library in question and how to embedd the converted library into a espidf project.

[joshwapohlmann]

So, after spending the rest of the evening ...

Platform.io specific

... figuring out how platform.io works and how everything is pulled together, this is what I found out:

1. When using this config

   [env:esp32dev]
   platform = https://github.com/platformio/platform-espressif32.git @ 3.0.0
   board = esp32dev
   framework = espidf

Generic

2. I followed the instructions to use arduino-esp32 as a ESP-IDF component and checked out esp-idf/v4.2. When I tried to run make menuconfig (with platformio it's pio run -t menuconfig) I get the error:

   fatal: Kein Git-Repository (oder irgendeines der Elternverzeichnisse): .git
   CMake Error at /home/joshwa/.platformio/packages/framework-espidf/tools/cmake/build.cmake:186 (message):
     Failed to resolve component 'main'.
   Call Stack (most recent call first):
     /home/joshwa/.platformio/packages/framework-espidf/tools/cmake/build.cmake:217 (__build_resolve_and_add_req)
     /home/joshwa/.platformio/packages/framework-espidf/tools/cmake/build.cmake:426 (__build_expand_requirements)
     /home/joshwa/.platformio/packages/framework-espidf/tools/cmake/project.cmake:395 (idf_build_process)
     CMakeLists.txt:3 (project)
   

3. When using the fork of @summivox the command works as intended.
   

4. Also with the current version of the branch esp-idf/v4.2 I got missing macros. With your changes, they are .. reduced...

   Compiling .pio/build/esp32dev/components/arduino/libraries/BLE/src/BLEServiceMap.o
   components/arduino/libraries/WiFiProv/src/WiFiProv.cpp: In function 'void get_device_service_name(prov_scheme_t, char*, size_t)':
   components/arduino/libraries/WiFiProv/src/WiFiProv.cpp:59:23: error: 'WIFI_PROV_SCHEME_BLE' was not declared in this scope
        if(prov_scheme == WIFI_PROV_SCHEME_BLE) {
                          ^~~~~~~~~~~~~~~~~~~~
   components/arduino/libraries/WiFiProv/src/WiFiProv.cpp:59:23: note: suggested alternative: 'WIFI_PROV_SCHEME_MAX'
        if(prov_scheme == WIFI_PROV_SCHEME_BLE) {
                          ^~~~~~~~~~~~~~~~~~~~
                          WIFI_PROV_SCHEME_MAX
   Compiling .pio/build/esp32dev/components/arduino/libraries/BLE/src/BLEUtils.o
   components/arduino/libraries/WiFiProv/src/WiFiProv.cpp: At global scope:
   components/arduino/libraries/WiFiProv/src/WiFiProv.cpp:69:18: warning: 'esp_err_t custom_prov_data_handler(uint32_t, const uint8_t*, ssize_t, uint8_t**, ssize_t*, void*)' defined but not used [-Wunused-function]
    static esp_err_t custom_prov_data_handler(uint32_t session_id, const uint8_t *inbuf, ssize_t inlen, uint8_t **outbuf, ssize_t *outlen, void *priv_data){
                     ^~~~~~~~~~~~~~~~~~~~~~~~
   Compiling .pio/build/esp32dev/components/arduino/libraries/BLE/src/BLEUUID.o
   Compiling .pio/build/esp32dev/components/arduino/libraries/BLE/src/BLEValue.o
   *** [.pio/build/esp32dev/components/arduino/libraries/WiFiProv/src/WiFiProv.o] Error 1

[enriquewph]

@joshwapohlmann
I got esp-idf(4.2.1) + arduino as a component working on platformIO
All that was needed was enabling bluetooth stack (or fully disabling it both in arduino + esp-idf)
check this out.
https://github.com/enriquewph/platform-espressif32/tree/develop/examples/espidf-arduino-blink

[Ryan-Carter]

I can't express how happy I am that this finally worked. Thank you @summivox
This is working for me as a component on platformIO with the default empty project and no changes to platformio.ini

I just followed the instructions here: https://github.com/summivox/arduino-esp32/blob/mod-idf-v4.2/docs/esp-idf_component.md

[GKFernandes]

Hello everyone,

First of all, thanks to @summivox for the solution, I spent a whole lot of time figuring this out and after finding this I was finally able to make it work!

As of now, what I'm trying to do is use the 1.22.0-80-g6c4433a5-5.2.0 toolchain to compile basic ESP-IDF examples, the reason to use this older toolchain is because I also want to run some Amazon FreeRTOS demos downloaded from the AWS console.

As of now, it seems 1.22.0-80-g6c4433a5-5.2.0 refers to ESP-IDF v3.3 and cannot compile v4.+ projects, even the most basic ones.

Does anyone know if arduino-esp32 works with the 1.22.0-80-g6c4433a5-5.2.0 toolchain?

I'll post my progress as I go through it.

Thanks!

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[pedros89]

Hello,
I would need to add arduino as IDF component to encrypt flash.
I have tried everything explained above but I have not been able to build successfully in windows10 IDF version 4.3 ; 4.2 ; 4.1. The closest I got was with @lbernstone docker.
https://hub.docker.com/r/lbernstone/esp32-arduino-lib-builder
the problem is that I need to add custom patrion table partitions.csv and I didn't manage to do it which doker.

Please can anybody point a way to add arduino as IDF component and the IDf version to be use, I have tried all without success for 2 days and nights now.

The fork of @summivox was promising but I ended up with a PIN_CONFIG error
https://github.com/summivox/arduino-esp32/tree/mod-idf-v4.2

If any of you could kidly address me to a correct way to add arduino as idf component (specifying the versions of IDF to be used) I commit to make a very clearly explained guide on how to do it and how to encrypt the ESP flash, which is why many people are driven to IDF.
Thank you for any help.

[stale]

[STALE_CLR] This issue has been removed from the stale queue. Please ensure activity to keep it openin the future.

[lbernstone]

Custom partitions (via partitions.csv) are handled by the Arduino IDE. If you need custom partitions with an IDF build, use the IDF functionality in menuconfig (https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/kconfig.html#config-partition-table-custom-filename).
There is fairly good documentation on building arduino as an IDF component. You are unlikely to find any help on this topic. In fact, your problem is the exact reason why I made the docker containers- it is not possible to just connect any version of IDF you want to a version of arduino. The lib-builder will provide a functional environment for building the master arduino branch with the appropriate IDF (typically master as well). The docker containers build a specific version of arduino against the IDF they were designed for.

[pedros89]

Yes @lbernstone thank you for sharing your Docker project which is very useful for standard projects. Unfortunately in my case I need to substitute bootloader, app and ota .bin files as explained in this post about flash encryption
#5645 (comment)
And I will need to add custom partitions.csv file.
As Docker is an image already made I cannot easily doing these manipulation. I will try again with Lib-builder but I had already done that with IDF 4.3.1 4.2.1 but without success.

[lbernstone]

If you look in the sdkconfig.h of whatever arduino you are trying to build, you can see the idf commit (git checkout xxxx) that matches. https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/esp32/include/config/sdkconfig.h#L676

[pedros89]

After many trials I have successfully built a project using Arduino IDF as component (but unfortunately with flash encryption not enabled as I am still working to make that work)

Here I share my experience. I am on Windows 10.

1 - It is very easy to install the IDF stack on Windows 10 as there is the Tools Installers at this link
then you can use both ESP-IDF Powershell or ESP-IDF Command Prompt to launch commands

2 - notice that in Arduino Core ESP32 there is a setting that tell us which IDF version to use with Arduino Master component as kindly suggested in this post by @lbernstone
It is release/v4.4

So select release/v4.4 and install this IDF version. If you have already installed another version follow the next steps. I tried also using IDF master and it is not building (errors coming up), so you cannot use IDF master. v5

(2.1) - At time of writing the first proposed version by IDF Tool Installers in the list is the 4.3.1, if you installed IDF version 4.3.1 and then built Arduino as component it will not work and will give you an error like I had experienced initially here

(2.2) - Check which IDF version you have with

git describe

(2.3) - To change IDF version:

git checkout release/v4.4
git pull
git submodule update --init --recursive

3 - now you can make a copy of the 'hello world' example and navigate in that folder

4- install Arduino component there (it will install master as showed on official website)

mkdir -p components && \
cd components && \
git clone https://github.com/espressif/arduino-esp32.git arduino && \
cd arduino && \
git submodule update --init --recursive && \
cd ../.. && \
idf.py menuconfig

4 - useidf.py menuconfigand

• Serial flash Config

  → Flash Sampling Mode DIO

  → Flash size according to your EPS32 hardware (16 MB in my case)

  → flash SPI speed according to your project (40Hz my case)

  → in my case I had a custom partition table default_16MB.csv taken from

C:\Users\YOUR_USER_NAME\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\partitions

just copy the .csv from the folder above and rename it as partitions.csv and place it in the main project folder (my case below)

C:\Users\YOUR_USERNAME\esp\esp-idf\examples\hello_worldMy

• Arduino configuration

   → Autostart Arduino setup and loop on boot
  
            → component config 
  
                  → mbedTLS 
  
                       → TLs key exchange method
  
                              → * Enable pre-shared-key ciphersuites
  

(details
https://esp32.com/viewtopic.php?t=15949
https://esp32.com/viewtopic.php?t=15148)

5 - useidf.py build

It should work

Unfortunately:

• if under security features you enable

  → enable flash encryption on boot

you will end up with this error

espressif/esp-idf#5043

I am still working on this

My aim is to make a guide that works for encrypting ESP32 flash as explained here
#5645 (comment)

but I need to manage to build with flash encryption option enable

I am doing all this because my application is flash encryption.

To have the solution for flash encryption just do as explained here
#5645 (comment)

[kostuch]

Thank you for the instruction.
But in my case (Platformio+IDF extension) it is not very helpful.
I managed to compile arduino component just commenting out lines causing errors in esp32-hal-i2c.c file.
Still have some warnings in other files, but they are harmless.
As I don't need I2C, it's enough for me. I hope that next PlatformIO releases will solve this issue.

[VojtechBartoska]

@summivox I'm closing this issue, seems it's not relevant anymore.

Current Arduino ESP32 core v2.0.2 works with IDF v4.4.

How to set it up is explained step by step here: https://docs.espressif.com/projects/arduino-esp32/en/latest/esp-idf_component.html

If needed, please reopen it.

Thanks for your contribution!

[QGB]

git clone --depth=1 --single-branch --branch idf-release/v4.4 [email protected]:espressif/arduino-esp32.git arduino-esp32-idf_v44

[alotionIG]

@summivox thank you so much,
I tried to use your modification and it works perfectly as expected, with a little bit of warning when it compiles.
But overall, it works for me.