CMake: From USCRPL/mbed-cmake to Mbed OS first-party CMake

[ladislas]

Description of defect

Intro

As mentioned in #13974, I'm in the process of moving our current project from USCRPL/mbed-cmake to mbed os first-party CMake.

I thought it might be interesting for others to share our progress, our questions, feedback and issues we might be facing. I think that github is better than the forum for this as it is easier to share code, other repos and reference issues and PR

Our project is quite complex, with a main src directory containing our product firmware (main program). We have different drivers and libraries that are being used by the firmware.

We also have a lot of spikes: standalone projects (with their own main.cpp) to test components, features, libs, technical solutions, etc.

The following repository is a simpler example of our big project, and I'll use it to test all the features.

https://github.com/ladislas/mbed-cmake-template

How it works now

USCRPL/mbed-cmake allows us to have the spikes and the main program live in the same repository and compile together without any issues. We then use openocd to flash the .bin we want to use in our product.

USCRPL/mbed-cmake first creates a mbed-os-static STATIC library (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L35-L37) which is then wrapped into an mbed-os INTERFACE that defined the linker script and link options (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L50-L58). It's this mbed-os INTERFACE that our main program, spikes drivers and libs link with.

From a user perspective, mbed-os-static is compiled once for the main program and for all the spikes, drivers and libs. Adding a new spike with just a main.cpp file, will only increment the whole compilation steps number by one.

First try with Mbed OS first-party CMake

On the other hand, mbed os first-party cmake "recompiles" all the needed sources for each target depending on it. Adding a new spike linking with mbed-os will throw in the compilation a few hundred more "steps". Looking at the output, one can clearly see that mbed os or target files are compiled multiple times for the libs, the spikes and the main program.

Issues

And then it stops as the way it handles the linker script is not made for multiple add_executable targets.

Another issue was that adding another executable and calling mbed_set_mbed_target_linker_script resulted in the following error about .compile_time_defs.txt:

CMake Error: Files to be generated by multiple different commands: "/Users/ladislas/dev/ladislas/mbed-cmake-template/build/compile_time_defs.txt"

The issue comes from here:

mbed-os/tools/cmake/toolchain.cmake

Lines 22 to 23 in 33a7e66

	file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
	set(${definitions_file} @${CMAKE_BINARY_DIR}/compile_time_defs.txt PARENT_SCOPE)

and is easily fixed by using:

-    file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
-    set(${definitions_file} @${CMAKE_BINARY_DIR}/compile_time_defs.txt PARENT_SCOPE)
+    file(GENERATE OUTPUT "${CMAKE_BINARY_DIR}/${target}.compile_time_defs.txt" CONTENT "${_compile_definitions}\n")
+    set(${definitions_file} @${CMAKE_BINARY_DIR}/${target}.compile_time_defs.txt PARENT_SCOPE)

Note that the two files generated are identical. If they can never be different, it would be better to juste generate one and use it for all executable.

Questions

Please note I'm neither a CMake expert nor a .ld expert.

From my understanding: for the same mbed target (DISCO_F769NI in our case), the linker script is always the same. If we change target, we must change the linker script. In real life projects this happens seldom as your board/target is usually the same for a long period of time. But even if we need to change it, a new run of mbed-tools configure will provide everything needed.

Currently mbed-os, mbed-core, mbed-rtos are defined as INTERFACE libraries, meaning they are not compiled on their own (as static or object libraries would be), but become part of the target (other library or executable) that links with it.

So calling mbed_set_mbed_target_linker_script for each executable will also call mbed_set_toolchain_options multiple times.

My first question is:

What is the reason for using INTERFACE for the whole mbed-os? Why not use a static library wrapped in an interface?

And secondly:

Is supporting a project structure such as mine with multiple executable targets on the roadmap?

I'd be happy to help in this case.

Thanks for reading till here and also thanks for bringing CMake to mbed-os! :)

(cc @0xc0170 @hugueskamba @rajkan01 @multiplemonomials @ProExpertProg)

Target(s) affected by this defect ?

DISCO_F769NI

Toolchain(s) (name and version) displaying this defect ?

arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Arm Embedded Toolchain 9-2020-q2-update) 9.3.1 20200408 (release)

What version of Mbed-os are you using (tag or sha) ?

33a7e66

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed --version
1.10.4

mbed-tools --version
3.5.1.dev36

How is this defect reproduced ?

to do

[multiplemonomials]

I haven't had a chance to poke around the official CMake scripts that much, but I do agree that building the linker script multiple times seems like the wrong way to do it since it doesn't change. It would be better to just generate it once and then attach it as part of a target's link flags (how mbed-cmake does it).

Also, it definitely doesn't seem right if the entire source code is getting built once for each target. My understanding is that the mbed source code is supposed to get built as an object library, then the objects are supposed to get attached using TARGET_SOURCES to an interface library. But it sounds like maybe the source files themselves got added as the interface sources, rather than the object files.

[ciarmcom]

@ladislas thank you for raising this issue.Please take a look at the following comments:

We cannot automatically identify a release based on the version of Mbed OS that you have provided.
Please provide either a single valid sha of the form #abcde12 or #3b8265d70af32261311a06e423ca33434d8d80de
or a single valid release tag of the form mbed-os-x.y.z .
E.g. 'master' is not a unique single snapshot.
NOTE: If there are fields which are not applicable then please just add 'n/a' or 'None'.This indicates to us that at least all the fields have been considered.
Please update the issue header with the missing information, the issue will not be mirroredto our internal defect tracking system or investigated until this has been fully resolved.

[ciarmcom]

Thank you for raising this detailed GitHub issue. I am now notifying our internal issue triagers.
Internal Jira reference: https://jira.arm.com/browse/IOTOSM-2888

[ladislas]

My understanding is that the mbed source code is supposed to get built as an object library, then the objects are supposed to get attached using TARGET_SOURCES to an interface library.

@multiplemonomials it was since e7c0d93 that changed it from OBJECT to INTERFACE.

Would OBJECT be better than STATIC libraries?

[multiplemonomials]

Yeah that commit is... not right, or at least it's not correct to the design we originally planned (I worked with Martin and Hugues to plan out how the object and interface libraries would be used). The interface libraries do need to be there, but there also need to be object libraries as well. The object libraries would be used to compile the sources, then the compiled objects from those libraries should get added via target_sources to the interface libraries.

[0xc0170]

Thanks for the feedback, we will review and respond to all issues/questions.

[0xc0170]

@ladislas can we convert example blinky to be able to reproduce the issue with spikes? I've checked your template repository, it does not reference mbedtools, so I might need an assistance to reproduce it there (how to).

Is supporting a project structure such as mine with multiple executable targets on the roadmap?

I would say it should work as CMake libraries should be reusable. We are currently looking at testing and it might be the same use case.

1. 2 or more applications
   2 spikes as you call, just 2 different applications using the same mbed-os library ? I've created a branch with 2 main object libraries to test with https://github.com/0xc0170/mbed-os/tree/cmake-fix-objects - can you test ?

2. linker issue
   we will review if we can fix the linker not to be dependent on application target (it's related to the first one so we might want to first fix that and then get to this one).

We initially worked only on object libraries but we faced cmake issues with object libraries (we should be able to find PRs/commits) and converted to interface libraries. We exposed the main libraries mbed-os and mbed-baremetal as interface as well - it has some drawbacks as mentioned here, we should be able to address them. The branch I shared above was a concept we had previously.

From my understanding: for the same mbed target (DISCO_F769NI in our case), the linker script is always the same. If we change target, we must change the linker script. In real life projects this happens seldom as your board/target is usually the same for a long period of time. But even if we need to change it, a new run of mbed-tools configure will provide everything needed.

Consider ld file being a template that is changed based on app needs (sections could be adjusted, an example is bootloader). Therefore we preprocess linker scripts. If you change an app, linker script must be regenerated - reusing the same template. We always wanted to have one .ld template and fill it in.
Also for linker script file, if you can provide us a repo where we can reproduce, we will review in more detail.

[ladislas]

can we convert example blinky to be able to reproduce the issue with spikes? I've checked your template repository, it does not reference mbedtools, so I might need an assistance to reproduce it there (how to).

@0xc0170 yes, I'll work on a PR! I'll also push to my template on a specific branch for testing.

I'll also test your branch and let you know.

And thanks for the explanation on the linker file. 👍

[multiplemonomials]

Okay, I've got time to take a look into what's going on here.
I think I found the source of some of these errors, but before that, I got distracted by a giant, horrible, flashing red flag:

Detecting C Compiler ABI -- Failed

This is the same issue that we talked about at our meeting, and it seems that it still hasn't been fixed. Basically, the compiler flags that CMake is using to do all of its internal tests are extremely messed up, and you guys just papered over the issue with:

set(CMAKE_C_COMPILER_WORKS TRUE)
set(CMAKE_CXX_COMPILER_WORKS TRUE)

This is a really bad idea, because it means that all sorts of things will be subtly screwed up later on, since CMake cannot successfully run compilation or linking tests.
The real way to fix it is to add the needed compiler flags to CMake's global flag variables, like I explained. So, I made a PR to fix this: #13987 . Please take a look when you have a chance!

[multiplemonomials]

@ladislas
When you submit your PR, this also needs to get fixed in mbed-os/CMakeLists.txt:

- COMMAND "arm-none-eabi-cpp" ${_linker_preprocess_definitions} -x assembler-with-cpp -E -Wp,-P
+ COMMAND ${CMAKE_CXX_COMPILER} ${_linker_preprocess_definitions} -x assembler-with-cpp -E -Wp,-P

The existing code requires arm-none-eabi-cpp to be on the PATH, which is not guaranteed. Instead, it's better to use the compiler that we already know the path to. The options -x assembler-with-cpp -E are enough to convert the compiler into a preprocessor.

[multiplemonomials]

Update: I understand a little better why the current method was used (compiling all mbed sources for each target). The issue is that you have a set of base Mbed sources, mbed-core. You also have a number of extra modules, like mbed-rtos. However, these extra modules can add extra compile definitions (such as MBED_CONF_RTOS_PRESENT) that change the behavior of source files in mbed-core. This means that you can theoretically end up with a different product in mbed-core depending on what extra modules you link.

IMO, this is somewhat bad code design to have things tightly coupled like this, and it's hard to fix at the build system level. However, I also don't think it's a good solution to require rebuilding all mbed sources for each target. For now, I'm going to try at least making mbed-baremetal and mbed-rtos separate libraries to fix that dependency, but I'm not sure how many extra modules have this behavior.

[multiplemonomials]

OK! I think I figured out a method that works to fix this.
@ladislas Check my branch here: https://github.com/multiplemonomials/mbed-os/tree/cmake-object-libs
It should fix your issue, I've converted mbed-os, mbed-baremetal, and mbed-storage-kvstore to the new system so they should only get built once regardless of how many targets they're used in.

I realize that the solution I told you guys originally doesn't actually work given the constraints above. However, I figured out something that should work for what you need.

For the main OS targets, I used this strategy:

• Add flags targets, mbed-base-flags and mbed-rtos-flags. All flags (options, include dirs, definitions) that used to get applied to mbed-core and mbed-rtos now get put in these interface targets.
• Add object libraries mbed-baremetal-obj and mbed-os-obj. All mbed-core sources get added to mbed-baremetal-obj to be compiled. mbed-os-obj receives the RTOS sources as well as the core sources.
• Add interface libraries mbed-baremetal and mbed-os to wrap the object libraries.

For extra modules, I used this strategy:

• Convert add_library(mbed-xxxxx INTERFACE) into add_library(mbed-xxxxx OBJECT EXCLUDE_FROM_ALL). EXCLUDE_FROM_ALL, of course, prevents the library from being built unless something actually depends on it.
• Change all flag-modifying functions (e.g. target_include_directories()) to be public so they apply to the library and its dependees
• Change all source-adding functions from INTERFACE to PRIVATE so that sources get added to the object lib but not its dependees.
• Make sure mbed-xxxx links to either mbed-base-flags or something that does
  I tested this process on kvstore and its dependencies and it seemed to work fine. I kinda forgot that in recent cmake you can just link to object libraries normally without an interface library (not so in the CMake version I have to use at my internship).

Note that this process makes some assumptions:

• None of the core sources are affected by compile definitions originating in any of the extra modules (please $diety let this be true)
• None of the extra modules are affected by the RTOS being enabled or disabled.
  If these are violated anywhere then some extra code would be needed, like making one version of a module that has the RTOS enabled and one that doesn't.

Let me know what you think about this plan -- I'm hoping that it can make life easier for us users. But if it seems totally unworkable, let me know as I have another idea that might also be feasible.

[0xc0170]

Thanks @multiplemonomials , we will review but most likely the next week (some of us are out of office in the next days)

[0xc0170]

I run initial compiler checks to see what flags and files get into the app. I'll look at the structure the next week.

Compiling K64F for GCC ARM shows startup is duplicated:

arm-none-eabi/bin/ld.exe: CMakeFiles/mbed-os-example-blinky.dir/mbed-os/targets/TARGET_Freescale/TARGET_MCUXpresso_MCUS/TARGET_MCU_K64F/device/TOOLCHAIN_GCC_ARM/startup_MK64F12.S.obj: in function `__isr_vector': (.isr_vector+0x0): multiple definition of `__isr_vector'; mbed-os/CMakeFiles/mbed-os-obj.dir/targets/TARGET_Freescale/TARGET_MCUXpresso_MCUS/TARGET_MCU_K64F/device/TOOLCHAIN_GCC_ARM/startup_MK64F12.S.obj:(.isr_vector+0x0): first defined here

ARMClang:

OOLCHAIN_ARM_STD\startup_MK64F12.S
armclang: error: ARM Compiler does not support '-mcpu=Cortex-M4-mcpu=cortex-m4'
armclang: error: ARM Compiler does not support '-mcpu=Cortex-M4-mcpu=cortex-m4'
armclang: error: armasm does not support CPU 'cortex-m4-mcpu=cortex-m4'
armclang: error: armasm does not support FPU 'unknown'

@ladislas if you can share the multiple target example, or we shall create our own to test the fix?

[multiplemonomials]

Oh whoops, there was an issue where the wrong linker script path was passed for GCC_ARM. I pushed a new commit to #13987 to fix it. I also found a variable name conflict with the profile scripts that was probably causing the arm compiler error, pushed a fix for that too.

[ladislas]

if you can share the multiple target example, or we shall create our own to test the fix?

@0xc0170 yes, I'm working on a simple example, I'll share it later today.

[ladislas]

@multiplemonomials @0xc0170

Alright, I've setup a working example of our project structure, available here:

https://github.com/ladislas/mbed-cmake-template/tree/feature/move-to-vanilla-cmake

Instructions are available here:

https://github.com/ladislas/mbed-cmake-template/tree/feature/move-to-vanilla-cmake#-wip---from-uscrplmbed-cmake-to-mbed-os-first-party-cmake

If you want to compare the same template with USCRPL/mbed-cmake, you can follow the instructions in develop

https://github.com/ladislas/mbed-cmake-template#how-to-use

[multiplemonomials]

OK @ladislas I tested your example, and my cmake-object-libs branch that I linked earlier is able to compile it correctly with no other changes and only 247 files compiled.

[ladislas]

@multiplemonomials that's great news, I've just tested it myself and I can confirm it compiles both the main src and the spike without any problems :)

The number of files to compile (spike + lib + src) is also much lower than it was previously:

• 249 - with cmake-object-libs
• 490 - with USCRPL/mbed-cmake
• 735 - 490 + 245 for first party mbed cmake

[multiplemonomials]

@0xc0170 @hugueskamba Have you had a chance to look at my cmake-object-libs branch? I think it's an important change to make but it might be too much of a project for me to do by myself, I was hoping you could take the ideas from it and make a PR that implements it.

[0xc0170]

I was looking for the reference this week and could not find it in PRs, it's on a branch! I'll create a task for the next sprint to look at it.

@ladislas Check my branch here: https://github.com/multiplemonomials/mbed-os/tree/cmake-object-libs

got it now!

[ladislas]

@multiplemonomials Any chance you could rebase your branch on top of master?

[boraozgen]

I stumbled upon this thread when I noticed that I cannot add compiler options only for my code without adding them to mbed-os. AFAIU, interface libraries are compiled with the options of the target, i.e. the compiler options apply to all linked interface libraries. Object libraries would solve this issue too.

[ladislas]

@0xc0170 where are we on that matter? :)

[0xc0170]

The state stands as it was, it is on hold for now (#13981 (comment)). We most likely can resume the work in Q3.

I am happy to help meanwhile if there are any contributions for the feature branch. I'll rebase the feature branch tomorrow to bring it up to date after unittests changes.

[multiplemonomials]

@boraozgen Note that my patch I submitted for #14274 will provide a workaround for that issue, and you may even just be able to drop mbed_create_distro.cmake into your project and go without changing Mbed source files in the latest version.

[0xc0170]

We hope to get #15126 into master and close this issue.

[ciarmcom]

We closed this issue because it has been inactive for quite some time and we believe it to be low priority. If you think that the priority should be higher, then please reopen with your justification for increasing the priority.

[ladislas]

#15126 has just been merged, we'll start moving to vanilla m bed cmake next week. I'll keep you posted here.

Maybe it's good to reopen until the transition is over.

[0xc0170]

The discussion can be ongoing here. If there are any other issues, you can report via a new bug report.

[ladislas]

We are currently trying to move from USCRPL/mbed-cmake to Mbed's "vanilla" CMake.

leka/LekaOS#397

The process is not as straight forward as I hoped it would be, compared to setting up USCRPL/mbed-cmake in the first place.

I'll report the issues here and open new ones if needed:

• We just want to use mbed-wifi to access the ESP8266 driver and we end up pull hundred of unrelated files as they all depend on each other. It adds a lot of compilation time and it doesn't feel right to have those files compiled when you don't need them -- related issues: CMake netsockets: too many libraries dependencies #14413 & CMake cellular: netsocket dependency #14414

• Custom targets must have an mbed- prefix with a name in lower case. We have multiple custom targets for different versions of our hardware. Using the mbed- prefix is strange enough as those targets are not coming to mbed and it is not written in the documentation. It took me quite sometime to understand what was wrong. In the end we are forced to change the name of our targets from target_LEKA_V1_2_DEV to mbed-leka-v1-2-dev. We can live with the leka-v1-2-dev but forcing users to format the name of their targets it wrong, the prefix should not be required and if we really can't do without it, documentation should explain the what and why -- related issue CMake - Custom hardware targets must start with "mbed-" prefix #15197

• Compilation times are multiplied by 3 - on a high end Macbook Pro 16, building from scratch, using create_distro (mbed_create_distro() reborn: a function to make adding multiple targets easy #15126 ) without ccache goes from 6 minutes to 18 minutes. We have 31 static libs + 24 executable targets. It seems that it takes a lot of time to build and link the final executables. I was not able to find why. Any help would be greatly appreciated.

• Juggling with .json files - This is not something new brought by mbed-tools, but something that has been bothering us for quite sometime. Previously with mbed-cli we did not have a choice to use something else, but with CMake, we want to add definitions "by hand" and make things more explicit. JSON is a really bad format to read and edit by humans. I wish there was a way to go full CMake.

Conclusion on 2021/12/31:

After spending 48h hours on this, I was hoping the transition would be easier. I'm a bit disappointed by the fact that compilation and linking are so long. The modularity CMake should bring is not there. This is mostly due by the use of INTERFACE everywhere. create_distro helps a bit, but it's still not the CMake way.

CMake target naming could be improved. drivers are in mbed-core, mbed-rtos is actually not the rtos directory but more the CMSIS/RTOS2 part. rtos is part of mbed-core. This is all very confusing for people digging deeper. mbed-rtos could be renamed mbed-cmsis-rtos, drivers directory could have it's own target mbed-drivers or even be separated in sub-drivers, like mbed-drivers-i2c, etc.

Right now, I'm not sure what to do. Should I keep fighting the tools or keep our current setup using USCRPL/mbed-cmake?

Our product will be on the market in one month, and as much as I would like to help improve mbed, I'm not sure we have the time and resources right now.

Again, a clear roadmap about what to expect in the coming month would really help us.

[multiplemonomials]

Wait, so create_distro() didn't work to reduce the compile time for building multiple targets? Is it because there are more source files than the corresponding mbed-cmake project? Or is it that you had to make multiple distros for things with different functionality?

[ladislas]

It does reduce compared to not using it, but not using it would be shooting yourself in the foot so I'm not comparing to that. I'm comparing to USCRPL/mbed-cmake.

We only have one "distro" with everything we need.

Compilation time is longer because there are more files, files we were able to exclude with .mbedignore before.

What worries me is the final compilation and link time it takes to build the final executables. With USCRPL/mbed-cmake, mbed was compiling, our libs/drivers were compiling, then all the executable targets were compiling and then linking and it a was all one after the other, with memap output, without any wait time in between. Now it just hangs for several seconds or tens of seconds and I don't know why.

Also I've noticed that they are like hundreds of flags passed for compiling each file. I don't know if it has an influence but the output is really hard to parse and read because of that.

[multiplemonomials]

Hmm, it's possible the additional compilation time is because mbed-cmake compiles all of mbed-os into a .a STATIC library, while the official build system uses an OBJECTlibrary, so the linker has to read hundreds of individual .o files every time it links something. The reason for this is that STATIC libraries are not usable with Arm Compiler since Arm Compiler lacks the -Wl,--whole-archive linker flag. So, to make something that works with both, you need to use OBJECT libraries. Maybe you could try editing the create_distro() code locally to use a STATIC library and see if that produces any performance benefit.

Also yeah, I was bothered too by the sheer number of #define flags that Mbed OS uses when I made mbed-cmake, which is why I had the Python script dump them all into a .h file which then gets force-included to everything with the -include flag. It makes for a much cleaner command line.

[ladislas]

Thanks @multiplemonomials for the ideas!

i've tried the STATIC library changing this line

mbed-os/tools/cmake/create_distro.cmake

Line 35 in 149c5b1

add_library(${NAME} OBJECT)

like this

- 	add_library(${NAME} OBJECT)
+ 	add_library(${NAME} STATIC)

But that produces the following error:

[1137/1166] Linking CXX executable spikes/mbed_blinky/spike_mbed_blinky.elf
FAILED: spikes/mbed_blinky/spike_mbed_blinky.elf
: && /usr/local/bin/arm-none-eabi-g++ -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -ffunction-sections -fdata-sections -funsigned-char -fomit-frame-pointer -g3 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=softfp -mcpu=cortex-m7 -Wl,--start-group -lstdc++ -lsupc++ -lm -lc -lgcc -lnosys -Wl,--end-group -specs=nosys.specs -Wl,--cref    -Wl,-Map=/Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf.map -Wl,--gc-sections -Wl,--wrap,main -Wl,--wrap,_malloc_r -Wl,--wrap,_free_r -Wl,--wrap,_realloc_r -Wl,--wrap,_memalign_r -Wl,--wrap,_calloc_r -Wl,--wrap,exit -Wl,--wrap,atexit -Wl,-n -T /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/extern/mbed-os/targets/TARGET_STM/TARGET_STM32F7/TARGET_STM32F769xI/mbed-stm32f769xi.link_script.ld spikes/mbed_blinky/CMakeFiles/spike_mbed_blinky.dir/main.cpp.obj -o spikes/mbed_blinky/spike_mbed_blinky.elf  libs/Utils/libUtils.a  libs/LogKit/libLogKit.a  libs/HelloWorld/libHelloWorld.a  libs/CriticalSection/libCriticalSection.a  libmbed-os-static.a  libs/ContainerKit/libContainerKit.a  libs/LogKit/libLogKit.a  libs/ContainerKit/libContainerKit.a  libs/Utils/libUtils.a  libs/HelloWorld/libHelloWorld.a  libs/CriticalSection/libCriticalSection.a  libmbed-os-static.a && cd /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky && arm-none-eabi-objcopy -O binary /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.bin && /usr/local/Cellar/cmake/3.22.1/bin/cmake -E echo "-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.bin" && arm-none-eabi-objcopy -O ihex /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.hex && /usr/local/Cellar/cmake/3.22.1/bin/cmake -E echo "-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.hex" && cd /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky && /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 /Users/ladislas/dev/leka/LekaOS/extern/mbed-os/tools/memap.py -t GCC_ARM /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf.map
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-fflush.o): in function `__sflush_r':
fflush.c:(.text.__sflush_r+0x94): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-malloc.o): in function `free':
malloc.c:(.text.free+0x6): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-fclose.o): in function `_fclose_r':
fclose.c:(.text._fclose_r+0x50): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: fclose.c:(.text._fclose_r+0x5e): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: fclose.c:(.text._fclose_r+0xbe): undefined reference to `__wrap__free_r'
collect2: error: ld returned 1 exit status

Any ideas on how to fix?

The reason for this is that STATIC libraries are not usable with Arm Compiler since Arm Compiler lacks the -Wl,--whole-archive linker flag. So, to make something that works with both, you need to use OBJECT libraries.

Can this be made "toolchain" dependent? If we use Arm Compiler, we call add_library with OBJECT and if we use amr-nobe-eabi-gcc, we call it with STATIC?

If so, I can make a PR.

[ladislas]

@multiplemonomials I've tried fixing it by adding the -Wl,--whole-archive option to the link_flags but it created new errors about the C Compiler not able to build a test program, so I added set(CMAKE_TRY_COMPILE_TARGET_TYPE "STATIC_LIBRARY") to the main CMakeLists.txt and now it does not link the executable, it's just hangs forever, takes 100% CPU and 100% RAM and crashes my high end MacBook Pro with 16GB of RAM and a 2,3 GHz 8-Core Intel Core i9... 😂

I'm far from an expert with CMake and compiler settings so I might do something very wrong! Do not hesitate to state the obvious things I might have missed. :)

[multiplemonomials]

Hmm, that is the right compiler option but you want to use it only when linking other things to the distro. What I did in mbed-cmake was create a wrapper target for mbed OS. For example, if you had mbed_create_distro(mbed_for_my_app ...), you would then add

target_link_libraries(mbed_for_my_app_wrapped INTERFACE
	-Wl,--whole-archive
	mbed_for_my_app
	-Wl,--no-whole-archive)

Then, executables will link to mbed_for_my_app_wrapped.

I will say, I'm not totally sure how this caused a crash though! I've never seen anything like that.

[ladislas]

@multiplemonomials so the wrapper worked! Thanks a lot!

It hangs on the first executable but then the rest is quite okay.

But the output doesn't look good, I've got this:

[1163/1166] Linking CXX executable spikes/lk_file_manager/spike_lk_file_manager.elf
-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/lk_file_manager/spike_lk_file_manager.bin
-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/lk_file_manager/spike_lk_file_manager.hex
Unknown object name found in GCC map file: libmbed-os-static-raw.a(startup_stm32f769xx.S.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(irq_cm4f.S.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_delay.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_delay.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
[...]
Unknown object name found in GCC map file: libmbed-os-static-raw.a(us_ticker.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(QSPIFBlockDevice.cpp.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(BLEInstanceBaseImpl.cpp.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(GattServerImpl.cpp.obj)
| Module          |         .text |       .data |        .bss |
|-----------------|---------------|-------------|-------------|
| [fill]          |     116(+116) |       8(+8) |     24(+24) |
| [lib]/CoreHTS.a |   1520(+1520) |       0(+0) |       0(+0) |
| [lib]/CoreI2C.a |       52(+52) |       0(+0) |       0(+0) |
| [lib]/Utils.a   |     140(+140) |       0(+0) |       0(+0) |
| [lib]/c.a       | 25296(+25296) | 2472(+2472) |     58(+58) |
| [lib]/gcc.a     |   3116(+3116) |       0(+0) |       0(+0) |
| [lib]/misc      |     188(+188) |       4(+4) |     28(+28) |
| [lib]/nosys.a   |       32(+32) |       0(+0) |       0(+0) |
| [misc]          | 29708(+29708) |   444(+444) | 8114(+8114) |
| main.cpp.obj    |       88(+88) |       0(+0) |       0(+0) |
| Subtotals       | 60256(+60256) | 2928(+2928) | 8224(+8224) |
Total Static RAM memory (data + bss): 11152(+11152) bytes
Total Flash memory (text + data): 63184(+63184) bytes

for hundreds of lines for each executable.

Also everything is in [misc] now.

[multiplemonomials]

Yeah, that happens, I get that issue with mbed-cmake too. I've been meaning to dive into memap.py and see why it can't handle objects inside static libraries.

[ladislas]

Yes, I figured it was a memap.py issue as well.

is there a way to know what cmake is doing when it hangs on the first executable? The first one is random, it changes every time I recompile, but it always hangs.

[multiplemonomials]

Does it hang before it starts compiling any source files, or during the link operation? I normally see a moderate (5-20 sec) hang before it compiles any source files (only the first time a new binary dir is used) due to the CMake dependency scanner running.

[0xc0170]

Right now, I'm not sure what to do. Should I keep fighting the tools or keep our current setup using USCRPL/mbed-cmake?

I wish the things were different and the migration would be much smoother.

@ladislas if it is causing that much trouble, I would stay with the setup you have working and continue using it. Thanks for sharing the feedback.