Debugging with Unit Testing

[tehnorm]

It is often handy to be able to set a breakpoint while debugging a unit test. This is useful when something is hard to track down - being able to see the whole stack/call chain is very helpful.

[giova-fr]

In case this feature request looks to complicate, it would be nice to add an API command to print onto Unit Test Terminal. This means no breakpoint, no stack view, but at least we can track some variables while running tests.

[ivankravets]

it would be nice to add an API command to print onto Unit Test Terminal.

Do you mean _MESSAGE?

For example, TEST_ASSERT_EQUAL_STRING_MESSAGE(expected, actual, message).

[lildent]

any idea when this feature will be added ?

[ivankravets]

Sorry for the delay on this issue. There are some difficulties to implement it. PlatformIO supports multi-level tests with own main functions. You can not debug all of them within the one debugging session. However, we have some interesting solution - adding new option named debug_test = yes|no (Bool). Default value is no.

So, you will be able to declare debug environment and configure which test to debug. For example:

[env:debug]
platform = x
framework = y
board = z
build_type = debug
debug_test = yes
test_filter = sometestname

Will it work?

[lgLindstrom]

what is current status on this?

I am trying to run:

pio test -e debug

Response is:
Warning! Ignore unknown configuration option debug_test in section [env:debug]

So I guess it is not implemented ???
It would be very useful if working.

[ivankravets]

Sorry for the delay. We planned this feature to upcoming PIO Core 4.4.0.

[madsdyd]

I too, would like this feature, or some workaround, for using the debugger while running a test. Seems sort of a strange limitation that you can not debug a test! :-)

[ASL07]

@madsdyd Maybe a solution would be to use cortex debug if you are using VSCode and an ARM mcu (https://marketplace.visualstudio.com/items?itemName=marus25.cortex-debug) You can compile the tests .elf file with platformio, then flash it and debug it using cortex debug (you will need a launch.json file)
Platformio connects to the device's serial port when running the tests in order to read the test result though, so this may cause problems

[leroyle]

As this issue is still open I assume it's not yet fixed. So, a couple of workarounds to suggest.

• First My env
  Platformio core: 4.3.3
  Home: 3.2.2

• Project Env:
  platform = nordicnrf51
  board = waveshare_ble400
  framework = arduino
  build_type = debug
  upload_protocol = stlink
  debug_tool = stlink

Work around 1: Command Line
Use Platformio to build and upload to your board the unit test image as normal.

1. in window 1: fire up openocd and connect to your board ( you will need the appropriate config files for openocd)
2. in window 2: fire up arm-none-eabi-gdb and connect to the remote target. On linux that would be something like:
   -> export PATH=~/.platformio/packages/toolchain-gccarmnoneeabi/bin:$PATH
   -> arm-none-eabi-gdb
   gdb> target extended-remote localhost:3333

From here on out you should be able to use gdb/openocd to debug the test case.
(this video may help with the use of openocd, go to about 19:00 to get to the openocd discussion: https://www.youtube.com/watch?v=4RjEmcT6JpM)

Work around 2: This uses VSCode as the debugger, much easier/nicer

1. You will need to add a new variable to your target definition within your projects platformio.ini file.
   Platformio will not allow willy nilly variables but fortunately the developers have made allowances
   by allowing custom variables prefixed with "custom_".
   So, I've added one called "custom_debugtest" as:
   custom_debugtest = true

   Set this variable to true if doing debug of unit test builds, anything else or non existing if
   debugging normal builds.

2. now for this to be acted on you need you will need to change some platformio python code. So, edit:
   ~.platformio/penv/lib/python3.6/site-packages/platformio/commands/debug/command.py

2a: just prior to about line 136 in command.py where is says;
if rebuild_prog:

add the following "BEFORE" this line. This will read the new variable and disable the normal rebuild of the non test case image: NOTE: be careful of the python required indention, copy and paste
probably will mess that up. The important thing here is to set "rebuild_prog = False".

   debug_test = env_options.get("custom_debugtest");

    if debug_test is not None:
        if debug_test == "true":
            rebuild_prog = False;
            click.echo(
                    helpers.escape_gdbmi_stream(
                    "~", "Setting for debug of test build"
           ),
                nl=False,
            )

3. Once you have this in place, build and upload the tests to the target board as normal with platformio. Once the test has uploaded

• switch platformio to debug mode
• just to the right of the run/green arrow icon, select the drop down and select the (skip Pre-Debug)
  option (this is important to prevent a rebuild of non unit test images)
• then click on the green arrow to start the debug
  you might need to terminate the existing execution of the unit test. I need to do this in the
  platformio "Terminal" view via a CTRL-C when re-running tests but that could be just my board.

4. If all goes well you should "not" see the target being rebuilt as it normally would. You should see
   "Setting for debug of test build" dumped somewhere in the debug output window. And the debugger
   should stop in main.

From here you should be able to debug normally using VSCode

Note: This has not been approved by Platformio developers, may not work with other release versions, you may get some unusual notifications popup, and of course your mileage may vary,

Hopefully I've not missed a step here and hopefully this is helpful.

[ivankravets]

See updated docs https://docs.platformio.org/en/latest/plus/debugging.html#debug-unit-tests

P.S: @tehnorm, sorry for the delay, and thanks for supporting PlatformIO! 🙏

[aleuarore]

Is there anyway to debug unit tests in native using the the VSCode IDE? I mean using the play button instead of the CLI.
I tried modifying the launch.json file and for simple project it worked, but we should't modify it manually when we use platformio.

I will set the code in a STM32H743, but I have some pure cpp classes and functions in lib that I want to test isolated. Some of the tests are failing and that's why I would like to debug them in native.

platformio.ini

[env:nucleo_h743zi]
platform = ststm32
board = nucleo_h743zi
framework = cmsis

[env:native]
platform = native
lib_deps =
dependency
dependency2
build_type = debug
debug_test = test_debug
debug_init_break = tbreak loop