doctest in embedded

[GregKon]

Dear

We are testing doctest in embedded environment, some small success so far,
However, it is quite big footprint for small devices, we use dual targeting first PC host when footprint do not matter at all and verbose information is useful, than test on target where footprint matter but verbose information do not.
So on embedded target what I am interesting in is only PASS or FAIL for all test.
Is there option to disable serializing or formatting output option (xml) what probably increase footprint ?

/Greg

[onqtam]

Well one option is to define your own reporter (just leave all the methods completely empty) and use it with the --reporters=my_reporter command line option:
https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md

Is this good enough?

[GregKon]

Hi
I think it is misunderstand, I have edit my issue and bold what is a case.

1. I mean footprint is very large
2. I suppose that xml formatting (or maybe other feature) is suppose to make footprint big.
3. Question is: is it possible to easy disable some more feature (during compile time), specially features responbisble for big footprint.

When you testing on embedded, it is last stage of testing procedure (after testing on PC where footprint does not matter), and only what is interesting is: if assert fails and in which line and file. That is in general all what you need (well, mostly).

[UnePierre]

Everthing is visible to your compiler and it should optimize the unused symbols away.

Unfortunately, it cannot detect, that you will never use the two built-in reporters, because the compiler does not know your comandline arguments.

I'd suggest some DOCTEST_CONFIG_DISABLE_XML_REPORTER (plus DOCTEST_CONFIG_DISABLE_CONSOLE_REPORTER) config flags. Could you please test, whether that would lower your memory footprint accordingly?

On my embedded platform (Cortex-M3 with IAR EWARM), just #include <stdio> costs about 16KiB code memory.

Or you could implement the macros that you use to almost nothing, except #define CHECK(x) exit(__LINE__) or alike, basically writing a very thin doctest yourself.

[onqtam]

Why is there a big footprint? Is memory use an issue? Or is processing time because of all the printing to stdout? Or both?

[UnePierre]

The "large footprint" @GregKon means, is probably the code memory -- I guess.

It is a common issue when using C++ with the STL, because all (used) template specializations produce slightly different assembly with little to no re-use. Of course this is good for performance on a PC, where binaries are on large hard drives, but it means that any embedded µC's ROM gets filled pretty fast.
And because the purchase department usually doesn't allow for ample code ROM, demanding a lot for pretty-printing during tests on the final HW is a no-go.
Having an easy way to configure the printing out of the test framework would be a nice solution!

[onqtam]

@UnePierre the printing can be removed by using a stub reporter as I mentioned before (which will make the runtime faster because of no printing to stdout). However, there will still be virtual calls to the reporter on each assert, and also that won't have any effect on the compiled code - all template specializations will be the same and still present in the binary. I don't think doctest will be able to provide anything lighter than that...

[GregKon]

Hi

I do also use IAR for ARM. some benchmark:

Cortex M0, IAR 8.50 Optimalziation highest - size
library: Full, semihosted
code:
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN #include <doctest/doctest.h>
footprint ~ 0x14000
Add test case: + 0x300 bytes
Add
CHECK(1 ==1) + 0x14C0 bytes

indeed assertion may be changed. but this is not big issue
I think "reporter" has also big impact.

I can live with it, there is still some space for test.
However it is still a lot for low resources uC.

[UnePierre]

Then maybe go for replacing the CHECK() and other macros in your trampoline header!
Don't forget to re-enter SUBCASE()s somehow!

[UnePierre]

Yet another solution, that needs no modification to doctest:
If you run your tests in a simulator, you can use another linker file in a special compilation configuration and virtually allow a lot more memory. If course, such binary can never run on the target HW, only the targer simulator. Still, it should be instruction accurate.
You get a debuggable run in the target's assembly, of the entire test collection, with pretty reporting.

[UnePierre]

I do also use IAR for ARM.

For IAR this might be handy as well: #369 (comment)

[GregKon]

Yet another solution, that needs no modification to doctest:
If you run your tests in a simulator, you can use another linker file in a special compilation configuration and virtually allow a lot more memory. If course, such binary can never run on the target HW, only the targer simulator. Still, it should be instruction accurate.
You get a debuggable run in the target's assembly, of the entire test collection, with pretty reporting.

[GregKon]

Yet another solution, that needs no modification to doctest:
If you run your tests in a simulator, you can use another linker file in a special compilation

this is easy an no problem at all.

[GregKon]

I do also use IAR for ARM.

For IAR this might be handy as well: #369 (comment)

does it decrease footprint?

[UnePierre]

I do also use IAR for ARM.

For IAR this might be handy as well: #369 (comment)

does it decrease footprint?

No. It might even increase -- so use it with your simulator.
Clickable links are really handy!

[onqtam]

I'm closing this as it seems there's nothing more to contribute to this issue - feel free to comment if that's not the case.