Count points based on the number of passed/failed cases?

[xrb936]

Hi guys, I am now trying to use doctest to grade c++ code assignments, and this is my first time to use doctest. Just one question, is there any way that I can count the points based on how many testcases are passed? Or is there any way that I can perform like IF test case passed, THEN grade ++, something like that?

Thanks a lot!

[onqtam]

perhaps these 2 issues are relevant: #200 #344

[xrb936]

perhaps these 2 issues are relevant: #200 #344

Thank you so much! I will take a look!

[xrb936]

perhaps these 2 issues are relevant: #200 #344

Hi again, I am trying to use understand how to use reporter but when I used REGISTER_REPORTER("my_xml", 1, MyXmlReporter); it shows me an error: Typedef redefinition with different types ('int' vs 'int ()'). I am simply using the example code from the repo.

[onqtam]

do you mean that the code here doesn't compile? I just tested it with GCC and it compiled...
https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md

Also the example compiles with a bunch of compilers on different OS-es:
https://github.com/onqtam/doctest/blob/master/examples/all_features/reporters_and_listeners.cpp

[xrb936]

do you mean that the code here doesn't compile? I just tested it with GCC and it compiled...
https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md

Also the example compiles with a bunch of compilers on different OS-es:
https://github.com/onqtam/doctest/blob/master/examples/all_features/reporters_and_listeners.cpp

Yep, unfortunately both of them are not working for me. Here is the error message:

`/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
CMakeFiles/test.dir/main.cpp.o: In function `__static_initialization_and_destruction_0(int, int)':
/home/xrb936/Desktop/test/doctest.h:2585: undefined reference to `doctest::detail::TestSuite::operator*(char const*)'
/home/xrb936/Desktop/test/doctest.h:2585: undefined reference to `doctest::detail::setTestSuite(doctest::detail::TestSuite const&)'
CMakeFiles/test.dir/main.cpp.o: In function `doctest::IReporter::IReporter()':
/home/xrb936/Desktop/test/doctest.h:1661: undefined reference to `vtable for doctest::IReporter'
CMakeFiles/test.dir/main.cpp.o: In function `int doctest::registerReporter<MyXmlReporter>(char const*, int, bool)':
/home/xrb936/Desktop/test/doctest.h:1724: undefined reference to `doctest::detail::registerReporterImpl(char const*, int, doctest::IReporter* (*)(doctest::ContextOptions const&), bool)'
CMakeFiles/test.dir/main.cpp.o: In function `MyXmlReporter::~MyXmlReporter()':
/home/xrb936/Desktop/test/main.cpp:7: undefined reference to `doctest::IReporter::~IReporter()'
CMakeFiles/test.dir/main.cpp.o:(.data.rel.ro._ZTI13MyXmlReporter[_ZTI13MyXmlReporter]+0x10): undefined reference to `typeinfo for doctest::IReporter'
collect2: error: ld returned 1 exit status
CMakeFiles/test.dir/build.make:83: recipe for target 'test' failed
make[3]: *** [test] Error 1
CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/test.dir/all' failed
make[2]: *** [CMakeFiles/test.dir/all] Error 2
CMakeFiles/Makefile2:82: recipe for target 'CMakeFiles/test.dir/rule' failed
make[1]: *** [CMakeFiles/test.dir/rule] Error 2
Makefile:118: recipe for target 'test' failed
make: *** [test] Error 2

I am using CLion on Ubuntu.

[onqtam]

Well this is different from what you previously said (Typedef redefinition with different types ('int' vs 'int ()') - right now you are missing the implementation of doctest - see here: https://github.com/onqtam/doctest/blob/master/doc/markdown/main.md

You need at least 1 .cpp file where you define DOCTEST_CONFIG_IMPLEMENT before including the header to compile in the test runner

[xrb936]

Sorry I forgot the #define DOCTEST_CONFIG_IMPLEMENT.
However I still got an error with REGISTER_REPORTER. It still says Typedef redefinition with different types ('int' vs 'int ()').

[onqtam]

well in that case I need to know which version of GCC (or clang) you are using - CLion is just an IDE afaik... sounds really strange to me... can you give more information? Are you including other headers? perhaps there is some preprocessor macro problem?

[xrb936]

GCC version is 7.5.0.
I am just using the latest release of the doctest.h and the code from https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md with #define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN.
Thanks!

The interesting thing is REGISTER_LISTENER is fine for me, tho.

[onqtam]

well the difference between registering a reporter or a listener is a boolean true/false within the macros...
the latest doctest version 2.3.8 (released 3 weeks ago) was build on CI with a plethora of compilers - including gcc 7.5.0 - see here: https://travis-ci.org/github/onqtam/doctest/jobs/688113670#L703 (line 703) - without a problem...

One thing I would suggest is that you run the tests of doctest itself - just call cmake in the root of the repo and then execute make and ctest --output-on-failure and tell me what happened.

Apart from that I'm clueless - you'll have to debug this locally - there must be something different in your setup.

[onqtam]

I just looked into the code - for each of the reporter/listener macros I use this at the end: typedef int DOCTEST_ANONYMOUS(_DOCTEST_ANON_FOR_SEMICOLON_) - that basically introduces a random typedef in global scope that forces you to write a semicolon after the macro. My best guess is that perhaps when the 2 macros are present (for reporter & a listener) the DOCTEST_ANONYMOUS macro doesn't work properly and generates 2 identical names - meaning that perhaps __COUNTER__ doesn't work on your machine for some reason (it's defined like this: #define DOCTEST_ANONYMOUS(x) DOCTEST_CAT(x, __COUNTER__)).

Is the typedef error a compilation error from GCC or is it an error from the intellisense of CLion - within the IDE?

[xrb936]

I just looked into the code - for each of the reporter/listener macros I use this at the end: typedef int DOCTEST_ANONYMOUS(_DOCTEST_ANON_FOR_SEMICOLON_) - that basically introduces a random typedef in global scope that forces you to write a semicolon after the macro. My best guess is that perhaps when the 2 macros are present (for reporter & a listener) the DOCTEST_ANONYMOUS macro doesn't work properly and generates 2 identical names - meaning that perhaps __COUNTER__ doesn't work on your machine for some reason (it's defined like this: #define DOCTEST_ANONYMOUS(x) DOCTEST_CAT(x, __COUNTER__)).

Is the typedef error a compilation error from GCC or is it an error from the intellisense of CLion - within the IDE?

The error is showed in the IDE and I can still compile the code and run it. However, the reporter is not working when I use the reporter filter to define it.

[onqtam]

Well in that case that's a problem with CLion - for not properly handling __COUNTER__ - perhaps it should be logged in their bug tracker...

how are you using the reporter filter?

[xrb936]

Well in that case that's a problem with CLion - for not properly handling __COUNTER__ - perhaps it should be logged in their bug tracker...

how are you using the reporter filter?

By context.addFilter("reporters", "myreporter");

[onqtam]

shouldn't it be "my_xml"?

[xrb936]

shouldn't it be "my_xml"?

Yeah it was, but I changed the name.

[onqtam]

However, the reporter is not working when I use the reporter filter to define it.

what do you mean by not working? I just tested compiling the reporter code from the repo and explicitly changing the reporter using context.addFilter("reporters", "my_xml"); and it worked (by which I mean suddenly I stopped getting output from the console reporter ===> the empty XML reporter was enabled!)

The reporter code in the example/docs is a stub - it doesn't do anything - it's up to you to implement whatever you need. For that you might look into the doctest header itself to figure things out... I guess one option would be to inherit from one of the 2 reporters in doctest and just override a specific method so that you don't have to redefine everything

[xrb936]

However, the reporter is not working when I use the reporter filter to define it.

what do you mean by not working? I just tested compiling the reporter code from the repo and explicitly changing the reporter using context.addFilter("reporters", "my_xml"); and it worked (by which I mean suddenly I stopped getting output from the console reporter ===> the empty XML reporter was enabled!)

The reporter code in the example/docs is a stub - it doesn't do anything - it's up to you to implement whatever you need. For that you might look into the doctest header itself to figure things out... I guess one option would be to inherit from one of the 2 reporters in doctest and just override a specific method so that you don't have to redefine everything

I am trying to use the code in https://github.com/onqtam/doctest/issues/344

[onqtam]

well I just tested that code and for the following asserts:

TEST_CASE("tests 1") {
    CHECK(1 == 1);
    CHECK(1 == 1);
    CHECK(1 == 2);
}

TEST_CASE("tests 2") {
    CHECK(1 == 1);
}

I got the following output:

[doctest] doctest version is "2.3.8"
[doctest] run with "--help" for options
===============================================================================
E:\doctest\scripts\playground\test.cpp:8:
TEST CASE:  tests 1

E:\doctest\scripts\playground\test.cpp:11: ERROR: CHECK( 1 == 2 ) is NOT correct
!
  values: CHECK( 1 == 2 )

===============================================================================
[doctest] test cases:      2 |      1 passed |      1 failed |      0 skipped
[doctest] assertions:      4 |      3 passed |      1 failed |
[doctest] Status: FAILURE!
Grade: 75

It printed 75 which means 75% correct asserts - that's what his code does. I'm not going to be able to help any further than this...

[xrb936]

well I just tested that code and for the following asserts:

TEST_CASE("tests 1") {
    CHECK(1 == 1);
    CHECK(1 == 1);
    CHECK(1 == 2);
}

TEST_CASE("tests 2") {
    CHECK(1 == 1);
}

I got the following output:

[doctest] doctest version is "2.3.8"
[doctest] run with "--help" for options
===============================================================================
E:\doctest\scripts\playground\test.cpp:8:
TEST CASE:  tests 1

E:\doctest\scripts\playground\test.cpp:11: ERROR: CHECK( 1 == 2 ) is NOT correct
!
  values: CHECK( 1 == 2 )

===============================================================================
[doctest] test cases:      2 |      1 passed |      1 failed |      0 skipped
[doctest] assertions:      4 |      3 passed |      1 failed |
[doctest] Status: FAILURE!
Grade: 75

It printed 75 which means 75% correct asserts - that's what his code does. I'm not going to be able to help any further than this...

Yep, I know this is how it should output, but I am not able to get the Grade: 75 at the end of the output.

[xrb936]

Let me try to compile it on macOS.

[xrb936]

well I just tested that code and for the following asserts:

TEST_CASE("tests 1") {
    CHECK(1 == 1);
    CHECK(1 == 1);
    CHECK(1 == 2);
}

TEST_CASE("tests 2") {
    CHECK(1 == 1);
}

I got the following output:

[doctest] doctest version is "2.3.8"
[doctest] run with "--help" for options
===============================================================================
E:\doctest\scripts\playground\test.cpp:8:
TEST CASE:  tests 1

E:\doctest\scripts\playground\test.cpp:11: ERROR: CHECK( 1 == 2 ) is NOT correct
!
  values: CHECK( 1 == 2 )

===============================================================================
[doctest] test cases:      2 |      1 passed |      1 failed |      0 skipped
[doctest] assertions:      4 |      3 passed |      1 failed |
[doctest] Status: FAILURE!
Grade: 75

It printed 75 which means 75% correct asserts - that's what his code does. I'm not going to be able to help any further than this...

It is working now, thank you so much for your help.
Another question, is there any way to let the test case ignore the error such as Segmentation fault and continue with other test case?
Thanks again.

[onqtam]

I wouldn't recommend trying to recover after a segmentation fault... the next tests might crash as well because of corruption from the previous one.

sounds to me you need something more advanced than a testing framework - perhaps a judge system which has sandboxed execution - protecting from crashes, exploits, measuring the time, etc. - there are many such systems out there which you can setup on a server

[xrb936]

I wouldn't recommend trying to recover after a segmentation fault... the next tests might crash as well because of corruption from the previous one.

sounds to me you need something more advanced than a testing framework - perhaps a judge system which has sandboxed execution - protecting from crashes, exploits, measuring the time, etc. - there are many such systems out there which you can setup on a server

Thank you so much for your suggestion and all the help! I will take a look at it later on. For now, everything is working properly.