Misleading error reporting when a CHECK calls a REQUIRE

[brevzin]

Description

If a CHECK() assertion internally calls a function that does a REQUIRE() assertion that fails, we get a very misleading message.

Steps to reproduce

Here is the complete test:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

int foo(int i) {
    REQUIRE(i > 10);
    return 42;
}

TEST_CASE("a", "[a]") {
    CHECK(foo(2) == 2);
}

Compiled with -std=c++11 and no other flags, run with no arguments. This emits:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a is a Catch v2.2.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
a
-------------------------------------------------------------------------------
test.cxx:9
...............................................................................

test.cxx:5: FAILED:
  REQUIRE( i > 10 )
with expansion:
  2 > 10

test.cxx:10: FAILED:
  CHECK( foo(2) == 2 )
due to unexpected exception with message:
  Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE

===============================================================================
test cases: 1 | 1 failed
assertions: 2 | 2 failed

Note the message about CATCH_CONFIG_FAST_COMPILE - which isn't actually defined here. This is super confusing.

Extra information

• Catch version: v2.2.2, Generated: 2018-04-06 12:05:03.186665
• Operating System: Cygwin on Windows
• Compiler+version: gcc 6.4.0

[onqtam]

what an interesting find! the doctest testing framework has the same issue - thanks for reporting!

[horenmar]

Yeah, I can see how that output would be very confusing.

[AlexisWilke]

I ran in the same problem. CATCH_CONFIG_DISABLE_EXCEPTIONS is not set and I get the same error message. I have 2.8.0 at the moment.

[slci]

Hi,

I've found a very similar case, that result in exactly the same misleading error report.
I'm using Catch2 together with Trompeloeil for unit testing. When I set an expectation that some method is forbidden to call as (shown in the example below) and then I check whether some expression (that should throw), calls the method, I get the aforementioned misleading error:

  CHECK_THROWS_AS( TestObject(d, true), std::runtime_error )
due to unexpected exception with message:
  Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE

The root cause is the same, my test runner calls CHECK inside CHECK_THROWS ( Use Trompeloeil with Catch2 )

Example test to reproduce

struct Dependency
{
    MAKE_MOCK0(Method, void());
};

struct TestObject
{
    TestObject(Dependency& d, bool condition)
    {
        if (condition)
        {
            // Should throw, but there's a bug...
            // throw std::invalid_argument{"invalid condition"};
        }
        d.Method();
    }
};

TEST_CASE("exception")
{
    Dependency d{};
    FORBID_CALL(d, Method());

    CHECK_THROWS_AS(TestObject(d, true), std::runtime_error);
}

Tested with Catch v2.13.0

[CelticMinstrel]

I'm also getting this, in my case a REQUIRE() that calls a FAIL() - I configured my logger to fail the test if anything unexpected is outputted to the error stream, so the REQUIRE() assertion calls some code that logs to the error stream which then calls FAIL(). Using v2.7.0 though.

[Sidelobe]

Is there any workaround for this? Other than disabling exceptions...

It keeps randomly popping up in my codebase... sometimes on Linux builds, sometimes on Windows builds only. Happens only on the build server (azure devops).

I have a custom assertion handler that can be configured to throw.
REQUIRE_THROWS(MY_ASSERT(false));

So don't even have a nested CHECK/REQUIRE...

[diablodale]

reproduced today with Catch v2.13.7. CATCH_CONFIG_FAST_COMPILE is not defined

..\tests\src\hardware_cputemp_perf.cpp(281): FAILED:
  REQUIRE( std::get<bool>(targetDevice) )
with expansion:
  false
with message:
  <INFO stuff>

..\tests\src\hardware_cputemp_perf.cpp(281): FAILED:
  REQUIRE_NOTHROW( [&]() { const auto targetDevice = getAvailableDevice(std::chrono::seconds(protocol == X_LINK_TCP_IP ? 60 : 15), protocol); do { ; Catch::AssertionHandler catchAssertionHandler( "REQUIRE"_catch_sr, ::Catch::SourceLineInfo( "..\\tests\\src\\hardware_cputemp_perf.cpp", static_cast<std::size_t>( 281 ) ), "std::get<bool>(targetDevice)", Catch::ResultDisposition::Normal ); try { __pragma( warning(push) ) catchAssertionHandler.handleExpr( Catch::Decomposer() <= std::get<bool>(targetDevice) ); __pragma( warning(pop) ) } catch(...) { catchAssertionHandler.handleUnexpectedInflightException(); } catchAssertionHandler.complete(); } while( (void)0, (false) && static_cast<bool>( !!(std::get<bool>(targetDevice)) ) ); device = std::make_unique<dai::Device>(p, std::get<dai::DeviceInfo>(targetDevice)); }() )
due to unexpected exception with messages:
  <INFO stuff>
  Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE

with code

INFO("<INFO stuff>");
REQUIRE_NOTHROW([&]() {
    const auto targetDevice = getAvailableDevice(std::chrono::seconds(protocol == X_LINK_TCP_IP ? 60 : 15), protocol);
    REQUIRE(std::get<bool>(targetDevice));
    device = std::make_unique<dai::Device>(p, std::get<dai::DeviceInfo>(targetDevice));
}());

[PKua007]

Any update on the issue? Today I ran into the same problem as @slci with trompeloeil mocking framework and was scratching my head what is going on. Tested on 2.13.10.

[Sidelobe]

Is there any workaround for this? Other than disabling exceptions...

It keeps randomly popping up in my codebase... sometimes on Linux builds, sometimes on Windows builds only. Happens only on the build server (azure devops).

I have a custom assertion handler that can be configured to throw. REQUIRE_THROWS(MY_ASSERT(false));

So don't even have a nested CHECK/REQUIRE...

Any update or workaround for this?
It keeps randomly re-appearing in my CI builds .. this time on Windows builds (x64 & arm64) , but only in Release

[eramongodb]

This issue occurs for any Catch exception thrown during the evaluation of any assertion macro (i.e. CHECK(), REQUIRE(), CHECK_NOTHROW(), CHECK_THROWS_AS(), etc.) which does not explicitly test for the specific Catch exception being thrown:

#include <catch2/catch_test_macros.hpp>

void fn() { FAIL("Throw a Catch::TestFailureException"); } // Line 3

TEST_CASE("example") {
  SECTION("misleading error message") { // Line 6
    CHECK_NOTHROW(fn());                // Line 7
  }

  SECTION("no misleading error message") {              // Line 10
    CHECK_THROWS_AS(fn(), Catch::TestFailureException); // Line 11
  }
}

-------------------------------------------------------------------------------
example
  misleading error message
-------------------------------------------------------------------------------
main.cpp:6
...............................................................................

main.cpp:3: FAILED:
explicitly with message:
  Throw a Catch::TestFailureException

main.cpp:7: FAILED:
  CHECK_NOTHROW( fn() )
due to unexpected exception with message:
  Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE

-------------------------------------------------------------------------------
example
  no misleading error message
-------------------------------------------------------------------------------
main.cpp:10
...............................................................................

main.cpp:3: FAILED:
explicitly with message:
  Throw a Catch::TestFailureException

===============================================================================
test cases: 1 | 1 failed
assertions: 4 | 1 passed | 3 failed

for any Catch exception

Yes, this includes Catch::TestSkipException as thrown by a call to SKIP():

#include <catch2/catch_test_macros.hpp>

void fn() { SKIP("This test should be skipped; it should not fail!"); } // Line 3

TEST_CASE("example") { // Line 5
  CHECK((fn(), true)); // Line 6
}

-------------------------------------------------------------------------------
example
-------------------------------------------------------------------------------
main.cpp:5
...............................................................................

main.cpp:3: SKIPPED:
explicitly with message:
  This test should be skipped; it should not fail!

main.cpp:6: FAILED:
  CHECK( (fn(), true) )
due to unexpected exception with message:
  Exception translation was disabled by CATCH_CONFIG_FAST_COMPILE

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

Unless nested test macros are not intended to be supported, this is arguably incorrect test suite behavior, not just a misleading error message.