- Instrumentation for testing
There are many legacy enterprise applications that were written without automated unit tests. It is often very difficult to maintain and modify such code, since we cannot verify the changes. A frequently used approach in this situation is to write additional tests without modifying the original source code. There are several techniques to do this (Alternatives). However, these techniques have their own limitations and disadvantages.
My aim is to provide an alternative technique without those limitations. I used the Clang compiler sanitizer infrastructure to implement testing specific instrumentation (prototype). The instrumentation makes it possible to replace any C/C++ function with a corresponding test double function.
The below examples are actually working code extracts of the tests from this repository.
// unit_under_test.hpp
template <typename T>
T FunTemp(T t) {
return t;
}
inline int foo(int p) {
return FunTemp(p);
}
// test.cpp
TEST_F(FooFixture, CallFunT) {
SUBSTITUTE(&FunTemp<int>, &fake_FunTemp);
int p = 13;
auto res = foo(p);
EXPECT_EQ(res, 39);
}// unit_under_test.hpp
template <typename T>
struct TemplateS {
int foo(int p) { return bar(p); }
int bar(int p) { return p; }
};
// test.cpp
int fake_bar_mem_fun(TemplateS<int>* self, int p) { return p * 3; }
TEST_F(FooFixture, ClassT) {
SUBSTITUTE(&TemplateS<int>::bar, &fake_bar_mem_fun);
TemplateS<int> t;
auto res = t.foo(13);
EXPECT_EQ(res, 39);
}Consider the following concurrent Entity:
// Entity.hpp
class Entity {
public:
int process(int i) const;
void add(int i);
private:
std::vector<int> v;
mutable std::mutex m;
};
// Entity.cpp
int Entity::process(int i) const {
std::unique_lock<std::mutex> lock{m, std::try_to_lock};
if (lock.owns_lock()) {
auto result = std::accumulate(v.begin(), v.end(), i);
return result;
} else {
return -1;
}
return 0;
}
void Entity::add(int i) {
std::lock_guard<std::mutex> lock{m};
v.push_back(i);
}We can test the behaviour based on whether the lock is already owned by another thread or not:
// test.cpp
#include "Entity.hpp"
bool owns_lock_result;
using Lock = std::unique_lock<std::mutex>;
bool fake_owns_lock(Lock*) { return owns_lock_result; }
TEST_F(FooFixture, MutexTest) {
SUBSTITUTE(&Lock::owns_lock, &fake_owns_lock);
Entity e;
owns_lock_result = false;
EXPECT_EQ(e.process(1), -1);
owns_lock_result = true;
EXPECT_EQ(e.process(1), 1);
}// Fread.hpp
#pragma once
struct FeofE {};
struct FerrorE {};
enum { SIZE = 5 };
double Fread(void);// Fread.cpp
double Fread(void) {
double b[SIZE];
FILE* fp = fopen("test.bin", "rb");
if (!fp) {
throw FerrorE{};
}
size_t ret_code =
fread(b, sizeof *b, SIZE, fp); // reads an array of doubles
double result = 0.0;
if (ret_code == SIZE) {
puts("Array read successfully, contents: ");
for (int n = 0; n < SIZE; ++n) {
printf("%f ", b[n]);
result += b[n];
}
putchar('\n');
} else { // error handling
if (feof(fp)) {
printf("Error reading test.bin: unexpected end of file\n");
throw FeofE{};
} else if (ferror(fp)) {
perror("Error reading test.bin");
throw FerrorE{};
}
}
fclose(fp);
return result;
}// Test.cpp
#include <stdio.h>
#include <cassert>
#include <array>
#include <numeric>
#include "FooFixture.hpp"
#include "Fread.hpp"
namespace t1 {
FILE f;
FILE *fake_fopen(const char *, const char *) { return &f; }
std::array<double, SIZE> a = {1., 2., 3., 4., 5.};
size_t fake_fread(void *buf, size_t, size_t count, FILE *) {
assert(count == SIZE);
double *b = reinterpret_cast<double *>(buf);
for (int i = 0; i < 5; ++i) {
b[i] = a[i];
}
return count;
}
int fake_feof(FILE *) { return 0; }
int fake_ferror(FILE *) { return 0; }
int fake_fclose(FILE *) { return 0; }
} // namespace t1
TEST_F(FooFixture, FreadSumIsOk) {
SUBSTITUTE(&fopen, &t1::fake_fopen);
SUBSTITUTE(&fread, &t1::fake_fread);
SUBSTITUTE(&feof, &t1::fake_feof);
SUBSTITUTE(&ferror, &t1::fake_ferror);
SUBSTITUTE(&fclose, &t1::fake_fclose);
auto res = Fread();
EXPECT_EQ(res, std::accumulate(t1::a.begin(), t1::a.end(), 0.0));
}
namespace t2 {
FILE f;
FILE *fake_fopen(const char *, const char *) { return &f; }
size_t fake_fread(void *buf, size_t, size_t count, FILE *) {
return 0;
}
int fake_feof(FILE *) { return 1; } // indicate EOF !
int fake_ferror(FILE *) { return 0; }
int fake_fclose(FILE *) { return 0; }
} // namespace t2
TEST_F(FooFixture, FreadHandles_feof) {
SUBSTITUTE(&fopen, &t2::fake_fopen);
SUBSTITUTE(&fread, &t2::fake_fread);
SUBSTITUTE(&feof, &t2::fake_feof);
SUBSTITUTE(&ferror, &t2::fake_ferror);
SUBSTITUTE(&fclose, &t2::fake_fclose);
EXPECT_THROW(Fread(), FeofE);
}struct Message {
int type;
std::vector<int> events;
};
void reloadConfig() { /*...*/ }
void processMessage(int msgType) { /*...*/ }
void processEvent(int event) { /*...*/ }
#define REALOAD_TIMEOUT 60
void handleMessage(const Message& msg) {
time_t start = time(NULL);
processMessage(msg.type);
for (int i = 0; i < msg.events.size(); ++i) {
processEvent(msg.events[i]);
}
time_t duration = (time(NULL) - start);
if (duration > REALOAD_TIMEOUT) {
reloadConfig();
}
}
// TEST.cpp
#include <gtest/gtest.h>
#include "hook.hpp"
time_t fake_time(time_t *) {
static int called = 0;
if (called == 0) {
++called;
return 0;
} else { // When called the 2. time, fake the timeout
return REALOAD_TIMEOUT + 1;
}
}
bool reloadConfigCalled = false;
void mock_reloadConfig() { reloadConfigCalled = true; }
TEST(handleMessage, reloadConfig_shall_be_called_on_timeout) {
SUBSTITUTE(&time, &fake_time);
SUBSTITUTE(&reloadConfig, &mock_reloadConfig);
handleMessage(Message());
EXPECT_EQ(reloadConfigCalled, true);
}
Note, actually syscalls are wrapped in libc and the compiler will generate a call to the specific wrapper function. If the compiler would emit inline assembly with the number of the syscall, then we would not be able ro replace the system calls.
Imagine the following little command line parser function:
void parseCommandLineArgs(int argc, char** argv) {
if (argc != 3) {
std::exit(1);
}
if (std::string(argv[1]) != "apple") {
std::exit(2);
}
if (std::string(argv[2]) != "banana") {
std::exit(3);
}
}Instead of death tests we might test like this:
int exit_code = -1;
struct Exc {};
void fake_exit(int ec) {
::exit_code = ec;
throw Exc{};
}
TEST_F(NoReturn, parseCommandLineArgs) {
SUBSTITUTE(&std::exit, &fake_exit);
/// argc mismatch
exit_code = -1;
try {
parseCommandLineArgs(1, nullptr);
} catch (Exc&) {
}
EXPECT_EQ(::exit_code, 1);
/// first param bad
{
char a0[] = "prog";
char a1[] = "asdf";
char a2[] = "asdff";
char* argv[3] = {a0, a1, a2};
exit_code = -1;
try {
parseCommandLineArgs(3, argv);
} catch (Exc&) {
}
EXPECT_EQ(::exit_code, 2);
}
/// second param bad
{
char a0[] = "prog";
char a1[] = "apple";
char a2[] = "asdff";
char* argv[3] = {a0, a1, a2};
exit_code = -1;
try {
parseCommandLineArgs(3, argv);
} catch (Exc&) {
}
EXPECT_EQ(::exit_code, 3);
}
/// all params are good
{
char a0[] = "prog";
char a1[] = "apple";
char a2[] = "banana";
char* argv[3] = {a0, a1, a2};
exit_code = -1;
try {
parseCommandLineArgs(3, argv);
} catch (Exc&) {
}
/// exit_code should have not changed
EXPECT_EQ(::exit_code, -1);
}
}Each and every function call expression is replaced with the following pseudo code (let's suppose, the callee is foo):
char* funptr = __fake_hook(&foo);
if (funptr) {
funptr(args...);
} else {
foo(args...);
}The call to __fake_hook resolves in runtime if we should replace the callee to a test double or not.
If the function does return anything other than void:
char* funptr = __fake_hook(&foo);
using ReturnType = decltype(foo(args...);
ReturnType ret;
if (funptr) {
ret = funptr(args...);
} else {
ret = foo(args...);
}
return ret;Consider the following definition of bar:
int foo(int);
int bar(int p) {
return foo(p);
}The LLVM IR of bar after optimization looks like this:
define i32 @_Z3bari(i32 %p) #0 {
entry:
%call = tail call i32 @_Z3fooi(i32 %p)
ret i32 %call
}
When we enable -fsanitize=mock then the IR has the following form:
define i32 @_Z3bari(i32 %p) #0 {
entry:
%fake_hook_result = tail call i8* @__fake_hook(i8* bitcast (i32 (i32)* @_Z3fooi to i8*))
%0 = icmp eq i8* %fake_hook_result, null
br i1 %0, label %else, label %then
then: ; preds = %entry
%1 = bitcast i8* %fake_hook_result to i32 (i32)*
%subst_fun_result = tail call i32 %1(i32 %p)
br label %cont
else: ; preds = %entry
%call = tail call i32 @_Z3fooi(i32 %p)
br label %cont
cont: ; preds = %else, %then
%call_res.0 = phi i32 [ %subst_fun_result, %then ], [ %call, %else ]
ret i32 %call_res.0
}
Actually, the IR is much more verbose if optimization is not enabled, therefore
it is a good idea to use -O2.
A constexpr function cannot be replaced when it is used in a compile-time expression. However, it can be replaced whenever it is used within a runtime context:
#include "FooFixture.hpp"
namespace {
constexpr int foo(int p) { return p*p; }
int fake_foo(int p) { return p*p*p; }
constexpr int bar(int p) {
return foo(p);
}
} // unnamed
TEST_F(FooFixture, Constexpr) {
SUBSTITUTE(&foo, &fake_foo);
static_assert(foo(2) == 4, "");
EXPECT_EQ(foo(2), 8);
int p = 2;
EXPECT_EQ(foo(p), 8);
static_assert(bar(2) == 4, "");
EXPECT_EQ(bar(2), 8);
EXPECT_EQ(bar(p), 8);
}Virtual functions are problematic, because a pointer-to-member function has a different layout in case of virtual functions than in case of regular member functions. Normally, we would have to get the address by getting the vtable from an object and then getting the proper element of the vtable.
GCC has a construct with which we can get the address without knowing the actually used ABI, but clang does not: https://llvm.org/bugs/show_bug.cgi?id=22121 https://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Bound-member-functions.html
In our patched Clang, we provide a compiler intrinsic which simply returns the
adress of the function which is known statically during compile time. There
is no need to provide an object. The use of the intrinsic is hidden in the
implementation of the SUBSTITUTE macro.
Do a regular clang build, but instead of the original repos clone llvm, clang, libcxx from https://github.com/martong?tab=repositories .
Checkout the branch finstrument_mock on all repos that just have been cloned.
Build the compiler.
# choose a directory where we want to place the source and build
export PROJ=~/finstrument_mock
export COMPILER_DIR=$PROJ/compiler/
export COMPILER_SRC=$COMPILER_DIR/git
export COMPILER_BUILD=$COMPILER_DIR/build
mkdir -p $COMPILER_SRC
cd $COMPILER_SRC
git clone https://github.com/martong/llvm.git
cd llvm
git checkout finstrument_mock
# Only needed on OSX
cd projects
git clone https://github.com/martong/libcxx.git
cd libcxx
git checkout finstrument_mock
cd $COMPILER_SRC/llvm/tools
git clone https://github.com/martong/clang.git
cd clang
git checkout finstrument_mock
# create a directory where we want to place the build
mkdir -p $COMPILER_BUILD
cd $COMPILER_BUILD
cmake $COMPILER_SRC/llvm -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja
# execute tests with lit.py
# tests for this instrumentation
python $COMPILER_SRC/llvm/utils/lit/lit.py -sv --param clang_site_config=$COMPILER_BUILD/tools/clang/test/lit.site.cfg $COMPILER_SRC/llvm/tools/clang/test/CodeGenCXX/InstrumentMock/
# regression tests for code generation for C++
python $COMPILER_SRC/llvm/utils/lit/lit.py -sv --param clang_site_config=$COMPILER_BUILD/tools/clang/test/lit.site.cfg $COMPILER_SRC/llvm/tools/clang/test/CodeGenCXX/
# regression tests for code generation in general
python $COMPILER_SRC/llvm/utils/lit/lit.py -sv --param clang_site_config=$COMPILER_BUILD/tools/clang/test/lit.site.cfg $COMPILER_SRC/llvm/tools/clang/test/CodeGen/
Once the compiler is built then we need to build the runtime library, which is in this repository. In order to build the tests we have to use the use the modified compiler.
mkdir -p $PROJ/rt
cd $PROJ/rt
git clone https://github.com/martong/finstrument_mock.git
git submodule init
git submodule update
cd finstrument_mock
mkdir build
cd build
cmake .. -G Ninja -DCMAKE_CXX_COMPILER=$COMPILER_BUILD/bin/clang++
# build
ninja
# execute tests
ctest
In your project setup the new compiler and link with the library. E.g. in a Makefile:
CXX=/Users/mg/Work/finstrument_mock/compiler/build.debug/bin/clang++
LDFLAGS = $(LDFLAGS) -rpath $PROJ/rt/finstrument_mock/build/compiler-rt/ -L $PROJ/rt/finstrument_mock/build/compiler-rt/ -lmock_san
Set up your project to use the modified compiler and link against the runtime library.
Put the header file of the runtime library into the include path of your project.
Use the -fno-inline-functions -fsanitize=mock switches to instrument all call expressions.
Test setup is pretty straightforward:
Use _clear_function_substitutions to clear the substitutions.
In C use _substitute_function to replace functions.
In C++ use SUBSTITUTE(&from, &to) to replace functions.
In case of free functions, the signature of from and to must be identical.
Unfortunately we don't assert on this currently (see future work).
In case of member functions the test double's signature must have the same signature, except the first argument. The first argument must be a pointer to the type whose member we are replacing.
Replace the virtual function in the base class does not imply that the function is replaced in the derived class. The replaced function will be called only if the object's dynamic type is the base class type. Similarly, replace the virtual function in the derived class has no impact on the base class. The replaced function will be called only if the object's dynamic type is the derived class type.
namespace {
namespace Basic {
enum class Values : int { B, D, FB, FD };
struct B {
virtual Values foo(int p) { return Values::B; }
};
struct D : B {
virtual Values foo(int p) override { return Values::D; }
};
Values B_fake_foo(B*, int p) { return Values::FB; }
Values D_fake_foo(D*, int p) { return Values::FD; }
} // namespace Basic
} // namespace unnamed
// Replace the virtual function in the base class
TEST_F(DynamicTypeWithFunctionId, Basic_B) {
using Basic::B;
using Basic::D;
using Basic::B_fake_foo;
using Basic::Values;
SUBSTITUTE(B::foo, B_fake_foo);
{
B* b0 = new B;
EXPECT_EQ(b0->foo(1), Values::FB);
}
{
B* b0 = new D;
EXPECT_EQ(b0->foo(1), Values::D);
}
}
// Replace the virtual function in the derived class
TEST_F(DynamicTypeWithFunctionId, Basic_D) {
using Basic::B;
using Basic::D;
using Basic::D_fake_foo;
using Basic::Values;
SUBSTITUTE(D::foo, D_fake_foo);
{
B* b0 = new B;
EXPECT_EQ(b0->foo(1), Values::B);
}
{
B* b0 = new D;
EXPECT_EQ(b0->foo(1), Values::FD);
}
}We can replace overloaded functions by passing the function signature as an
argument to the SUBSTITUTE macro.
int foo(int) { return 1; }
int foo(double) { return 2; }
int foo(int, int) { return 3; }
int fake_foo_i(int) { return 11; }
int fake_foo_d(double) { return 22; }
int fake_foo_ii(int, int) { return 33; }
TEST_F(Overload, freeFunction) {
SUBSTITUTE(int(int), foo, fake_foo_i);
EXPECT_EQ(foo(1), 11);
SUBSTITUTE(int(double), foo, fake_foo_d);
EXPECT_EQ(foo(1.0f), 22);
SUBSTITUTE(int(int, int), foo, fake_foo_ii);
EXPECT_EQ(foo(1, 1), 33);
}
In some white box testing cases it might be a must to access privates. More on this here and here.
If you want to instrument those calls where the callee has the always_inline attribute then you have to specify -fno-inline-functions -fsanitize=mock -fsanitize=mock_always_inline.
Most of the getters and setters in libcxx have these attributes.
If you instrument always inline functions then make sure that they are emitted.
For example in libcxx basic_string template's c_str is always inline but there is an extern template declaration in the <string> header for basic_string<char>.
Therefore, your compiled object file would have an undefined reference to c_str, since the code is not emitted because of the extern template declaration.
But the already existing libcxx has an implicit template instantiation with basic_string<char> which does not expose the c_str function since that is declared always_inline.
So to make the instrumentation work either you recompile libcxx with -fsanitize=mock and you link against the instrumented libcxx, or you eliminate somehow the extern template declaration.
The latter is possible in the finstrument_mock branch of the libcxx repo if you define _LIBCPP_MOCK_SAN.
For more details please check the CMakeLists.txt in instrument_always_inline.
Note that this problem does not arise on Linux/GCC/6.2/libstdc++.
Our measurements show that the compilation process does not slow down noticeably.
The slow down is quite similar to other sanitizers.
The runtime performance might have a slow down of 5x - 60x.
It depends on the number of inlined functions and call expressions.
We have built the following c++ projects with -fsanitize=mock -fno-inline-functions:
- jsoncpp https://github.com/martong/jsoncpp/tree/finstrument_mock_test
- LLVM itself https://github.com/martong/llvm/tree/finstrument_mock_test
Currently the -fsanitize=mock_always_inline is tested only with the tests of this repository.
At the moment it looks like we can replace a destructor, but it is still very experimental.
Replace a constructor is not supported (yet).
In short, it would require to change the parser and the semantic actions to be able to parse the second X in X::X as a function instead of a type.
I think this is feasible but requires some more work, see https://github.com/martong/clang/commit/aff53813f4b22d0e5d5ab1098c801747a14ab787
Replace the operator() of a lambda is not supported unless we can take the address of the lambda.
Similarly, member functions of structs/classes which are defined inside a function cannot be replaced, because there is no valid expression to get their address.
E.g. we can't replace bar outside of the scope of X:
void foo() {
struct X {
void bar() {}
};
}
// ...
SUBSTITUTE(foo()::X::bar, fake_bar)
// ^^ ERROR
To be able to parse such expressions would require much more effort than in case of constructors, and I am not sure if it would be userful ever.
- Enforce type checking in the header of the runtime library. http://stackoverflow.com/questions/25044869/member-function-traits http://www.boost.org/doc/libs/1_61_0/libs/type_traits/doc/html/boost_typetraits/reference/function_traits.html
- Use shadow memory (similarly as asan do in asan_rtl.cc/
AsanInitInternal()) instead of a simple hash map in the runtime lib. That could result approximately a slowdown of 5x - 15x. - Make more precise measurements on compilation slow down.
With this instrumentation it is not possible to replace a whole type with a test double type. We can replace only one function. This can be cumbersome, if we want to replace types. Hopefully, in the future we might be able to write such test code:
// Don't change Entity just because of testing purposes
class Entity {
public:
int process(int i) { if(m.try_lock()) { ... } else { ... } }
...
private:
std::mutex m;
...
};
void testClient() {
using EntityUnderTest =
test::ReplaceMemberType<Entity, std::mutex, StubMutex>;
EntityUnderTest e;
auto& m = e.get<StubMutex>();
m.try_lock_result = false;
ASSERT_EQUAL(e.process(1), -1);
m.try_lock_result = true;
ASSERT_EQUAL(e.process(1), 1);
}
Of course this method has it's own disadvantages as well:
- Cannot replace a type in an already compiled translation unit.
- It works only if all type information is available i.e. if classes are header only classes.
Though, this kind of compile time reflection would be a nice complement of the instrumentation presented in this repository.
More about non-intrusive testing here.
Currently in C/C++ we have techniques to test without modifying the original source code: using a seam. A seam is an abstract concept introduced by Feathers [1] as an instrument via we can alter the behaviour of our unit without changing its source code. There are four different kind of seams in C++ [2]:
- Link seam: Change the definition of a function via some linker specific setup.
- Preprocessor seam: With the help of the preprocessor, redefine function names to use an alternative implementation.
- Object seam: Based on inheritance to inject a subclass with an alternative implementation.
- Compile seam: Inject dependencies at compile-time through template parameters.
The enabling point of a seam is the place where we can make the decision to use one behavior or another. Different seams have different enabling points.
[1] Michael Feathers, Working Effectively with Legacy Code, 2004
[2] https://accu.org/index.php/journals/1927
We can use a link seam, e.g. to replace the implementation of a free function or a member function. For instance:
// A.hpp
void foo();
// A.cpp
void foo() { ... };
// MockA.cpp
void foo() { ... };
// B.cpp
#include "A.hpp"
void bar() { foo(); ... }In one hand, when we would test the bar() function then we would link the test executable with the MockA.o object file.
On the other hand, we would link the production code with A.o.
More information on how to use link seams in practice:
- http://jayconrod.com/posts/23/tutorial-function-interposition-in-linux
- http://samanbarghi.com/blog/2014/09/05/how-to-wrap-a-system-call-libc-function-in-linux/
- http://stackoverflow.com/questions/31131163/how-to-hook-all-linux-system-calls-during-a-binary-execution
Link-time dependency replacement is not possible if the dependency is defined in a static library or in the same translation unit where the SUT is defined. It is also not feasible to use link seams if the dependency is implemented as an inline function. This makes the use of this seam cumbersome or impossible when the dependant unit is a template or when the dependency is a template. The enabling point for a link seam is always outside of the program text. This makes the use of link seams somewhat hard to notice. On top of all, link-time substitution requires strong support from the build system we are using. Thus, we might have to specialize the building of the tests for each and every unit. This does not scale well and can be really demanding regarding to maintenance.
As a summary, the major limitations of link seams are that we cannot use them with:
- static libraries
- header only libraries
- inline functions
- templates
Preprocessor seams can be applied to replace the invocation of a global function to an invocation of a test double. Let's consider the following code snippet:
void *my_malloc(size_t size) {
//...
return malloc(size);
}
void my_free(void *p) {
//...
return free(p);
}
#define free my_free
#define malloc my_malloc
void unitUnderTest() {
int *array = (int *)malloc(4 * sizeof(int));
// do something with array
free(array);
}We can replace the standard malloc() and free() functions with our own implementation.
One example usage can be to collect statistics or do sanity checks in my_malloc and my_free functions.
These seams can be applied conveniently in C, but not in C++.
As soon as we are using namespaces, then the preprocessor might generate code which cannot be compiled because of ambiguous use of names.
Dangerous side effects of macros are also well-known.
Object/compile seams are realized by introducing a runtime/compile-time interface. If we are lucky, then the unit we want to test depends only on the interface. In this case we can provide an alternative implementation for the test.
However, in legacy code it is quite often that the dependency is hardwired to a specific implementation. In these cases, using an object or compile seam would require instrusive changes in the original source code (e.g. adding a new interface and a setter/constructor). Consequently, there are numerous situations when we cannot use object and compile seams.
With Isolator++ one can create non-intrusive tests on Windows, but Windows only. https://www.typemock.com/isolatorpp-product-page The project is closed source and commercial, therefore we cannot know what technology (or seam) they use. Perhaps they use runtime instrumentation to alter the behaviour of the unit.
Thanks to Imre Szekeres, Gábor Horváth, Péter Bolla, Máté Csákvári, Zoltán Porkoláb, Zsolt Parragi for all the discussions we had.