Fixtures.jl provides fixtures, mocks, matchers and patching to improve your test life with Julia.
You need the latest version of MetaTools. Run this to get the latest,
Pkg.add("MetaTools")
Pkg.checkout("MetaTools")
According to wikipedia.org:
In software testing, a test fixture is a fixed state of the software under test used as a baseline for running tests; also known as the test context. It may also refer to the actions performed in order to bring the system into such a state.
Examples of fixtures:
- Loading a database with a specific, known set of data
- Erasing a hard disk and installing a known clean operating system installation
- Copying a specific known set of files
- Preparation of input data and set-up/creation of fake or mock objects
Practically, a test fixture is composed of one or more setup steps run before the test(s) and corresponding teardown step(s) performed afterwards. This kind of thing can be expressed rather nicely in Julia using the do syntax. Given a function like:
function example_fixture(fn::Function)
# Setup code goes here
try
return fn()
finally
# Teardown code goes here
end
end
You can write your tests like:
example_fixture() do
# Test code goes here
end
There are a number of handy functions of this form, that we may wish to use as test fixtures, already defined in the Julia standard library, for example Base.cd.
Defining a fixture this way does involve a certain amount of boilerplate code so Fixtures.jl provides the @fixture macro to streamline fixture writing in Julia. The example above could be written, using @fixture, as:
using Fixtures
@fixture function example_fixture()
# Setup code goes here
yield_fixture()
# Teardown code goes here
end
and the calling code remains unchanged:
example_fixture() do
# Test code goes here
end
It is very common for fixtures to take one or more arguments, for example:
@fixture function greetings(who::String)
println("Hello $who")
yield_fixture()
println("Good bye $who")
end
greetings("Bob") do
println("Test code here")
end
would produce the output:
Hello Bob
Test code here
Good bye Bob
@fixture supports all argument types, including default arguments, keyword arguments and varargs.
It is also rather common for fixtures to produce a value that needs to be used in the tests. For example a database fixture might setup a test database then want to pass a connection object to the tests. This can easily be done by calling yield_fixture with the values to be returned to the test code. For example:
@fixture function greetings(who::String)
println("Hello $who")
yield_fixture("Secret message for $who")
println("Good bye $who")
end
greetings("Bob") do message
println(message)
end
would produce the output:
Hello Bob
Secret message for Bob
Good bye Bob
You can pass any number of values to the test code, however keyword arguments are not supported by do blocks so you can't use those.
Often you want to resuse the same fixture several times, for example you might want the same setup and teardown code to run before and after each database test. Using Fixtures.jl you can add fixtures to a named scope then repeatedly use that named scope:
@fixture function testdb()
# setup database
yield_fixture()
# teardown database
end
add_fixture(:databasetests, testdb)
apply_fixtures(:databasetests) do
# some test that uses the database
end
apply_fixtures(:databasetests) do
# another test that uses the database
end
You can define multiple fixtures for a named scope and they will all be used.
@fixture function fixture1()
# ...
end
@fixture function fixture2()
# ...
end
@fixture function fixture3()
# ...
end
add_fixture(:myscope, fixture1)
add_fixture(:myscope, fixture2)
add_fixture(:myscope, fixture3)
You can also nest scopes. If a fixture is added to a nested scope then it will be removed at the end of the parent scope. An example should make this clearer:
function demo(fn::Function)
println("before")
try
return fn()
finally
println("after")
end
end
function another(fn::Function)
println("avant")
try
return fn()
finally
println("après")
end
end
add_fixture(:childscope, demo)
apply_fixtures(:parentscope) do
add_fixture(:childscope, another)
apply_fixtures(:childscope) do
println("hello world")
end
end
apply_fixtures(:childscope) do
println("Bonjour tout le monde")
end
the output is:
before
avant
hello world
après
after
before
Bonjour tout le monde
after
You can add fixtures that take arguments by specifying the arguments in the call to add_fixture:
add_fixture(:some_scope, greetings, "Bob")
positional, default, vararg and keyword arguments are supported. Also if you want to capture the values produced by the fixture you must give it a name:
add_fixture(:some_scope, :greet_bob, greetings, "Bob")
and then setting fixture_values=true when calling apply_fixtures makes all the fixture outputs available to the test code via a Dict passed to the do which is indexed by fixture name:
apply_fixtures(:some_scope, fixture_values=true) do values
println(values[:greet_bob])
end
Output is still:
Hello Bob
Secret message for Bob
Good bye Bob
Fixtures.jl has a handy add_simple_fixture method that lets you define the setup and teardown functions separately:
function add_fixture(scope::Symbol, before::Function, after::Function)
For users of FactCheck.jl methods are provided to make it simple to use Fixtures.jl, here is an example using the two packages together:
using FactCheck
using Fixtures
add_fixture(:facts, demo)
add_fixture(:context, another)
facts("FactCheck support tests", using_fixtures) do
context("Example with fixtures set on a context level", using_fixtures) do
@fact 100 => 100
end
end
All the features listed in the sections above are available when using FactCheck.jl
Fixtures.jl comes with a number of fixtures for using files in your tests:
temp_filename([extension=<filename extension>],[create=<true, false or string content>])temp_file([mode=<file mode, default is "w">],[extension=<filename extension>],[content=<string content>])temp_dir()cleanup_path(<path>[, ignore_missing=true|false])
A common type of fixture is to patch a function, method or value within a module or object. This is most often done in unit testing to isolate the function under test from the rest of the system. Fixtures.jl provides a patch fixture to do this. For example if we wanted to test this function:
function firstline(filename)
f = open(filename)
try
return chomp(readlines(f)[1])
finally
close(f)
end
end
we might want to isolate it from the real filesystem. We can do this by patching Base.open with our own implementation, just for the duration of the test:
function fake_open(filename)
return IOBuffer("Hello Julia\nGoodbye Julia")
end
patch(Base, :open, fake_open) do
@Test.test firstline("foobar.txt") == "Hello Julia"
end
You can use patch() as above or you can use it with add_fixture() and apply_fixtures().
add_fixture(:mock_io, patch, Base, :open, fake_open)
apply_fixtures(:mock_io) do
@Test.test firstline("foobar.txt") == "Hello Julia"
end
You can also have multiple patches passed in as an array.
function firstline(filename)
foo = open(filename)
bar = close(filename)
return foo, bar
end
function fake_open(filename)
return "hello,"
end
function fake_close(filename)
return "world"
end
patchers = [
Patcher(Base, :open, fake_open),
Patcher(Base, :close, fake_close),
]
patch(patchers) do
@Test.test firstline("foobar.txt") == ("hello,", "world")
end
Note: Due to a current issue in Julia your ability to patch a function may be limited if the code calling that function has already been called.
But Fixtures.jl also provides mocks so we can patch open with a mock, this also allows us to verify it was called:
function firstline(filename)
f = open(filename)
try
return chomp(readlines(f)[1])
finally
close(f)
end
end
mock_open = mock(return_value=IOBuffer("Hello Julia\nGoodbye Julia"))
patch(Base, :open, mock_open) do
@Test.test firstline("foobar.txt") == "Hello Julia"
end
@Test.test calls(mock_open) == [call("foobar.txt")]
Mocks are just generated functions that record their arguments everytime they are called. You can access the call history using calls(mock) as shown above and clear it with reset(mock). When creating a mock you can (optionally) specify its return value or an implementation for the mock:
mock1 = mock(return_value=200)
@Test.test mock1(100) == 200
mock2 = mock(side_effect=x->x+200)
@Test.test mock2(100) == 300
mock3 = mock()
@Test.test mock3(100) == nothing
The call() function makes it easy to express and test the expected calls to a mock (see above). And you can ignore any given argument by using ANYTHING
mock1 = mock()
mock1(rand(), 200)
@Test.test calls(mock1) == [ call(ANYTHING, 200) ]
ANYTHING is just one example of a matcher, a kind of wildcard that can be used when verifying mock calls. Fixtures.jl provides the following matchers (and it is possible to define your own):
anything_of_type(T::Type)anything_in(value::Any)anything_containing(value::Any)
So we could be a bit stricter in our previous example:
mock1 = mock()
mock1(rand(), 200)
@Test.test calls(mock1) == [ call(anything_of_type(Number), 200) ]
Code Coverage is not saying 100% because of the code in red. We did test to make sure it ran by putting a break point in there. If there are no other untested lines of code, the coverage should be at 100%.