Home

Exactly tests a command line program by executing it in a temporary sandbox directory and inspecting its result.

• Example
  • A test case is as a plain text file
  • Using shell commands
• About
• Some features
  • A complex example
  • "act" is the default phase
  • Print output from the tested program
  • Keeping the sandbox directory for later inspection
• A complex example
• More examples
• Author

Example

A test case is as a plain text file

[setup]

stdin an-address-book.txt

[act]

addressbook --get-email-of --name 'Test Testingson'

[assert]

exitcode 0

stdout equals <<EOF
[email protected]
EOF

If the file 'addressbook.case' contains this test case, then Exactly can execute it:

> exactly addressbook.case
PASS

"PASS" means that the two assertions were satisfied.

This test assumes that

• the system under test - the addressbook program - is is found in the same directory as the test case file
• the file "an-address-book.txt" (that is referenced from the test case) is found in the same directory as the test case file

The home instruction can be used to change where Exactly looks for files referenced from the test case.

Using shell commands

Shell commands can be used both as the sut (system under test), and as assertions.

$ echo ${PATH}

[assert]

$ < ../result/stdout tr ':' '\n' | grep '^/usr/local/bin$'

About

A test is organized into "phases", where each phase is a sequence of instructions, that behave a bit like a script.

The phases are:

• conf
• setup
• act
• before-assert
• assert
• cleanup

The test fixture and assertions are written using "instructions" that mimics the syntax of unix shell commands, with options and arguments.

Exactly supports individual test cases and test suites.

Exactly has a built in help system.

Exactly has a Reference manual.

Exactly is a Python 3 program, with no dependencies on external libraries.

Exactly has been tested on Linux and OS X.

Most, or all, features work on Windows, but Exactly's own test suite has not been completely ported to Windows.

Some features

[act] is the default phase

[act] is not needed to indicate what is being checked, since the "act" phase is the default "phase".

The following is a valid test case, and if run by Exactly, it won't remove anything (since it is executed inside a temporary sandbox directory):

$ rm -rf *

Print output from the tested program

If --act is used, the output of the tested program (the "act" phase) will become the output of Exactly - stdout, stderr and exit code.

$ echo Hello World

[assert]

stdout contains Hello


> exactly --act hello-world.case
Hello World

The test case is executed in the sandbox, as usual.

Keeping the sandbox directory for later inspection

If --keep is used, the sandbox directory will not be deleted, and its name will be printed.

This can be used to inspect the outcome of the "setup" phase, e.g.

[setup]

file my-file <<EOF
file contents
EOF

[act]

my-prog my-file

[assert]

exitcode 0

To see the effect of the setup of this case::

> exactly --keep my-test.case
/tmp/exactly-8fe6wbagmm0wutlt

> find /tmp/exactly-8fe6wbagmm0wutlt
/tmp/exactly-8fe6wbagmm0wutlt
/tmp/exactly-8fe6wbagmm0wutlt/tmp
/tmp/exactly-8fe6wbagmm0wutlt/tmp/user
/tmp/exactly-8fe6wbagmm0wutlt/tmp/internal
/tmp/exactly-8fe6wbagmm0wutlt/testcase
/tmp/exactly-8fe6wbagmm0wutlt/act
/tmp/exactly-8fe6wbagmm0wutlt/act/my-file
/tmp/exactly-8fe6wbagmm0wutlt/result
/tmp/exactly-8fe6wbagmm0wutlt/result/exitcode
/tmp/exactly-8fe6wbagmm0wutlt/result/stderr
/tmp/exactly-8fe6wbagmm0wutlt/result/stdout
/tmp/exactly-8fe6wbagmm0wutlt/log

A complex example

The following test case displays a potpurri of functionality. (Beware that this test case does not make sense! - it just displays some of Exactly's functionality.)

[conf]


mode SKIP
# This will cause the test case to not be executed.


[setup]


install this-is-an-existing-file-in-same-dir-as-test-case.txt

dir first/second/third

file in/a/dir/file-name.txt <<EOF
contents of the file
EOF

dir root-dir-for-act-phase

cd root-dir-for-act-phase
# This will be current directory for the "act" phase. 

stdin <<EOF
this will be stdin for the program in the "act" phase
EOF
# (It is also possible to have stdin redirected to an existing file.)

env MY_VAR = 'value of my environment variable'

env unset VARIABLE_THAT_SHOULD_NOT_BE_SET

run my-prog--located-in-same-dir-as-test-case--that-does-some-more-setup 'with an argument'


[act]


the-system-under-test


[before-assert]


cd ..
# Moves back to the original current directory.

$ sort root-dir-for-act-phase/output-from-sut.txt > sorted.txt


[assert]


exitcode != 0

stdout equals <<EOF
This is the expected output from the-system-under-test
EOF

stdout --with-replaced-env-vars contains 'EXACTLY_ACT:[0-9]+'

stderr empty

contents a-file.txt empty

contents a-second-file.txt ! empty

contents another-file.txt --with-replaced-env-vars equals expected-content.txt

contents file.txt contains 'my .* reg ex'

type actual-file directory

cd this-dir-is-where-we-should-be-for-the-following-assertions

run my-prog--located-in-same-dir-as-test-case--that-does-some-assertions


[cleanup]


$ umount my-test-mount-point

run my-prog-that-removes-database 'my test database'

More examples

See the examples/ directory of the source repository for examples.

Author

Emil Karlén

[email protected]