Example Standard Output Exit Code And Files

Assume you want to test your new hello-world program, which is an executable file.

You can do this by writing a test case file, located in the same directory as your hello-world executable:

[act]

hello-world

[assert]

stdout equals <<EOF
Hello, World!
EOF

exit-code == 0

If the file hello-world.case contains this text, you can execute the test:

> exactly hello-world.case
PASS

Maybe you prefer to have the expected output in a separate file:

[act]

hello-world

[assert]

stdout equals --file hello-world-output.txt

exit-code == 0

If hello-world-output.txt is found in the same directory as the test case file, and contains the expected output, then the test will pass:

> exactly hello-world.case
PASS

Maybe you are not interested in the whole output, but just that it contains the word 'Hello':

[act]

hello-world

[assert]

stdout any line : regex 'Hello'

The output should also satisfy the stronger assertion that every line contains 'Hello':

[act]

hello-world

[assert]

stdout every line : regex 'Hello'

The contents of stderr, and also arbitrary files may be checked the same way as stdout.

This version of hello-word has an option for storing the output in a file:

[act]

hello-world -o output.txt

[assert]

contents output.txt
         every line : regex 'Hello'

stderr empty

It might be a good idea to explicitly check that the expected file has been created:

[act]

hello-world -o output.txt

[assert]

exists --file output.txt

contents output.txt
         every line : regex 'Hello'

Sometimes the output from a program needs to be transformed before you can apply your assertion:

[act]

hello-world

[assert]

stdout --transformed-by TO_UPPER_CASE | replace '^HELLO, ' ''
       every line : regex ^WORLD!$

A transformation is only applied for the purpose of the assertion, it does not modify the transformed file. So both of the following assertions on stdout will pass:

[act]

hello-world

[assert]

stdout --transformed-by TO_UPPER_CASE | replace '^HELLO, ' ''
       every line : regex ^WORLD!$

stdout equals <<EOF
Hello, World!
EOF

If you need to do multiple assertions on some transformation of a file, it might be convenient to store the result of the transformation and then do the assertions on the stored result.

Exactly does not has a concept of variables (unfortunately), so the transformation needs to be stored in a new file:

[act]

hello-world

[before-assert]

file transformed-output.txt = --file
                              --rel-result stdout
                              --transformed-by TO_UPPER_CASE | replace '^HELLO, ' ''

[assert]

contents transformed-output.txt
         every line : regex ^WORLD!$

contents transformed-output.txt
         num-lines == 1

In the example above, the special file stdout is transformed. It is located in the special "result" directory, which is referenced using the option --rel-result - the path is relative the "result" directory.

And since a test case is executed in a temporary sandbox directory, the temporary file transformed-output.txt will be automatically removed and will not pollute your source folder.

To be able to reuse a transformer, or just to make the test more readable, you may define a "symbol" for it:

[act]

hello-world

[assert]

def string-transformer MY_TRANSFORMER = TO_UPPER_CASE | replace '^HELLO, ' ''

stdout --transformed-by MY_TRANSFORMER
       every line : regex ^WORLD!$

The same can be done with "line matchers":

[act]

hello-world

[assert]

def string-transformer MY_TRANSFORMER  = TO_UPPER_CASE | replace '^HELLO, ' ''

def line-matcher       MY_LINE_MATCHER = regex ^WORLD!$

stdout --transformed-by MY_TRANSFORMER
       every line : MY_LINE_MATCHER

To make the assertions clearer, you can put the symbol definitions in a different phase. The order of the phases in the test case file is irrelevant, so [assert] may appear before [before-assert], even though the execution order is the opposite.

Also the "act" phase is the default phase, so it does not need to be declared if the action to check appears at the beginning of the file.

hello-world -o output.txt

[assert]

contents output.txt
         --transformed-by MY_TRANSFORMER
         every line : MY_LINE_MATCHER

stderr empty

[before-assert]

def string-transformer MY_TRANSFORMER  = TO_UPPER_CASE | replace '^HELLO, ' ''

def line-matcher       MY_LINE_MATCHER = regex ^WORLD!$