Skip to content

Sandbox directories

Jump to bottom
Emil Karlén edited this page Feb 6, 2021 · 9 revisions

Introduction

The sandbox directory structure (sandbox) is a set of temporary directories created for each execution of a test case. As soon as the test case is finished, the directories are deleted.

At the start of [setup], the current directory is changed to a directory inside the sandbox.

The purpose of the sandbox is to avoid polluting source files and directories.

The sandbox directory structure is kind of the opposite to the home directory structure, that contains files that exist before the test is executed, and that should not be altered by the test.

For example, the following variant of hello-world, that writes it's output to a given file, will not pollute any existing directory:

[act]

hello-world -o output.txt

[assert]

contents output.txt : equals <<EOF
Hello, World!
EOF

The sandbox is initially empty.

Sometimes, this is just fine, as in the example above. But in other situations, it must be setup to an environment that the SUT / [act] phase or the test case implementation needs. For example:

  • Populate the current directory, or another directory, with files and directories that the SUT / [act] phase should operate on, and need to be able to modify

  • Storage of generated files and directories that are needed by the implementation the test case

  • Set the current directory

The cd instruction sets the current directory, and a couple of other instructions are designed to populate the sandbox with files and directories:

  • copy Copies existing files into the sandbox The default is to copy files from the case home directory, and to copy them into the current directory.
  • file Creates a file
  • dir Creates a directory

run, and the shell command instruction $, can be used to perform custom setup and custom assertions.

Relativity roots

There are three relativity roots inside the sandbox.

They are all fixed when [setup] starts - none of them can be changed.

A kind of custom relativity root can be defined though, using a symbol.

The act directory

This directory is the current directory when [setup] starts. This means that it will also be the current directory of the [act] phase - and all following phases - until it is changed by the cd instruction.

act is a relativity root

act is a relativity root with the following properties:

  • Option: -rel-act

  • Name of symbol and env var: EXACTLY_ACT

Using relativity roots tells how to use these properties.

Examples of using act

Here, the [setup] phase creates a file with contents that should be modified by the SUT:

[setup]

file file-to-modify.txt =
<<EOF
a b
EOF

[act]

$ my-convert-to-uppercase-tool file-to-modify.txt

[assert]

contents file-to-modify.txt :
         equals
<<EOF
A B
EOF

The result directory

This directory is initially empty. It is populated when the [act] phase finishes with the following files (with obvious contents):

  • stdout
  • stderr
  • exitcode

A test case implementation is allowed to read these files, but not to modify them or the contents of the directory. Doing so might cause Exactly to misbehave.

result is a relativity root

result is a relativity root with the following properties:

  • Option: -rel-result

  • Name of symbol and env var: EXACTLY_RESULT

Using relativity roots tells how to use these properties.

Examples of using result

Here, a custom tool is used to check that the output on stdout is the expected:

[act]

my-system-under-test

[assert]

run my-custom-file-check @[EXACTLY_RESULT]@/stdout

The tmp directory

This directory is a place for temporary files used by the test case implementation. Exactly creates it, but after that does not touch it.

tmp is a relativity root

tmp is a relativity root with the following properties:

  • Option: -rel-tmp

  • Name of symbol and env var: EXACTLY_TMP

Examples of using tmp

Here, a custom tool is used to generate the expected output:

[act]

my-system-under-test important-argument

[before-assert]

run my-generator-of-execpted-output important-argument @[EXACTLY_TMP]@/expected.txt

[assert]

stdout equals -rel-tmp expected.txt

Using relativity roots tells how to use these properties.

Keeping the sandbox for inspection

Exactly can keep the sandbox, to let you inspect what files have been created and what they contain.

This might be useful for debugging (either the program you are testing, or the test case itself).

The --keep option makes Exactly keep the sandbox. Instead of printing the result of the test, it prints the root of the sandbox directory structure.

The following test case populates the sandbox directory structure:

[setup]

file my-file.txt

file -rel-tmp my-tmp-file.txt

dir my-dir

cd  my-dir

[act]

hello-world -o output-from-action-to-check.txt

[assert]

exit-code == 0

Run it and take a look at the sandbox:

> exactly --keep my-test.case
/tmp/exactly-68jn8c08
PASS

> find /tmp/exactly-68jn8c08
/tmp/exactly-68jn8c08/
/tmp/exactly-68jn8c08/act/
/tmp/exactly-68jn8c08/act/my-file.txt
/tmp/exactly-68jn8c08/act/my-dir/
/tmp/exactly-68jn8c08/act/my-dir/output-from-action-to-check.txt
/tmp/exactly-68jn8c08/tmp/
/tmp/exactly-68jn8c08/tmp/my-tmp-file.txt
/tmp/exactly-68jn8c08/result/
/tmp/exactly-68jn8c08/result/stdout
/tmp/exactly-68jn8c08/result/stderr
/tmp/exactly-68jn8c08/result/exitcode
/tmp/exactly-68jn8c08/internal/
/tmp/exactly-68jn8c08/internal/tmp/
/tmp/exactly-68jn8c08/internal/testcase/
/tmp/exactly-68jn8c08/internal/log/

Summary of directories

  • act

    • Option: -rel-act

    • Name of symbol and env var: EXACTLY_ACT

    • Description: This is the current directory when [setup] starts, and remains the current directory until it is changed with the cd instruction.

  • result

    • Option: -rel-result

    • Name of symbol and env var: EXACTLY_RESULT

    • Description: This directory is initially empty. It is populated when the [act] phase finishes with the following files (with obvious contents):

      • stdout
      • stderr
      • exitcode

  • tmp

    • Option: -rel-tmp

    • Name of symbol and env var: EXACTLY_TMP

    • Description: This directory is a place for temporary files used by the test case implementation. Exactly creates it, but after that does not touch it.

Clone this wiki locally