Skip to content

Commit

Permalink
Fishtape 3 (#53)
Browse files Browse the repository at this point in the history
- Rewrite to take full advantage of Fish 3.x.
  - No Awk or external commands.
  - No syntax breaking changes.
  - About 1/3 less code.
  - Remove $current_dirname and $current_filename variables
    in favor of (status dirname) or (status filename).
  - Escape actual and expected arguments via string escape.
    - This makes it possible to present your output in the
      same line without breaking TAP.
  - Add a new `at` error field to indicate the file name
    and line where a test failed.
  - Add new documentation.

- Run tests serially.
  - To enable parallelism, we also had to buffer output from
    each test file to produce correct TAP, making pipelines
    like `fishtape | report` less useful.
  - Preprocessing files is no longer necessary, making tests
    behave more predictably. WYSIWYG.
  - Net result is faster tests for typical usage since
    there's no async overhead.

- Remove `setup` and `teardown` functions.
  - No longer needed when we run tests serially.
  - The best way to do work before and after a test is is
    right there in your test file.

- Close #53.
  • Loading branch information
jorgebucaran committed Jan 12, 2021
1 parent 6f93f16 commit 641f2b0
Show file tree
Hide file tree
Showing 14 changed files with 212 additions and 364 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ jobs:
run: |
curl -sL https://git.io/fisher | source
fisher install $GITHUB_WORKSPACE
fishtape test/*.fish
fishtape test/*
shell: fish {0}
124 changes: 65 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,37 @@
# Fishtape

> <a href=https://testanything.org title="Test Anything Protocol">TAP</a>-based testing for [Fish](https://fishshell.com).
> 100% _pure_-[Fish](https://fishshell.com) test runner.
Fishtape is a Test-Anything-Protocol test runner. Because each test runs concurrently in its own sub-shell, you can define functions, set variables, and modify the executing environment without hijacking the current session or other tests. What learning curve? If you know how to use [`test`](https://fishshell.com/docs/current/commands.html#test), you can start using Fishtape now.
Fishtape is a <a href=https://testanything.org title="Test Anything Protocol">Test Anything Protocol</a> compliant test runner for Fish. Use it to test anything: scripts, functions, plugins without ever leaving your favorite shell. Here's the first example to get you started:

```fish
@test "the ultimate question" (math "6 * 7") -eq 42
@test "got root?" $USER = root
```

Now put that in a `fish` file and run it with `fishtape` installed. Behold, the TAP stream!

```console
$ fishtape example.fish
TAP version 13
ok 1 the ultimate question
not ok 2 got root?
---
operator: =
expected: root
actual: jb
at: ~/example.fish:3
...

1..2
# pass 1
# fail 1
```

> See [reporting options](#reporting-options) for alternatives to TAP output.
Each test file runs inside its own shell, so you can modify the global environment without cluttering your session or breaking other tests. If all the tests pass, `fishtape` exits with `0` or `1` otherwise.

## Installation

Expand All @@ -12,85 +41,62 @@ Install with [Fisher](https://github.com/jorgebucaran/fisher):
fisher install jorgebucaran/fishtape
```

## Getting Started
## Writing Tests

A test file is a regular `fish` file with `@test` declarations. A test declaration (or test case) consists of a description, followed by one or more operators and their arguments. You can use any operator supported by the [`test`](https://fishshell.com/docs/current/commands.html#test) builtin except for the `-a` and `-o` conditional operators.
Tests are defined with the `@test` function. Each test begins with a description, followed by a typical `test` expression. Refer to the `test` builtin [documentation](https://fishshell.com/docs/current/cmds/test.html) for operators and usage details.

```fish
@test "math is real" (math 41 + 1) -eq 42
> Operators to combine expressions are not currently supported: `!`, `-a`, `-o`.
@test "basename is fish" (
string split -rm1 / /usr/local/bin/fish
)[-1] = "fish"
@test "test is a builtin" (
contains -- test (builtin -n)
) $status -eq 0
@test "print a sequence of numbers" (seq 3) = "1 2 3"
```fish
@test "has a config.fish file" -e ~/.config/fish/config.fish
```

Run `fishtape` with one or more test files to run your tests.
Sometimes you need to test the exit status of running one or more commands and for that, you use command substitutions. Just make sure to suppress stdout to avoid cluttering your `test` expression.

```console
fishtape tests/*.fish
TAP version 13
ok 1 math is real
ok 2 basename is fish
ok 3 test is a builtin
ok 4 print a sequence of numbers

1..4
# pass 4
# ok
```fish
@test "repo is clean" (git diff-index --quiet @) $status -eq 0
```

Test files run in the background in a subshell while individual test cases run sequentially. The output is buffered (delivered in batches) until all jobs are complete. If all the tests pass, `fishtape` exits with status `0`—else, it exits with status `1`.

A buffered output means we can't write to stdout or stderr without running into race conditions. To print a TAP message along with a batch of test results, use the `@mesg` declaration.
Often you have work that needs to happen before and after tests run like preparing the environment and cleaning up after you're done. The best way to do this is directly in your test file.

```fish
@mesg "Brought to you by the friendly interactive shell."
```
set temp (mktemp -d)
### Setup and Teardown
cd $temp
You can define special `setup` and `teardown` functions, which run before and after each test case, respectively. Use them to load fixtures, set up your environment, and clean up when you're done.
@test "a regular file" (touch file) -f file
@test "nothing to see here" -z (read < file)
```fish
function setup
set -g tmp (command mktemp -d /tmp/foo.XXXXX)
command mkdir -p $tmp
end
function teardown
command rm -rf $tmp
end
@test "directory is empty" -z (
pushd $tmp
command ls -1
popd
)
rm -rf $temp
```

### Special Variables
When comparing multiline output you usually have two options, collapse newlines using `echo` or collect your input into a single argument with [`string collect`](https://fishshell.com/docs/current/cmds/string-collect.html). It's your call.

The following variables are globally available for all test files:
```fish
@test "first six evens" (echo (seq 2 2 12)) = "2 4 6 8 10 12"
- `$current_dirname` is the directory where the currently running test file is located.
- `$current_filename` is the name and extension of the currently running test file.
@test "one two three" (seq 3 | string collect) = "1
2
3"
```

## Reporting Options
If you want to write to stdout while tests are running, use the `@echo` function. It's equivalent to `echo "# $argv"`, which prints a TAP comment.

TAP is a simple text-based protocol for reporting test results. It's easy to parse for machines and still readable for humans. If you are looking for reporting alternatives, see [this list of reporters](https://github.com/substack/tape#pretty-reporters) or try [tap-mocha-reporter](https://github.com/tapjs/tap-mocha-reporter) for an all-in-one solution.
```fish
@echo -- strings --
```

Once you've downloaded a TAP-compliant reporter and put it somewhere in your `$PATH`, pipe `fishtape` to it.
## Reporting Options

> Redirections and pipes involving blocks are run serially in fish (see [fish-shell/#1396](https://github.com/fish-shell/fish-shell/issues/1396)). This means we must run `fishtape` in a subshell to enable streaming support.
If you're looking for something fancier than plaintext, [here's a list](https://github.com/sindresorhus/awesome-tap#reporters) of reporters that you can pipe TAP into.

```fish
fish -c "fishtape test/*.fish" | tap-nyan
```console
$ fishtape test/* | tnyan
30 -_-_-_-_-_-_-_-_-_-_-_-_-_-_-__,------,
0 -_-_-_-_-_-_-_-_-_-_-_-_-_-_-__| /\_/\
0 -_-_-_-_-_-_-_-_-_-_-_-_-_-_-_~|_( ^ .^)
-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_ "" ""
Pass!
```

## License
Expand Down
4 changes: 2 additions & 2 deletions completions/fishtape.fish
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
complete --command fishtape --long version --description "Print version"
complete --command fishtape --long help --description "Print help"
complete --command fishtape --short v --long version --description "Print version"
complete --command fishtape --short h --long help --description "Print help"
7 changes: 0 additions & 7 deletions conf.d/fishtape.fish

This file was deleted.

Loading

0 comments on commit 641f2b0

Please sign in to comment.