add trace option

[dmonakhov]

User may collect verbose execution logs in dedicated file which simplify debugging.
Usage example:

 ./bash_unit -t trace.log tests/test_*.sh

[dmonakhov]

ping, any news?

[pgrange]

Hello @dmonakhov and thank you for your contribution.

Sorry for the delay, I was AFK.

Would you be so kind to explain what's your current issue with bash unit that leads you to making this proposal? It's really important to me to keep bash unit outputs to the bare minimum so we avoid as much noise as we can. So I don't feel comfortable with the idea of just tracing everything. But I'd really like to ensure your problem is solved though. So if you can quickly explain the problem, maybe with a sample problematic code and how you had to struggle with bash unit to move forward that would really help me.

Thanks in advance,

[dmonakhov]

I do like bash_unit because is single file bash script. I use it for mostly for system validation.
Idea is to always having real trace of all stdout/stderr even for success runs. Automation will run unittests and also save trace files regardless of status. So I have a history of traces and if something goes wrong (environment change, hw change etc.) it is possible to compare two traces.

[pgrange]

Thank you @dmonakhov that is more clear now.

I'd like to keep bash_unit code as short and simple as possible, hence avoid adding code to it without, first, thinking of an alternative.

For your use case, I would naturally see two alternatives to adding more code to bash_unit.

The first, more complicated, alternative would be to create a specialized version of the notify_* functions in your specific test environment. They've been introduced for that kind of use cases, although I don't think it's been really used yet. For instance, if you define the following code in a test file then bash_unit will use your notify_test_succeeded function instead of the original ones, without changing bash_unit code:

notify_test_succeeded() {
    local test="$1"
    trace_test_succeeded "$test"
    echo -n "ok" | pretty_success -
    echo "$test" | color "$BLUE"
}

trace_test_succeeded() {
    local test="$1"
    notify_trace_info "Success $test"
}

notify_trace_info() {
  [ -z $trace_file ] && return

  local message=${1:-}
  echo "info> $message"  >> $trace_file
}

Another alternative I can think of, if what you want is to run the tests and also log the bash_unit output to a file for later use could be, maybe, to simply redirect bash_unit output to this file:

bash_unit test* | tee <trace_file>

That way you would, both, see the tests outputs in the CI but also store them in a file for later reference.

Just in case you'd like to have a more compact output to store, note that you may use the tap option so that tests are using the tap format instead of the more human friendly output:

bash_unit -f tap test* | tee <trace_file>

Looking at your pull request, using tee and redirecting the whole output to a file from outside bash_unit looks really natural to me and would avoid adding code to bash_unit. What do you think?

Cheers,

[pgrange]

Looking more into your PR I'm starting to feel that what you want is not to keep test run output to a file for later reference but you would rather log each and every assertion that has been successful during bash_unit execution. Is it correct?

Indeed, bash_unit will only output information about an assertion if it fails and be silent if everything went fine.

In your situation, when you see that the CI used to work and now is not working anymore, what I'm expecting is for you to have access to the source code of the tests run during the successful CI execution. Having access to this source code, you know exactly what assertions have been successfully run during this test (all of them, since the test was successful).

I'm also expecting you to have access to the source code of the test run for the failing case of the CI. In which case you can also know exactly what assertions have been successfully run during this test (all the ones before the failing one).

Could that help you here?