Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let test_cli produce stack traces in case of test errors #3414

Merged
merged 1 commit into from
Mar 24, 2023

Conversation

reneme
Copy link
Collaborator

@reneme reneme commented Mar 24, 2023

This should be particularly useful if CLI functions are used that are unrelated to the function under test. E.g. test_cli("hash", ...) to check the output of a previous CLI invocation. Especially when CLI tests are run in parallel it is hard to figure out which test case produced the error at times.

Note stack_info was added in Python 3.2 so should well be available on all targets.

Example (cli_hash_tests was made to fail):

The stacktrace shows exactly where test_cli("hash", ...) was called. Namely line 252 (second-last stack frame).

  ERROR: Got unexpected output running cmd hash ['--algo=SHA-224', '--format=base58check', '--no-fsname']
Stack (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/meusel/Projects/botan/src/scripts/test_cli.py", line 1365, in run_test
    fn(tmp_dir)
  File "/home/meusel/Projects/botan/src/scripts/test_cli.py", line 252, in cli_hash_tests
    test_cli("hash", ["--algo=SHA-224", "--format=base58check", "--no-fsname"],
  File "/home/meusel/Projects/botan/src/scripts/test_cli.py", line 114, in test_cli
    logging.error("Got unexpected output running cmd %s %s", cmd, cmd_options, stack_info=True)
   INFO: Output lengths 44 vs expected 44
   INFO: Got 3MmfMqgrhemdVa9bDAGfooukbviWtKMBx2xauL2RsyAe
   INFO: Exp 3MmfMqgrhemdVa9bDAGfooukbviWtKMBx2xauL2RsyAf
   INFO: Ran cli_hash_tests in 0.45 sec
Ran 5 tests with 1 failures in 0.46 seconds

This should be particularly useful if CLI functions are used that are
unrelated to the function under test. E.g. `test_cli(hash, ...)`
to check the output of a previous CLI invocation.
Copy link
Owner

@randombit randombit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@reneme reneme merged commit a735162 into master Mar 24, 2023
@randombit randombit deleted the feature/test_cli_stack_traces branch March 24, 2023 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants