Added test timeout

[MeirShpilraien]

The PR adds a new option --test-timeout that allows set a test timeout (in seconds) after which the test will be considered as failed. The timeout works as follow:

1. Open a watcher thread that sleep for the given timeout
2. Once wake up, check if the test that is been watched finished, if not:
   • Send a SIGUSR1 to the main processes thread that will cause it to print the current trace and enter a deep sleep.
   • Shutdown the environment using SIGSEGV to make sure we will get all the needed information from the shards
   • Print verbose information if needed (in case --verbose-information-on-failure was used).
   • Exit the processes using os._exit(1). Notice that it is important to exit using os._exit(1), if we exit in any other way, python might wait for active connections or thread to be close. We are killing the processes in the middle of its execution and we have no idea in which state it hang, so we prefer to wait for nothing.

For backward compatibility, the default timeout is 0 which implies no timeout.

Notice, when choosing the best way to trigger a timeout, 2 approaches was tested:

1. Using a background watcher thread.
2. Using signal.setitimer.

Eventually, the first approach was chosen, mainly because if we use signal.setitimer, the test itself might also use it and override our timer and callback. I believe the thread approach is safer and more reliable.

Extra additions/fixes:

1. Progress bar indicating how many tests finished and how much is still left to run. Progress bar can be turned off using --no-progress. Progress bar will automatically turned off if --no-output-catch was used or if the stdout in not a terminal (output was redirected).
2. Fix issue where on some cases the same test would have appear multiple time in the summary report.

Technical Low Level Details on Progressbar

Till today, when running with parallelism on more than one. Each processes reported its own progress. The PR changes this approach in way that only the main processes reports the progress and each sub-processes reports to the main processes. This gives us 2 main adventages:

1. Avoid print collisions between processes
2. Ability to report progress

To achieve this, each sub-processes introspects its own stdout and send the tests output to the main processes on a new channel called results. When the main processes gets a message on the results channel, it prints its connect to the stdout and increase the progress bar.

When running without parallelism, the output is printed to the stdout right away.

To avoid code duplication with the parallel and the none parallel flow, we extracted the code that runs a single tests to its own function, run_single_test, and we call it from the 2 different flows: run_jobs_main_thread, run_jobs.

[codecov-commenter]

Codecov Report

Attention: 211 lines in your changes are missing coverage. Please review.

Comparison is base (409e17c) 34.16% compared to head (2eb4f1b) 32.28%.

Files	Patch %	Lines
RLTest/__main__.py	0.00%	183 Missing ⚠️
RLTest/redis_std.py	10.71%	25 Missing ⚠️
RLTest/redis_cluster.py	33.33%	2 Missing ⚠️
RLTest/env.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #211      +/-   ##
==========================================
- Coverage   34.16%   32.28%   -1.89%     
==========================================
  Files          17       17              
  Lines        2409     2565     +156     
==========================================
+ Hits          823      828       +5     
- Misses       1586     1737     +151     

Flag	Coverage Δ
unittests	32.28% <2.31%> (-1.89%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GuyAv46]

shouldn't this be indented? and maybe give up the if bar: then if ProgressBar cannot return None

[GuyAv46]

rename functions

[GuyAv46]

delete

[GuyAv46]

delete

[GuyAv46]

Suggested change

	for test_name, failures in res['failures'].items():
	self.testsFailed[test_name] = failures
	self.testsFailed.update(res['failures'])