Run integration tests with pytest #919
Conversation
With this commit we add a proof-of-concept implementation of how integration tests could look like with py.test instead of shell scripts. For the proof of concept we migrate the integration tests for the list subcommands and source builds. It also spins up an Elasticsearch metrics store and checks that any preconditions (basically Docker being present and up and running) are met. Finally, it also integrates into CI by writing JUnit XML files for test results. Relates elastic#736
danielmitterdorfer
added
enhancement
|
Unfortunately we don't see any Rally output in the PR checks. Therefore I have raised #923. |
There was a problem hiding this comment.
Great work and foundation!
I love the iterative approach (i.e. gradually replace the functionality in integration-test.sh) and the fact that this is so readable and easy to read makes me happy.
I left a comment/suggestion to use Template rather than replacing strings (and a FYI comment).
| def copy_config(name): | ||
| f = ConfigFile(name) | ||
| io.ensure_dir(f.rally_home) | ||
| with open(f.target_path, "w", encoding="UTF-8") as target: |
There was a problem hiding this comment.
Would it make sense to just read src into contents with something:
with open(f.source_path, "r", encoding="UTF-8") as src:
contents = src.read()
and then replace using Template's native capabilities, to avoid pure string based replacements, i.e.
from string import Template
with open(f.target_path, "w", encoding="UTF-8") as target:
target.write(Template(contents).substitute(USER_HOME=f.user_name)
AFAIK ${identifier}-like template strings are supported out of the box.
There was a problem hiding this comment.
Thanks! Much better. Addressed in eb03450.
| return | ||
| else: | ||
| c.close() | ||
| time.sleep(0.5) |
There was a problem hiding this comment.
Not directly related to this PR, but going forwards should we start embracing the https://github.com/jd/tenacity library, which seems to be becoming the standard in retrying in Python? It offers a lot flexibility in what to retry with boolean operations.
There was a problem hiding this comment.
Should we have some timeout here just so our it tests dont run forever on a PR?
There was a problem hiding this comment.
I am reluctant adding additional dependencies for such small pieces of functionality but I added a two minute timeout in 3537d0b.
| return | ||
| else: | ||
| c.close() | ||
| time.sleep(0.5) |
There was a problem hiding this comment.
Should we have some timeout here just so our it tests dont run forever on a PR?
it/__init__.py
Outdated
| if process.run_subprocess_with_logging("docker") != 0: | ||
| raise AssertionError("Docker is required to run integration tests.") | ||
| if process.run_subprocess_with_logging("docker ps") != 0: | ||
| raise AssertionError("Docker daemon must be up and running to run integration tests.") |
There was a problem hiding this comment.
Do we need both of these checks? We can just say "docker must be installed and running" if docker ps fails
There was a problem hiding this comment.
Good point. I combined the checks in 7786bdc.
|
|
||
| class ConfigFile: | ||
| def __init__(self, config_name): | ||
| self.user_home = os.path.expanduser("~") |
There was a problem hiding this comment.
maybe use pathlib's Path.home() instead of the ~ since tilde expansion is black magic
There was a problem hiding this comment.
I'd tend to be consistent across our code base and keep this idiom.
|
Thanks for your reviews! |
With this commit we add a proof-of-concept implementation of how
integration tests could look like with py.test instead of shell scripts.
For the proof of concept we migrate the integration tests for the list
subcommands and source builds.
It also spins up an Elasticsearch metrics store and checks that any
preconditions (basically Docker being present and up and running) are
met. Finally, it also integrates into CI by writing JUnit XML files for
test results.
Relates #736