Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage depending on the order of tests run #296

Closed
srprash opened this issue Apr 28, 2021 · 0 comments · Fixed by #297
Closed

Coverage depending on the order of tests run #296

srprash opened this issue Apr 28, 2021 · 0 comments · Fixed by #297

Comments

@srprash
Copy link
Contributor

srprash commented Apr 28, 2021

While doing this PR, the coverage dropped significantly. I believe the reason for such a massive drop in the coverage is due to the fact that every test is run with coverage run py.test ... with different set of test cases. For each run of such command, only a single .coverage file is produced and overwriten by the run of next test cases.
Depending on which set of test cases are run as the command in the very last, the .coverage file (and thus the generated coverage.xml) will contain the coverage report about that run. In this case, I believe the last command to overwrite the .coverage was coverage run --source aws_xray_sdk -m py.test tests/ext/sqlalchemy_core/ --ignore tests/ext/sqlalchemy_core/test_sqlalchemy_core_2.py and so other modules' coverage dropped to 0% as per this report.

This makes the coverage report quite unreliable, and the solution is to generate different coverage report for each test run and aggregate them together in a codecov report.

srprash pushed a commit that referenced this issue Apr 28, 2021
Run coverage in Append mode, so that we build a full coverage report for codecov

fixes: #296
Hargrav3s pushed a commit to Gavant/aws-xray-sdk-python that referenced this issue Mar 22, 2022
Run coverage in Append mode, so that we build a full coverage report for codecov

fixes: aws#296
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant