Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Flaky Test #12

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pretty-parrot
Copy link

What is the purpose of this PR

  • This PR patches lithoxyl/tests/test_stats.py::test_acc_random and prevents it from failing when it is run by itself
  • Test is flaky (non-deterministic) and depends on lithoxyl/tests/test_stats.py::test_momentacc_norm to set up a state to pass, but the test fails when it is run by itself otherwise

Expected Result

  • Test lithoxyl/tests/test_stats.py::test_acc_random should pass when run both by itself and after lithoxyl/tests/test_stats.py::test_momentacc_norm

Actual Result

  • Test lithoxyl/tests/test_stats.py::test_acc_random fails when it is run by itself
  • Specifically, we get the following:
_________________________________________________________________________________________________ test_acc_random _________________________________________________________________________________________________

    def test_acc_random():
        data = test_sets['random.random 0.0-1.0']

        qa = ReservoirAccumulator(data)
        capqa = ReservoirAccumulator(data, cap=True)
        p2qa = P2Accumulator(data)
        for acc in (qa, capqa, p2qa):
            for qp, v in acc.get_quantiles():
                if qp > 0:
>                   assert 0.95 < (v / qp) < 1.05
E                   assert (0.010605121375992893 / 0.01) < 1.05

lithoxyl/tests/test_stats.py:118: AssertionError

Reproduce the test failure

  • Run python3 -m pytest lithoxyl/tests/test_stats.py::test_acc_random

Why the Test Fails

  • The test fails because the test is dependent on some state that is not set when it is run by itself.

Fix

  • The changes in this pull request sets the state and makes the test pass when it is run by itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant