Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test in nightlies: target distribution data check #2581

Closed
dsherry opened this issue Aug 3, 2021 · 4 comments · Fixed by #2608 or #2748
Closed

Flaky test in nightlies: target distribution data check #2581

dsherry opened this issue Aug 3, 2021 · 4 comments · Fixed by #2608 or #2748
Assignees
Labels
bug Issues tracking problems with existing features. testing Issues related to testing.

Comments

@dsherry
Copy link
Contributor

dsherry commented Aug 3, 2021

I saw the following fail in the linux nightlies yesterday

FAILED evalml/tests/data_checks_tests/test_target_distribution_data_check.py::test_target_distribution_data_check_unsupported_target_type[double]
...
        if target_type in ["integer", "double"]:
>           assert target_dist_check.validate(X, y) == {
                "warnings": [],
                "errors": [],
                "actions": [],
            }
E           AssertionError: assert {'actions': [...rning', ...}]} == {'actions': [...warnings': []}
E             Omitting 1 identical items, use -vv to show
E             Differing items:
E             {'actions': [{'code': 'TRANSFORM_TARGET', 'metadata': {'column': None, 'is_target': True, 'transformation_strategy': 'lognormal'}}]} != {'actions': []}
E             {'warnings': [{'code': 'TARGET_LOGNORMAL_DISTRIBUTION', 'data_check_name': 'TargetDistributionDataCheck', 'details': {'shapiro-statistic/pvalue': '0.969/0.018'}, 'level': 'warning', ...}]} != {'warnings': []}
E             Use -v to get the full diff

evalml/tests/data_checks_tests/test_target_distribution_data_check.py:59: AssertionError

Tagging @ParthivNaresh since I think he touched this code recently and may have an idea on the fix

@dsherry dsherry added bug Issues tracking problems with existing features. testing Issues related to testing. labels Aug 3, 2021
@freddyaboulton
Copy link
Contributor

I think we just got a flake on a different test for build_conda_pkg:

https://github.com/alteryx/evalml/pull/2585/checks?check_run_id=3242371966

@dsherry
Copy link
Contributor Author

dsherry commented Aug 4, 2021

Got it @freddyaboulton , so it was a different test but from the same file / area.

@freddyaboulton
Copy link
Contributor

Looks like we saw a similar flake due to rounding in the 3.9 nightlies: https://github.com/alteryx/evalml/runs/3503548207

Reopening this issue to track if it happens again!

@freddyaboulton freddyaboulton reopened this Sep 3, 2021
@ParthivNaresh
Copy link
Contributor

I have a fix for this, will be putting it up now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues tracking problems with existing features. testing Issues related to testing.
Projects
None yet
3 participants