TST- Fixing issue with test_parquet test unexpectedly passing #24480

RjLi13 · 2018-12-29T06:56:01Z

closes TST: Test_parquet failing #24479
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This pr aims to make the failing test_parquet not strict since it now passes at least on my machine of Mac 10.14 running the latest python 3.7 and conda.

codecov · 2018-12-29T07:31:06Z

Codecov Report

Merging #24480 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24480   +/-   ##
=======================================
  Coverage   92.32%   92.32%           
=======================================
  Files         166      166           
  Lines       52298    52298           
=======================================
  Hits        48285    48285           
  Misses       4013     4013

Flag	Coverage Δ
#multiple	`90.74% <ø> (ø)`	⬆️
#single	`43.06% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80295f9...d9bc62d. Read the comment docs.

codecov · 2018-12-29T07:31:09Z

Codecov Report

Merging #24480 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24480      +/-   ##
==========================================
- Coverage   92.32%   92.31%   -0.01%     
==========================================
  Files         166      166              
  Lines       52328    52335       +7     
==========================================
+ Hits        48310    48312       +2     
- Misses       4018     4023       +5

Flag	Coverage Δ
#multiple	`90.73% <ø> (-0.01%)`	⬇️
#single	`43.04% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/datetimes.py	`97.77% <0%> (-0.67%)`	⬇️
pandas/compat/__init__.py	`57.91% <0%> (-0.39%)`	⬇️
pandas/core/series.py	`93.73% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8dd9cdd...41427a3. Read the comment docs.

TomAugspurger · 2018-12-29T12:09:59Z

pandas/tests/io/test_parquet.py

@@ -201,7 +201,8 @@ def test_options_get_engine(fp, pa):


 @pytest.mark.xfail(is_platform_windows() or is_platform_mac(),


Can you try removing the xfail entirely? It looks like it may be unnecessary now.

I did do that in another branch and it passed the CI tests of travis, but it seems there's discussion of pyarrow and fastparquet as well as windows causing problems? Not completely sure.

Edit: Also passed CI tests in azure

jreback

add pyarrow and fastparquet to this build: https://github.com/pandas-dev/pandas/blob/master/ci/deps/azure-macos-35.yaml as these are not being tested. I suspect this will actually still fail. I think then this test needs to be conditionally skipped on older pyarrow / fastparquet versions. We could add another macosx build that is a bit more modern, or just relax the numpy on this build to have it install a more modern pyarrow.

cc @datapythonista

jreback · 2018-12-29T15:16:19Z

@datapythonista do we have the xfails shows somewhere? e.g. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6035

datapythonista · 2018-12-29T16:03:06Z

I don't exactly understand the rationale behind the builds, but a modern Mac build sounds reasonable.

Without know much about the builds we have, what would make sense to me is to have a "base" dependencies file, and have a build with the same conda deps in Linux, Windows and Mac. Happy to work on this if it makes sense.

Regarding xfails, I wanted to standardize the output of all the builds in #23115 (which still needs a bit of work). Not sure what is the preferred output, I think we have -r sxX in some places (like the windows builds).

jreback · 2018-12-29T16:59:58Z

Seems to be passing now.

RjLi13 · 2018-12-29T18:46:53Z

So to catch up a bit, we still need the xfail because of pyarrow and fastparquet having differing versions on windows? Or is it Mac now supports cross-compat with differing reading/writing engines, but Windows doesn't?

jreback · 2018-12-29T19:06:40Z

no i think this is ok now

RjLi13 · 2018-12-29T19:18:59Z

@jreback I have an unpublished branch that removes the xfail completely from both windows and mac. It seems to be passing the CI tests. Can you advise if we should take out the xfail completely?

jreback · 2018-12-29T19:35:59Z

yeah might be good to try that

RjLi13 · 2018-12-30T01:15:39Z

Looks like removing the xfail works, seems whatever issue arose from running this test (old dependency maybe?) is gone now.

TomAugspurger · 2018-12-30T16:54:21Z

Thanks @RjLi13!

* upstream/master: DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)

…strings * upstream/master: TST: Skip db tests unless explicitly specified in -m pattern (pandas-dev#24492) Mix EA into DTA/TDA; part of 24024 (pandas-dev#24502) DOC: Fix building of a single API document (pandas-dev#24506) DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)

…-dev#24480) * Fixing issue with test_parquet passing

Fixing issue with test_parquet passing

d9bc62d

TomAugspurger reviewed Dec 29, 2018

View reviewed changes

jreback added Testing pandas testing functions or related to the test suite IO Parquet parquet, feather labels Dec 29, 2018

jreback requested changes Dec 29, 2018

View reviewed changes

jreback added 2 commits December 29, 2018 09:40

add in fp and pyarrow

3a17930

Merge branch 'master' into PR_TOOL_MERGE_PR_24480

c3cbc8f

jreback added 2 commits December 29, 2018 11:22

Merge branch 'master' into PR_TOOL_MERGE_PR_24480

e3c29e2

remove macosx xfail

9f7e6aa

jreback added this to the 0.24.0 milestone Dec 29, 2018

jreback approved these changes Dec 29, 2018

View reviewed changes

Fix linting issue

ba384ba

removing xfail completely

41427a3

TomAugspurger merged commit 681e75c into pandas-dev:master Dec 30, 2018

RjLi13 deleted the fix_parquet_test branch December 31, 2018 00:16

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

TST- Fixing issue with test_parquet test unexpectedly passing (pandas…

1ecab59

…-dev#24480) * Fixing issue with test_parquet passing

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

TST- Fixing issue with test_parquet test unexpectedly passing (pandas…

7e89095

…-dev#24480) * Fixing issue with test_parquet passing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST- Fixing issue with test_parquet test unexpectedly passing #24480

TST- Fixing issue with test_parquet test unexpectedly passing #24480

RjLi13 commented Dec 29, 2018

codecov bot commented Dec 29, 2018

codecov bot commented Dec 29, 2018 •

edited

Loading

TomAugspurger Dec 29, 2018 •

edited

Loading

RjLi13 Dec 29, 2018 •

edited

Loading

jreback left a comment

jreback commented Dec 29, 2018

datapythonista commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 30, 2018

TomAugspurger commented Dec 30, 2018

		@@ -201,7 +201,8 @@ def test_options_get_engine(fp, pa):


		@pytest.mark.xfail(is_platform_windows() or is_platform_mac(),

TST- Fixing issue with test_parquet test unexpectedly passing #24480

TST- Fixing issue with test_parquet test unexpectedly passing #24480

Conversation

RjLi13 commented Dec 29, 2018

codecov bot commented Dec 29, 2018

Codecov Report

codecov bot commented Dec 29, 2018 • edited Loading

Codecov Report

TomAugspurger Dec 29, 2018 • edited Loading

Choose a reason for hiding this comment

RjLi13 Dec 29, 2018 • edited Loading

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented Dec 29, 2018

datapythonista commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 29, 2018

jreback commented Dec 29, 2018

RjLi13 commented Dec 30, 2018

TomAugspurger commented Dec 30, 2018

codecov bot commented Dec 29, 2018 •

edited

Loading

TomAugspurger Dec 29, 2018 •

edited

Loading

RjLi13 Dec 29, 2018 •

edited

Loading