Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#19431] Regression in make_block_same_class (tests failing for new fastparquet release) #19434

Merged
merged 23 commits into from
Jan 29, 2018
Merged

[#19431] Regression in make_block_same_class (tests failing for new fastparquet release) #19434

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
eb55928
udpate test_parquet since fastparquet now handles tz
minggli Jan 28, 2018
2f4fc07
bring back dtype kwarg because it is needed for DatetimeTZBlock
minggli Jan 28, 2018
e18f8e6
version dependence test_datetime_tz
minggli Jan 28, 2018
9a07d97
separate test cases for new and old behaviour of fastparquet
minggli Jan 28, 2018
3d9810d
tidy test_datetime_tz to test old behaviour of fastparquet<0.14
minggli Jan 28, 2018
ee75fdf
rephase reason to skip test case for oldder fp
minggli Jan 28, 2018
68d6324
follow pytest fixture pattern as in pyarrow
minggli Jan 28, 2018
985081d
follow pyarrow test_basic style for fastparquet new behaviour>=0.1.4
minggli Jan 28, 2018
0cfcd37
fastparquet=0.1.3
minggli Jan 28, 2018
0c4a6d7
other api change
minggli Jan 28, 2018
6ce68cf
fix typo
minggli Jan 28, 2018
f414743
deprecation warning for dtype in make_block_same_class.
minggli Jan 28, 2018
0d76fe7
Future warning for dtype in make_block_same_class.
minggli Jan 28, 2018
bb95dc6
update notes as fastparquet nows supports timezone
minggli Jan 28, 2018
d9a2e2a
remove fastparquet pin
minggli Jan 29, 2018
97b17e9
remove other api change as it is internal
minggli Jan 29, 2018
800b741
remove version
minggli Jan 29, 2018
52220b9
remove dtype for make_block and DeprecationWarning on make_block_same…
minggli Jan 29, 2018
c602a76
FutureWarning on make_block_same_class
minggli Jan 29, 2018
ddbbde3
test case for dtype and warning generation
minggli Jan 29, 2018
6e6b5f0
issue number and simplify test
minggli Jan 29, 2018
326394f
misc doc
minggli Jan 29, 2018
77422ba
use pandas warning assert
minggli Jan 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions pandas/core/internals.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ def array_dtype(self):
"""
return self.dtype

def make_block(self, values, placement=None, ndim=None):
def make_block(self, values, placement=None, ndim=None, dtype=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can add this back, but it needs to be deprecated anyhow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then please answer in the original issue how to achieve the same result (see the code example I posted there, to create a new DatetimeTZ block from a numpy array).
But I don't see why this needs to be deprecated. The dtype argument was actually doing something, in contrast to fastpath (which we therefore deprecated)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fp should be using make_block and not make_block_same_class

"""
Create a new block, with type inference propagate any values that are
not specified
Expand All @@ -216,20 +216,21 @@ def make_block(self, values, placement=None, ndim=None):
if ndim is None:
ndim = self.ndim

return make_block(values, placement=placement, ndim=ndim)
return make_block(values, placement=placement, ndim=ndim, dtype=dtype)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe remove dtype here? As fastparquet was not using it (and I don't think anybody else will)
Or otherwise add the same deprecation warning as you did for make_block_same_class


def make_block_scalar(self, values):
"""
Create a ScalarBlock
"""
return ScalarBlock(values)

def make_block_same_class(self, values, placement=None, ndim=None):
def make_block_same_class(self, values, placement=None, ndim=None,
dtype=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deprecate dtype here (add as a FutureWarning)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

""" Wrap given values in a block of same type as self. """
if placement is None:
placement = self.mgr_locs
return make_block(values, placement=placement, ndim=ndim,
klass=self.__class__)
klass=self.__class__, dtype=dtype)

def __unicode__(self):

Expand Down
13 changes: 9 additions & 4 deletions pandas/tests/io/test_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,13 +483,18 @@ def test_categorical(self, fp):
check_round_trip(df, fp)

def test_datetime_tz(self, fp):
# doesn't preserve tz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same need 2 tests (1 with each fixture)

Copy link
Contributor Author

@minggli minggli Jan 28, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyarrow is using similar style. there are already two tests for datetime_tz, one < 0.1.4 one >.

df = pd.DataFrame({'a': pd.date_range('20130101', periods=3,
tz='US/Eastern')})

# generic test data
df = pd.DataFrame({'dt': pd.date_range('20130101', periods=3)})

# fastparquet supports timezone since 0.1.4, and not before
import fastparquet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to do this in a separate test. Have 1 more fp <= 0.1.3 and 1 for fp > 0.1.3 (essentially duplicate the test with 1 working for the old version and 1 for the new)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to add 1 build with fp pinned to 0.1.3, the so pin in requirements-2.7.sh

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minggli we can indeed keep a test for the old behaviour (so only run for for older versions), but can you move the new behaviour to test_basic of this class (so we use the same pattern as in the pyarrow test), that is what I meant with my previous issue.

need to add 1 build with fp pinned to 0.1.3, the so pin in requirements-2.7.sh

@jreback We have one build for fastparquet 0.1.1 (that's the reason the tests were failing here in the original PR), so I would say that is enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for sure.

if LooseVersion(fastparquet.__version__) > LooseVersion('0.1.3'):
df['dt_tz'] = pd.date_range('20130101', periods=3, tz='US/Eastern')

# warns on the coercion
with catch_warnings(record=True):
check_round_trip(df, fp, expected=df.astype('datetime64[ns]'))
check_round_trip(df, fp)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to do this dependent on the fastparquet version, as older versions still return non-tz data (or we could skip the test for older versions)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or probably better: check how it is done for Arrow (test_basic in TestParquetPyarrow), it adds a tz column to the tested dataframe dependent on the version.


def test_filter_row_groups(self, fp):
d = {'a': list(range(0, 3))}
Expand Down