Floor and ceil methods during pandas.eval which are provided by numexpr #24355

anjsudh · 2018-12-19T17:11:51Z

closes Need support for floor and ceil methods during pandas.eval which are provided by numexpr #24353
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

pep8speaks · 2018-12-19T17:11:53Z

Hello @anjsudh! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on December 25, 2018 at 18:06 Hours UTC

TomAugspurger

Needs tests too :)

TomAugspurger · 2018-12-19T17:34:25Z

doc/source/whatsnew/v0.24.0.rst

@@ -372,6 +372,7 @@ Other Enhancements
 - :meth:`DataFrame.to_stata` and :class:`pandas.io.stata.StataWriter117` can write mixed sting columns to Stata strl format (:issue:`23633`)
 - :meth:`DataFrame.between_time` and :meth:`DataFrame.at_time` have gained the an ``axis`` parameter (:issue:`8839`)
 - :class:`IntervalIndex` has gained the :attr:`~IntervalIndex.is_overlapping` attribute to indicate if the ``IntervalIndex`` contains any overlapping intervals (:issue:`23309`)
+- :meth:`eval()` now supports use of `floor` and `ceil` in the expression


Add the issue number here. This will also work in DataFrame.query, right?

move this underneath this note:

- Added log10 to the list of supported functions in :meth:`DataFrame.eval` (:issue:`24139`)

codecov · 2018-12-19T18:09:44Z

Codecov Report

Merging #24355 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24355   +/-   ##
=======================================
  Coverage   92.29%   92.29%           
=======================================
  Files         162      162           
  Lines       51808    51808           
=======================================
  Hits        47817    47817           
  Misses       3991     3991

Flag	Coverage Δ
#multiple	`90.7% <ø> (ø)`	⬆️
#single	`42.99% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/computation/ops.py	`95.6% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14c33b0...6abf412. Read the comment docs.

codecov · 2018-12-19T18:09:44Z

Codecov Report

Merging #24355 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24355      +/-   ##
==========================================
+ Coverage   92.31%   92.31%   +<.01%     
==========================================
  Files         166      166              
  Lines       52412    52416       +4     
==========================================
+ Hits        48382    48386       +4     
  Misses       4030     4030

Flag	Coverage Δ
#multiple	`90.73% <100%> (ø)`	⬆️
#single	`43.06% <66.66%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/computation/check.py	`92.3% <100%> (+1.39%)`	⬆️
pandas/core/computation/ops.py	`95.63% <100%> (+0.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ebfd8a...3ccb40c. Read the comment docs.

jreback · 2018-12-19T22:58:57Z

doc/source/whatsnew/v0.24.0.rst

@@ -372,6 +372,7 @@ Other Enhancements
 - :meth:`DataFrame.to_stata` and :class:`pandas.io.stata.StataWriter117` can write mixed sting columns to Stata strl format (:issue:`23633`)
 - :meth:`DataFrame.between_time` and :meth:`DataFrame.at_time` have gained the an ``axis`` parameter (:issue:`8839`)
 - :class:`IntervalIndex` has gained the :attr:`~IntervalIndex.is_overlapping` attribute to indicate if the ``IntervalIndex`` contains any overlapping intervals (:issue:`23309`)
+- :meth:`eval()` now supports use of `floor` and `ceil` in the expression


move this underneath this note:

- Added log10 to the list of supported functions in :meth:`DataFrame.eval` (:issue:`24139`)

pandas/core/computation/ops.py

pandas/tests/computation/test_eval.py

jreback · 2018-12-24T19:27:06Z

pandas/core/computation/check.py


 try:
    import numexpr as ne
    ver = LooseVersion(ne.__version__)
    _NUMEXPR_INSTALLED = ver >= LooseVersion(_MIN_NUMEXPR_VERSION)
+    _ne_version_under_2_6_9 = ver < LooseVersion('2.6.9')


you don't need to add this here

rather export _NUMEXPR_VERSION as well

jreback · 2018-12-24T19:27:16Z

pandas/core/computation/expressions.py

@@ -20,6 +20,7 @@
 _TEST_MODE = None
 _TEST_RESULT = None
 _USE_NUMEXPR = _NUMEXPR_INSTALLED
+_ne_version_under2_6_9 = _ne_version_under_2_6_9


you don't need to add this here

jreback · 2018-12-24T19:32:28Z

pandas/core/computation/ops.py

@@ -23,8 +25,10 @@

 _unary_math_ops = ('sin', 'cos', 'exp', 'log', 'expm1', 'log1p',
                   'sqrt', 'sinh', 'cosh', 'tanh', 'arcsin', 'arccos',
-                   'arctan', 'arccosh', 'arcsinh', 'arctanh', 'abs', 'log10')
+                   'arctan', 'arccosh', 'arcsinh', 'arctanh', 'abs', 'log10',
+                   'floor', 'ceil')


just do this

_unary_math_ops = [...] if _NUMEXPR_INSTALLED and _NUMEXPR_VERSION >= LooseVersion('2.6.9'): _unary_math_ops += ['floor', 'ceil']

pandas/tests/computation/test_eval.py

anjsudh · 2018-12-25T14:44:50Z

Getting an error: "err: pandas should not import: numexpr". I haven't imported numexpr in my diff. The error resolves when I don't import _NUMEXPR_INSTALLED and _NUMEXPR_VERSION. Any ideas on how to go about the version check without the import?

Also, is there a reason for us to check that numexpr is not imported? The import is anyways in a try catch block

Any inputs would help here?

pandas/core/computation/ops.py

jreback · 2018-12-25T17:01:35Z

pandas/tests/computation/test_eval.py

@@ -31,6 +32,8 @@
                                 assert_numpy_array_equal, assert_series_equal,
                                 assert_produces_warning)
 from pandas.compat import PY3, reduce
+from pandas.core.computation.check import _NUMEXPR_VERSION


add this on to line 20 above

jreback · 2018-12-25T17:17:58Z

lgtm. ping on green.

jreback · 2018-12-25T17:54:12Z

you have a lint error: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=5788

jreback · 2018-12-25T23:55:17Z

pandas/core/computation/ops.py

@@ -14,6 +15,7 @@
 import pandas as pd
 from pandas.core.base import StringMixin
 import pandas.core.common as com
+from pandas.core.computation.check import _NUMEXPR_INSTALLED, _NUMEXPR_VERSION


@anjsudh ok, the problem is we can't import here (we have a check in code_checks to avoid importing the optional dependencies until they are needed).

So. I think just add floor & ceil to the _math_ops (always) and better to do a run-time check if this op exists in ne (not exactly sure how to do that).

jreback · 2018-12-28T21:26:47Z

can you merge master and update

removed unused imports

This reverts commit 5933d5e.

anjsudh · 2018-12-30T19:22:56Z

@jreback Hope you can have a look?

jreback · 2018-12-30T19:29:04Z

thanks @anjsudh

yeah this was a bit tricky because of our delayed import of numexpr :>

* upstream/master: DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)

…strings * upstream/master: TST: Skip db tests unless explicitly specified in -m pattern (pandas-dev#24492) Mix EA into DTA/TDA; part of 24024 (pandas-dev#24502) DOC: Fix building of a single API document (pandas-dev#24506) DOC: Fixing broken references in the docs (pandas-dev#24497) DOC: Splitting api.rst in several files (pandas-dev#24462) Fix misdescription in escapechar (pandas-dev#24490) Floor and ceil methods during pandas.eval which are provided by numexpr (pandas-dev#24355) BUG: Pandas any() returning false with true values present (GH pandas-dev#23070) (pandas-dev#24434) Misc separable pieces of pandas-dev#24024 (pandas-dev#24488) use capsys.readouterr() as named tuple (pandas-dev#24489) REF/TST: replace capture_stderr with pytest capsys fixture (pandas-dev#24496) TST- Fixing issue with test_parquet test unexpectedly passing (pandas-dev#24480) DOC: Doc build for a single doc made much faster, and clean up (pandas-dev#24428) BUG: Fix+test timezone-preservation in DTA.repeat (pandas-dev#24483) Implement reductions from pandas-dev#24024 (pandas-dev#24484)

…pr (pandas-dev#24355)

anjsudh force-pushed the master branch from bad9d3e to 6abf412 Compare December 19, 2018 17:12

TomAugspurger reviewed Dec 19, 2018

View reviewed changes

WillAyd changed the title ~~closes #24353~~ Floor and ceil methods during pandas.eval which are provided by numexpr Dec 19, 2018

jreback requested changes Dec 19, 2018

View reviewed changes

jreback added the Numeric Operations Arithmetic, Comparison, and Logical operations label Dec 19, 2018

jreback requested changes Dec 23, 2018

View reviewed changes

pandas/core/computation/ops.py Outdated Show resolved Hide resolved

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

anjsudh force-pushed the master branch from 062923f to 42bc642 Compare December 24, 2018 19:28

jreback requested changes Dec 24, 2018

View reviewed changes

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

pandas/tests/computation/test_eval.py Outdated Show resolved Hide resolved

jreback requested changes Dec 25, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Dec 25, 2018

jreback approved these changes Dec 25, 2018

View reviewed changes

jreback requested changes Dec 25, 2018

View reviewed changes

anjsudh force-pushed the master branch 3 times, most recently from de48cc2 to bf64953 Compare December 30, 2018 09:35

anjsudh added 7 commits December 30, 2018 16:00

closes pandas-dev#24353

6d7834d

update whatsnew

e92bd07

raise error when function not supported by older version of numpy

f5b7c0e

Fix PEP8 issue

eb6c621

Fix compilation issue in py 2.7

7e8a643

fix linting issue

693449d

numexpr less than 2.6.9

13568a6

anjsudh added 10 commits December 30, 2018 16:00

refactored to pass _NUMEXPR_VERSION

f963077

using self.unary_fns

c8a8a02

removed unused imports

fix linting issue

bdd9c68

restored sys modules after import

1e6ea3d

remove unused ne_gt_2_6_9

a3a4b9e

Revert "restored sys modules after import"

bc154e2

This reverts commit 5933d5e.

exporting _NUMEXPR_VERSION as well

caaf634

fix imports

bb07d26

fixed imports

38115f6

Fix numexpr version issue

3ccb40c

anjsudh force-pushed the master branch from bf64953 to 3ccb40c Compare December 30, 2018 10:30

jreback approved these changes Dec 30, 2018

View reviewed changes

jreback merged commit 2d1d697 into pandas-dev:master Dec 30, 2018

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Floor and ceil methods during pandas.eval which are provided by numex…

aaebb2b

…pr (pandas-dev#24355)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Floor and ceil methods during pandas.eval which are provided by numex…

c74eacc

…pr (pandas-dev#24355)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Floor and ceil methods during pandas.eval which are provided by numexpr #24355

Floor and ceil methods during pandas.eval which are provided by numexpr #24355

anjsudh commented Dec 19, 2018 •

edited by TomAugspurger

Loading

pep8speaks commented Dec 19, 2018 •

edited

Loading

TomAugspurger left a comment

TomAugspurger Dec 19, 2018

jreback Dec 19, 2018

codecov bot commented Dec 19, 2018

codecov bot commented Dec 19, 2018 •

edited

Loading

jreback Dec 19, 2018

jreback Dec 24, 2018

jreback Dec 24, 2018

jreback Dec 24, 2018

jreback Dec 24, 2018

anjsudh commented Dec 25, 2018

jreback Dec 25, 2018

jreback commented Dec 25, 2018

jreback commented Dec 25, 2018

jreback Dec 25, 2018

jreback commented Dec 28, 2018

anjsudh commented Dec 30, 2018

jreback commented Dec 30, 2018

Floor and ceil methods during pandas.eval which are provided by numexpr #24355

Floor and ceil methods during pandas.eval which are provided by numexpr #24355

Conversation

anjsudh commented Dec 19, 2018 • edited by TomAugspurger Loading

pep8speaks commented Dec 19, 2018 • edited Loading

Comment last updated on December 25, 2018 at 18:06 Hours UTC

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 19, 2018

Codecov Report

codecov bot commented Dec 19, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anjsudh commented Dec 25, 2018

Choose a reason for hiding this comment

jreback commented Dec 25, 2018

jreback commented Dec 25, 2018

Choose a reason for hiding this comment

jreback commented Dec 28, 2018

anjsudh commented Dec 30, 2018

jreback commented Dec 30, 2018

anjsudh commented Dec 19, 2018 •

edited by TomAugspurger

Loading

pep8speaks commented Dec 19, 2018 •

edited

Loading

codecov bot commented Dec 19, 2018 •

edited

Loading