-
-
Notifications
You must be signed in to change notification settings - Fork 18.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: Patch Checked Add Method #14324
Conversation
Current coverage is 85.25% (diff: 90.47%)@@ master #14324 diff @@
==========================================
Files 140 140
Lines 50634 50645 +11
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 43173 43175 +2
- Misses 7461 7470 +9
Partials 0 0
|
Although this is probably more correct, I don't think the performance degradation is allowable:
The problem is in |
@jorisvandenbossche : I draw a hard line on correctness. This is impeding my ability to patch the other TDI overflow issues because there are tests that fail because of this bug. Perhaps it would instead be worthwhile to break the check up into two separate statements, though not sure that would help too much given that it's Python. |
try this. The key to this expression is lazy evaluation as its a compound expression.
|
numba can do even better (but we don't have this in the quiver atm)
|
Something like this is quite convoluted, but is a bit faster as the eval approach:
|
9d31ec7
to
ad4cd85
Compare
@jorisvandenbossche : Are you proposing I use this more convoluted method then? Thoughts? |
c395af3
to
760b1bb
Compare
@jorisvandenbossche : Some feedback on my question above would be greatly appreciated. |
@gfyoung yes pls use one of the methods proposed above |
d0563a7
to
cf4da12
Compare
@jorisvandenbossche : Your solution is unfortunately not backwards compatible with older versions of |
so put a conditional on the numpy version (or use the eval soln) |
@jreback : I thought we don't do those kind of checks (or try to avoid them). In any case, I'm trying with another method as I mentioned above. IMO this isn't something THAT important that we need to check |
i don't see a reason not to do a check in the numpy version or use the eval soln (which is fine but will fall back to numpy if numexpr is not installed) |
@jreback : Fair enough. Also, I have no objections to your eval solution. I'm trying it right now, as |
cf4da12
to
08b6395
Compare
The |
@jorisvandenbossche : Unfortunately, your solution is not backwards compatible. And if you don't like @jreback suggestion, then what do you suggest? Compared to my original change, the |
just check the numpy version i am not sure why you think this is an issue |
@jreback : And then what, use the slower solution that I used before if the version is not compatible? Would it be preferable to fallback to my original solution or use the |
yep |
…projects (pandas-dev#14406) As per [their blog post of the 27th April](https://blog.readthedocs.com/securing-subdomains/) ‘Securing subdomains’: > Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard. Test Plan: Manually visited all the links I’ve modified.
896c9f1
to
ba515fe
Compare
This looks good to me. |
lgtm |
@jorisvandenbossche : Using the same setup as above, 100 loops, best of 3: 3.08 ms per loop |
And compared to 0.18? (on my machine I get 1.15 ms, so probably something of 3x slowdown) |
ba515fe
to
ee1653b
Compare
@jorisvandenbossche : That timing information is for |
@gfyoung That is a bit strange, because then you have 4.06 for this PR and 3.08 for 0.18. I get 5.11 vs 1.11, so a much larger difference. Anyway, it's a rather large performance degredation, but we decided to go down this path. |
ee1653b
to
de31e6a
Compare
…x (GH14327) (pandas-dev#14428) Existing logic under "if level is not None:" assumed that index was a MultiIndex. Now we check and also handle the case where an Index is passed in with a None grouper. This resolves GH 14327
Deprecated back in `v0.17.0` <a href="https://github.com/pydata/pandas /commit/987b7e7e586b8df1d127406c69e0a9094a1a5322">here</a>. Seems appropriate to carry though now. Author: gfyoung <[email protected]> Closes pandas-dev#13602 from gfyoung/remove-coerce-to-timedelta and squashes the following commits: 091af83 [gfyoung] CLN: Removed coerce param in pd.to_timedelta
closes pandas-dev#14369 Author: Brandon M. Burroughs <[email protected]> Closes pandas-dev#14416 from brandonmburroughs/concat_with_axis_rows and squashes the following commits: 9aa260d [Brandon M. Burroughs] Changing axis handling to depend on object passed 49442be [Brandon M. Burroughs] Using dataframe _get_axis_number instance method 64702fb [Brandon M. Burroughs] Updating documentation for concat fdd5260 [Brandon M. Burroughs] Removing duplicate expected dfs 3f08b07 [Brandon M. Burroughs] Adding concat tests for axis 0, 1, and 'index' cf3f998 [Brandon M. Burroughs] Adding ValueError test for concat Series axis 'columns' a6694b9 [Brandon M. Burroughs] Updating documentation 584ebd2 [Brandon M. Burroughs] BUG: Allow concat to take string axis names
de31e6a
to
94076aa
Compare
1) Add checks to ensure that add overflow does not occur both in the positive or negative directions. 2) Add benchmarks to ensure that operations involving this checked add function are significantly impacted.
94076aa
to
85ca49d
Compare
@gfyoung you may need to close this and open a new PR. It seems that people are unable to rebase and push correctly to the original pandas repo. (further appevey was disconnected) so might be a good test. |
@jreback : Ah, I can see that now. Will close and repost. |
@jreback BTW, I contacted github about it, and it should be solved now (was an error on their side). So the closing/open new PR should no longer be needed for all older PRs. |
@jorisvandenbossche good to know! thanks! |
A little to late now but oh well. 😄 New PR in #14453. |
Title is self-explanatory. Follow-up to #14237.
Merged in PR #14453