BUG: Don't over-optimize memory with jagged CSV #23527
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hello @gfyoung! Thanks for submitting the PR.
|
Codecov Report
@@ Coverage Diff @@
## master #23527 +/- ##
=======================================
Coverage 92.23% 92.23%
=======================================
Files 161 161
Lines 51324 51324
=======================================
Hits 47339 47339
Misses 3985 3985
Continue to review full report at Codecov.
|
|
@jreback : Any thoughts on this one? (I only removed the |
|
can you run the current benchmarks on this. do we have any that specifically target this? |
|
@jreback : No specific benchmarks as far as know, though I did not observe any meaningful changes to the benchmarks after making this change. |
|
@jreback : Any other thoughts on this? |
gfyoung
force-pushed
the
jagged-csv-buffer-overflow
branch
from
feccd27 to
015a193
Compare
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks. Closes pandas-devgh-23509.
gfyoung
force-pushed
the
jagged-csv-buffer-overflow
branch
from
015a193 to
17f7822
Compare
|
@jreback : Addressed the doc comments, and all is still green. PTAL. |
jreback
approved these changes
Nov 12, 2018
|
thanks @gfyoung |
thoo
added a commit
to thoo/pandas
that referenced
this pull request
Nov 12, 2018
* upstream/master: BUG: Don't over-optimize memory with jagged CSV (pandas-dev#23527) DEPR: Deprecate usecols as int in read_excel (pandas-dev#23635) More helpful Stata string length error. (pandas-dev#23629) BUG: astype fill_value for SparseArray.astype (pandas-dev#23547) CLN: datetimelike arrays: isort, small reorg (pandas-dev#23587) CI: Check in the CI that assert_raises_regex is not being used (pandas-dev#23627) CLN:Remove unused **kwargs from user facing methods (pandas-dev#23249) DOC: Enhancing pivot / reshape docs (pandas-dev#21038) TST: Fix xfailing DataFrame arithmetic tests by transposing (pandas-dev#23620)
thoo
added a commit
to thoo/pandas
that referenced
this pull request
Nov 12, 2018
…fixed * upstream/master: DOC: avoid SparseArray.take error (pandas-dev#23637) CLN: remove incorrect usages of com.AbstractMethodError (pandas-dev#23625) DOC: Adding validation of the section order in docstrings (pandas-dev#23607) BUG: Don't over-optimize memory with jagged CSV (pandas-dev#23527) DEPR: Deprecate usecols as int in read_excel (pandas-dev#23635) More helpful Stata string length error. (pandas-dev#23629) BUG: astype fill_value for SparseArray.astype (pandas-dev#23547) CLN: datetimelike arrays: isort, small reorg (pandas-dev#23587) CI: Check in the CI that assert_raises_regex is not being used (pandas-dev#23627) CLN:Remove unused **kwargs from user facing methods (pandas-dev#23249)
JustinZhengBC
pushed a commit
to JustinZhengBC/pandas
that referenced
this pull request
Nov 14, 2018
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks. Closes pandas-devgh-23509.
tm9k1
pushed a commit
to tm9k1/pandas
that referenced
this pull request
Nov 19, 2018
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks. Closes pandas-devgh-23509.
gfyoung
added a commit
to forking-repos/pandas
that referenced
this pull request
Jan 19, 2019
The edge case where we hit powers of 2 every time during allocation can be painful. Closes pandas-devgh-24805. xref pandas-devgh-23527.
gfyoung
added a commit
to forking-repos/pandas
that referenced
this pull request
Jan 20, 2019
The edge case where we hit powers of 2 every time during allocation can be painful. Closes pandas-devgh-24805. xref pandas-devgh-23527.
jreback
pushed a commit
that referenced
this pull request
Jan 20, 2019
Pingviinituutti
pushed a commit
to Pingviinituutti/pandas
that referenced
this pull request
Feb 28, 2019
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks. Closes pandas-devgh-23509.
Pingviinituutti
pushed a commit
to Pingviinituutti/pandas
that referenced
this pull request
Feb 28, 2019
* Fix memory growth bug in read_csv The edge case where we hit powers of 2 every time during allocation can be painful. Closes pandas-devgh-24805. xref pandas-devgh-23527. * TST: Add ASV benchmark for issue
Pingviinituutti
pushed a commit
to Pingviinituutti/pandas
that referenced
this pull request
Feb 28, 2019
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks. Closes pandas-devgh-23509.
Pingviinituutti
pushed a commit
to Pingviinituutti/pandas
that referenced
this pull request
Feb 28, 2019
* Fix memory growth bug in read_csv The edge case where we hit powers of 2 every time during allocation can be painful. Closes pandas-devgh-24805. xref pandas-devgh-23527. * TST: Add ASV benchmark for issue
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With jagged CSV's, we risk being too quick to dump memory that we need to allocate because previous chunks would have indicated much larger rows than we can anticipate in subsequent chunks.
Closes #23509.