Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add chunksize param to read_json when lines=True #17168

Merged
merged 62 commits into from
Sep 28, 2017
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
4cd506e
Add chunksize param to read_json when lines=True
louispotok Jul 24, 2017
9fe44f1
Add read_json chunksize change to whatsnew
louispotok Aug 4, 2017
6de3a27
Add versionadded to docstring
louispotok Aug 4, 2017
e235c70
add docstring for _read_json_as_lines
louispotok Aug 4, 2017
0a5a8f9
add basic read_json chunksize test
louispotok Aug 4, 2017
ce23444
validate read_json chunksize is an integer and >=1
louispotok Aug 4, 2017
a97ca0b
Add more tests to read_json chunksize
louispotok Aug 4, 2017
2861d0e
Return JsonLineReader from read_json
louispotok Aug 13, 2017
da59b4a
Raise ValueError if chunksize is not None, but not lines
louispotok Aug 13, 2017
4544f82
Add issue number to test docstring and add test.
louispotok Aug 13, 2017
dad7f11
bugfix: raise StopIteration, dont return it
louispotok Aug 13, 2017
bb8b1b6
PEP8 cleanup
louispotok Aug 13, 2017
3e81bba
Bugfixes for chunksize
louispotok Aug 14, 2017
e049d29
add chunksize test for reading from file
louispotok Aug 14, 2017
400d313
pep8 cleanup
louispotok Aug 14, 2017
b756c90
Run chunksize checks before file is opened
louispotok Aug 14, 2017
b71f65b
move strio df in test to fixture
louispotok Sep 10, 2017
d6e86af
Improve read_json chunking tests
louispotok Sep 10, 2017
4d91280
bugfix in read_json tests, remove fixture
louispotok Sep 10, 2017
2474429
JsonLineReader opens and closes filepaths
louispotok Sep 10, 2017
b18b3df
pep8 cleanup
louispotok Sep 12, 2017
4c1d6a6
update whatsnew
louispotok Sep 13, 2017
d589b0b
update docs on read_json chunksize
louispotok Sep 14, 2017
eba45a2
Always use JsonReader in read_json
louispotok Sep 14, 2017
d0ea295
make lines_json_df a fixture
louispotok Sep 14, 2017
bb3182d
remove unneeded concats in tests
louispotok Sep 14, 2017
8cc43ff
parametrize some tests
louispotok Sep 14, 2017
de03462
add __close__ method to JsonReader and use it
louispotok Sep 15, 2017
07b31c7
remove import io in docs
louispotok Sep 15, 2017
7d0642f
move read_json in whatsnew to Other Enhancements
louispotok Sep 15, 2017
398961b
move chunksize and lines validation into JsonReader
louispotok Sep 15, 2017
dfa2967
remove extraneous else
louispotok Sep 15, 2017
b0e4bb0
remove unneccessary cast to list
louispotok Sep 15, 2017
e3197c5
move combine_lines call into read
louispotok Sep 15, 2017
39f9881
remove another extraneous else
louispotok Sep 15, 2017
c2247c3
always close JsonReader
louispotok Sep 15, 2017
46d8a68
add test that read_json closes file correctly
louispotok Sep 15, 2017
066e26d
minor formatting fixups
louispotok Sep 15, 2017
08e8b6c
remove extraneous else
louispotok Sep 15, 2017
1ac6953
add benchmarks for read_json
louispotok Sep 15, 2017
0782df9
update benchmarks
louispotok Sep 15, 2017
014d493
move json_lines tests to io_bench
louispotok Sep 15, 2017
a913d8e
add peakmem for jsonlines
louispotok Sep 15, 2017
ce7aef6
smaller benchmark
louispotok Sep 15, 2017
1dc1526
refactor JsonReader
louispotok Sep 21, 2017
03b6069
add test for reading with multiple empty lines
louispotok Sep 21, 2017
aef6bbc
add support for JSON docs with multiple consecutive newlines
louispotok Sep 21, 2017
30e4043
remove raw_json init param
louispotok Sep 21, 2017
7dae78a
DRY for combining lines
louispotok Sep 21, 2017
fe95445
use floor division in asv bench
louispotok Sep 21, 2017
e41124a
add teardown to asv bench
louispotok Sep 21, 2017
9cfd012
add docs
louispotok Sep 21, 2017
035ca84
pep fixup
louispotok Sep 21, 2017
4c92287
update documentation
louispotok Sep 22, 2017
61178be
simplify JsonReader._preprocess_data
louispotok Sep 22, 2017
a284187
simplify _get_data_from_filepath
louispotok Sep 22, 2017
55170dd
Update read_json tests
louispotok Sep 22, 2017
1d7087d
JsonReader should only close if it opened
louispotok Sep 22, 2017
6a76c55
split out json readlines to sep test class
louispotok Sep 26, 2017
a72411f
add encoding to test_readlines
louispotok Sep 26, 2017
5612934
pep8 cleanup
louispotok Sep 27, 2017
28d1cbe
minor fixups
louispotok Sep 27, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pep8 cleanup
louispotok committed Sep 28, 2017
commit 56129344586c84d32359dcb07cd033d448b7c90b
2 changes: 0 additions & 2 deletions pandas/tests/io/json/test_pandas.py
Original file line number Diff line number Diff line change
@@ -1037,5 +1037,3 @@ def test_data_frame_size_after_to_json(self):
size_after = df.memory_usage(index=True, deep=True).sum()

assert size_before == size_after