Replace codecs with TextIOWrapper to fix newline issues when reading text files

[markopy]

Title

Use TextIOWrapper instead of codecs to fix various newline issues when opening files in text mode.

Motivation

smart_open currently uses the codecs module to read text files which incorrectly splits lines as described in #557. This patch replaces codecs with TextIOWrapper fixing those issues and making the newline parameter work as expected for all files. Previously it was ignored unless the file was local and the builtin open was used.

• Fixes Line splitting on \u2028 from S3 #557
• Fixes A S3 csv file UTF-8 file with invalid \x1c characters being interpreted as new lines #536
• Fixes inconsistent with bz2.open on files containing vertical tab ^K #394
• Fixes Incorrect endlines handling with compressed input #269

Tests

New tests test_newline_read and test_newline_write have been added to ensure behavior of the newline parameter is the same as for python's builtin open.

Unicode line separator has been added to several other test for readline() to ensure line separation behavior is correct for differnent use cases.

Benchmarks

There seems to be no practical speed difference when reading read/writing text files on S3.

master branch:
---------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------
Name (time in ms)                    Min                 Max                Mean             StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_s3_readwrite_text_gzip     220.6736 (1.0)      781.1166 (1.06)     317.5370 (1.00)     76.3814 (1.10)     300.2566 (1.02)     78.3614 (1.16)         16;2  3.1492 (1.00)        100           1
test_s3_readwrite_text          228.1418 (1.03)     735.1833 (1.0)      316.9454 (1.0)      69.3133 (1.0)      295.4379 (1.0)      67.6500 (1.0)          14;5  3.1551 (1.0)         100           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

this PR:
---------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------
Name (time in ms)                    Min                 Max                Mean             StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_s3_readwrite_text          227.0207 (1.0)      595.1862 (1.13)     312.4748 (1.01)     61.7262 (1.09)     298.0243 (1.01)     79.3987 (1.10)         20;3  3.2003 (0.99)        100           1
test_s3_readwrite_text_gzip     228.5019 (1.01)     528.6511 (1.0)      308.6282 (1.0)      56.8423 (1.0)      295.3955 (1.0)      72.4423 (1.0)          22;2  3.2401 (1.0)         100           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[mpenkov]

Looks like flake8 picked up some boo-boos in your code. Can you please have a look and fix them?

[markopy]

There seem to be some issues with the windows tests. I need to look at this in more detail but it will be hard for me to debug if the solution is not obvious since I don't have any windows systems around.

[markopy]

Seems all good now, the two tests failing on windows are now fixed.

The pathlib test didn't consider universal newline mode which is the default with python 3. It only worked before because smart_open didn't correctly implement this.

The csv test failed because the encoding wasn't explicitly specified and defaulted to cp852 on windows which broke the test when I added the unicode newline separator.

[mpenkov]

OK, looks great. Thank you for your contribution @markopy !

[piskvorky]

This is great, thanks @markopy!

[markopy]

Thanks. Going back to my usual fare, ping me if there are any issues with this in the wild.