read_csv character encoding bug?

[hayd]

This is a weird one from StackOverflow, this file has some \x00s which seem to be ignored when printing but confuse read_csv:

x = 'x,y\n \x00\x00\x00,Reg\n \x00\x00\x00,Reg\nI,Swp\nI,Swp\n'
X = StringIO(x)

In [3]: pd.read_csv(X)
Out[3]: 
     x    y
0          
1  NaN  NaN
2    I  Swp
3    I  Swp

In [4]: print x
x,y
 ,Reg
 ,Reg
I,Swp
I,Swp

[wesm]

Yes. The tokenizer uses null terminators in a couple of places as a marker, I'll have to look to see exactly why this is failing.

[wesm]

In complete fairness the csv module doesn't handle NULL bytes:

In [6]: import csv; f = csv.reader(StringIO(x))

In [7]: next(f)
Out[7]: ['x', 'y']

In [8]: next(f)
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-8-468f0afdf1b9> in <module>()
----> 1 next(f)

Error: line contains NULL byte

Pushing this issue to some other day (not 0.11)

[msiler]

I know this is an old issue, but I'd like to give it a little bump. I have customers giving us a dump of some of their production database tables. The files we are getting are tab delimited and values that were null in the database are null bytes in the text file. I don't know how to get pandas to read this without having to do some manual munging first.

[wesm]

I'm sorry this never got fixed! I have also seen null bytes used to code nulls. @jreback @jorisvandenbossche let's slate this for 0.18?

[wesm]

I'd like to make some changes to the CSV tokenizer to hopefully improve its performance, that will also allow us to get rid of the null terminators that are complicating issues like this, but it would make most sense to understand for 1.0 / libpandas. Question is whether there is a quick fix for this particular issue with the existing tokenization strategy

[gfyoung]

I don't believe this is an issue with the C engine anymore:

>>> from pandas.compat import StringIO
>>> from pandas import read_csv
>>> data = 'x,y\n\x00\x00\x00,Reg\n\x00\x00\x00,Reg\nI,Swp\nI,Swp\n'
>>> read_csv(StringIO(data), engine='c')
     x    y
0  NaN  Reg
1  NaN  Reg
2    I  Swp
3    I  Swp

Unfortunately, as @wesm pointed out here, it still does fail with the Python engine:

>>> read_csv(StringIO(data), engine='python')
...
_csv.Error: line contains NULL byte

This issue however, seems beyond our control, so I'm not sure if we should still classify this as a BUG on the pandas end if the issue is originating in Python's csv module.

[jreback]

if u want to put up tests for the c engine and s nice error message Python engine then can close

[wesm]

Relatedly: what is the current implementation gap between the C and pure Python CSV parsers?

[gfyoung]

@wesm: We have a list of known differences in #12686. That alone indicates a pretty noticeable gap.

[wesm]

Seems like if you can address the regex delimiter problem (easier said than done) then it may be possible to deprecate the Python engine. This would be easier in the possible pandas 2.0 future in which we might add libre2 to the build / development toolchain