str.cat not working with binary data on Python3

[h-vetinari]

In Python 2, the following works:

s = pd.Series(np.array(list('abc'), 'S1').astype(object))
t = pd.Series(np.array(list('def'), 'S1').astype(object))
s.str.cat(t)
# 0    b'ad'
# 1    b'be'
# 2    b'cf'
# dtype: object

whereas in Python 3, s.str.cat(t) yields:

TypeError: sequence item 0: expected str instance, bytes found

Clearly, Python 3 fundamentally changed the separation between bytes and strings, but in principle, there is no reason (other than implementation) why this should not have the same result in Python 3 as in Python 2.

[WillAyd]

Not sure I agree here at the .str accessor is for handling strings not bytes. Let's see what others think though

[jreback]

yeah this is correct
.str is strictly for strings and not bytes

[h-vetinari]

@WillAyd @jreback

The example is out of pandas/tests/test_strings.test_method_on_bytes - it works on Python 2. I see the point that Python 2 didn't have the right separation between bytes/str and I'm fine with not having this functionality. I was just working on this functionality by refactoring the str.cat internals (#22725), and found that this now works in Python 3 as well (so I opened this issue before the PR).

Beyond that, it's IMO the .str-accessor that should be enforcing the correct types, but since there's no dedicated string-dtype yet, the methods that do work for other object data (e.g. lists) are also used like that (e.g. people use .str.len() to get the different length of a Series of lists).

I guess the better place to have this discussion is #22725, because the implementation would have to be adapted to restore the current test behaviour.

[h-vetinari]

@jreback
I haven't edited the OP of #22725 yet, but I'll open another issue to supersede this based on your review that the .str accessor should not work for bytes.

In other words, this does not need the 0.24 milestone, IMO

[h-vetinari]

Superseded by #23011