Series.str.rsplit not working with regex patterns (v 0.24.2 and 0.25.2)

[jamespreed]

Code Sample, a copy-pastable example if possible

import pandas as pd

S = pd.Series(['1+1=2'])
S.str.rsplit(r'\+|=')
# returns:
0    [1+1=2]

S.str.rsplit(r'\+|=', expand=True)
# returns:
       0
0  1+1=2

Problem description

The str.rsplit method is not recognizing regex patterns. This example above is from the documentation.

Expected Output

S.str.rsplit(r'\+|=')
0    [1, 2, 3]

S.str.rsplit(r'\+|=', expand=True)
     0    1    2
0    1    1    2

Output of pd.show_versions()

Version 0.24.2:

INSTALLED VERSIONS ------------------ commit: None python: 3.7.3.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

pandas: 0.24.2
pytest: 5.0.1
pip: 19.1.1
setuptools: 41.0.1
Cython: 0.29.12
numpy: 1.16.4
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.6.1
sphinx: 2.1.2
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: 1.2.1
tables: 3.5.2
numexpr: 2.6.9
feather: None
matplotlib: 3.1.0
openpyxl: 2.6.2
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.8
lxml.etree: 4.3.4
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.3.5
pymysql: None
psycopg2: None
jinja2: 2.10.1
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

Version 0.25.2:

INSTALLED VERSIONS ------------------ commit : None python : 3.6.7.final.0 python-bits : 64 OS : Windows OS-release : 10 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : None.None

pandas : 0.25.2
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.8.0
pandas_datareader: None
bs4 : 4.8.1
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[asishm]

Had a look at it,

the rsplit method (https://github.com/pandas-dev/pandas/blob/master/pandas/core/strings.py#L2535-L2539) calls str_rsplit (

pandas/pandas/core/strings.py

Lines 1407 to 1413 in 94412ee

	def str_rsplit(arr, pat=None, n=None):
	if n is None or n == 0:
	n = -1
	f = lambda x: x.rsplit(pat, n)
	res = _na_map(f, arr)
	return res

) which just seems to be applying str.rsplit which of course doesn't use regex.

[jamespreed]

Yeah, I saw that as well. Need to either update the function or update the docs.

[jamespreed]

This is not the most efficient method, but it does accomplish the regex rsplit by regex splitting the string, grabbing the n last results, then finding the index in the string of the last character of the last non-included results and slicing the string to that location.

def str_rsplit(arr, pat=None, n=None):
    
    if pat is None or len(pat) == 1:
        if n is None or n == 0:
            n = -1
        f = lambda x: x.rsplit(pat, n)
    else:
        if n is None or n == -1:
            n = 0
        regex = re.compile(pat)
        def f(x):
            s = regex.split(x)
            a, b = s[:-n], s[-n:]
            if not a:
                return b
            ix = 0
            for a_ in a:
                ix = x.find(a_, ix) + len(a_)
            x_ = [x[:ix]]
            return x_ + b
    return f
    res = _na_map(f, arr)
    return res

Here is a test similar to that in the documentation:

for n in range(-1, 6):
    print(f'#{n:>2}:', str_rsplit(None,  r'\+|=', n=n)('1+1+1+1=4'))

#-1: ['1', '1', '1', '1', '4']
# 0: ['1', '1', '1', '1', '4']
# 1: ['1+1+1+1', '4']
# 2: ['1+1+1', '1', '4']
# 3: ['1+1', '1', '1', '4']
# 4: ['1', '1', '1', '1', '4']
# 5: ['1', '1', '1', '1', '4']