[BUG] Regex: $ does not match \n at end of string

[andygrove]

Describe the bug
There is an edge case with multiline / EOL matching that seems to be incorrect in cuDF.

Given the pattern 2$ I would expect it to match the inputs 2 and 2\n as seen in Python:

>>> print(re.compile('2$').search("2"))
<re.Match object; span=(0, 1), match='2'>

>>> print(re.compile('2$').search("2\n"))
<re.Match object; span=(0, 1), match='2'>

However, cuDF does not match in the 2\n case.

Steps/Code to reproduce bug

I don't know how to use cuDF from the Python repl with the latest code so I have not actually tested this, but this should be the repro case.

cudf.Series(['2\n']).str.contains('2$', regex=True)

Expected behavior
$ should match EOL even if the input ends with a line terminator.

Environment overview (please complete the following information)
N/A

Environment details
N/A

Additional context
None

[davidwendt]

The default behavior of contains changed in #9482 in regards to ^ and $ and the new-line character as explained in that PR. To make ^ and $ match with new-line characters instead of the whole string requires passing the MULTILINE flags parameter.

>>> import cudf
>>> import re
>>> cudf.Series(['2\n']).str.contains('2$', flags=re.MULTILINE, regex=True)
0    True
dtype: bool

The flags are defined here

cudf/cpp/include/cudf/strings/regex/flags.hpp

Lines 35 to 39 in eda31b6

	enum regex_flags : uint32_t {
	DEFAULT = 0, /// default
	MULTILINE = 8, /// the '^' and '$' honor new-line characters
	DOTALL = 16 /// the '.' matching includes new-line characters
	};

[andygrove]

I think I could have done a better job explaining this issue. What I am looking for is to have the default cuDF behavior (MULTILINE disabled) match Python's default behavior (also MULTILINE disabled), as described here. The key point is around how $ behaves when the string ends with \n:

By default, '^' matches only at the beginning of the string, and '$' only at the end of the string and immediately before the newline (if any) at the end of the string.

Does that make sense?

[davidwendt]

Ok, got it, thanks. This is some wacky, yet documented edge case.

... By default, '^' matches only at the beginning of the string, and '$' only at the end of the string and immediately before the newline (if any) at the end of the string.