Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support some escape chars when rewriting regexp_replace to stringReplace #11813

Merged
merged 4 commits into from
Dec 5, 2024
Merged

Support some escape chars when rewriting regexp_replace to stringReplace #11813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b48335a
Support some escape characters in search list when rewriting regexp_r…
thirtiseven Dec 3, 2024
8b36064
add a case
thirtiseven Dec 4, 2024
3305c8b
address comment
thirtiseven Dec 4, 2024
9ee442f
update datagen
thirtiseven Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions integration_tests/src/main/python/regexp_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1012,14 +1012,16 @@ def test_regexp_replace_simple(regexp_enabled):
'REGEXP_REPLACE(a, "ab", "PROD")',
'REGEXP_REPLACE(a, "ae", "PROD")',
'REGEXP_REPLACE(a, "bc", "PROD")',
'REGEXP_REPLACE(a, "fa", "PROD")'
'REGEXP_REPLACE(a, "fa", "PROD")',
'REGEXP_REPLACE(a, "a\n", "PROD")',
'REGEXP_REPLACE(a, "\n", "PROD")'
),
conf=conf
)

@pytest.mark.parametrize("regexp_enabled", ['true', 'false'])
def test_regexp_replace_multi_optimization(regexp_enabled):
gen = mk_str_gen('[abcdef]{0,2}')
gen = mk_str_gen('[abcdef\t\n\a]{0,3}')

conf = { 'spark.rapids.sql.regexp.enabled': regexp_enabled }

Expand All @@ -1032,7 +1034,9 @@ def test_regexp_replace_multi_optimization(regexp_enabled):
'REGEXP_REPLACE(a, "aa|bb|cc|dd", "PROD")',
'REGEXP_REPLACE(a, "(aa|bb)|(cc|dd)", "PROD")',
'REGEXP_REPLACE(a, "aa|bb|cc|dd|ee", "PROD")',
'REGEXP_REPLACE(a, "aa|bb|cc|dd|ee|ff", "PROD")'
'REGEXP_REPLACE(a, "aa|bb|cc|dd|ee|ff", "PROD")',
'REGEXP_REPLACE(a, "a\n|b\a|c\t", "PROD")',
'REGEXP_REPLACE(a, "a\ta|b\nb", "PROD")'
),
conf=conf
)
Expand Down
6 changes: 3 additions & 3 deletions sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,8 +593,9 @@ object GpuOverrides extends Logging {
}

def isSupportedStringReplacePattern(strLit: String): Boolean = {
// check for regex special characters, except for \u0000 which we can support
!regexList.filterNot(_ == "\u0000").exists(pattern => strLit.contains(pattern))
// check for regex special characters, except for \u0000, \n, \r, and \t which we can support
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala

Line 609 in b48335a

// check for regex special characters, except for \u0000 which we can support

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted this comment as it is duplicated and intended to describe the internal logic of this function.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity is there a reason we cannot support all \uXXXX escaped characters? It is just replaced with a unicode character and we should be able to do that directly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is to filter out regex special characters that can be replaced, normal utf8 characters are not in this list and are already supported.

val supported = Seq("\u0000", "\n", "\r", "\t")
!regexList.filterNot(supported.contains(_)).exists(pattern => strLit.contains(pattern))
}

def isSupportedStringReplacePattern(exp: Expression): Boolean = {
Expand All @@ -605,7 +606,6 @@ object GpuOverrides extends Logging {
if (strLit.isEmpty) {
false
} else {
// check for regex special characters, except for \u0000 which we can support
isSupportedStringReplacePattern(strLit)
}
case _ => false
Expand Down
5 changes: 3 additions & 2 deletions tests/src/test/scala/com/nvidia/spark/rapids/StringFunctionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2019-2023, NVIDIA CORPORATION.
* Copyright (c) 2019-2024, NVIDIA CORPORATION.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -207,7 +207,8 @@ class RegExpUtilsSuite extends AnyFunSuite {
"aa|bb|cc|dd" -> Seq("aa", "bb", "cc", "dd"),
"(aa|bb)|(cc|dd)" -> Seq("aa", "bb", "cc", "dd"),
"aa|bb|cc|dd|ee" -> Seq("aa", "bb", "cc", "dd", "ee"),
"aa|bb|cc|dd|ee|ff" -> Seq("aa", "bb", "cc", "dd", "ee", "ff")
"aa|bb|cc|dd|ee|ff" -> Seq("aa", "bb", "cc", "dd", "ee", "ff"),
"a\n|b\t|c\r" -> Seq("a\n", "b\t", "c\r")
)

regexChoices.foreach { case (pattern, choices) =>
Expand Down
Loading