[BUG] ScanJson and JsonToStructs does not deal with escaped single quotes properly

[revans2]

Describe the bug
This is a little odd, but ' in a JSON string is only disallowed if allowBackslashEscapingAnyCharacter and allowSingleQuotes are both false. The default is to have allowSingleQuotes true, so on the GPU we end up returning the wrong thing, which is null all of the time.

scala> Seq("""{"data": "\'TESTING\'"}""").toDF("value").repartition(1).selectExpr("from_json(value, 'data string', map('allowBackslashEscapingAnyCharacter', 'true', 'allowSingleQuotes', 'true'))", "*").show(100, false)
+----------------+-----------------------+
|from_json(value)|value                  |
+----------------+-----------------------+
|{'TESTING'}     |{"data": "\'TESTING\'"}|
+----------------+-----------------------+


scala> Seq("""{"data": "\'TESTING\'"}""").toDF("value").repartition(1).selectExpr("from_json(value, 'data string', map('allowBackslashEscapingAnyCharacter', 'false', 'allowSingleQuotes', 'true'))", "*").show(100, false)
+----------------+-----------------------+
|from_json(value)|value                  |
+----------------+-----------------------+
|{'TESTING'}     |{"data": "\'TESTING\'"}|
+----------------+-----------------------+


scala> Seq("""{"data": "\'TESTING\'"}""").toDF("value").repartition(1).selectExpr("from_json(value, 'data string', map('allowBackslashEscapingAnyCharacter', 'true', 'allowSingleQuotes', 'false'))", "*").show(100, false)
+----------------+-----------------------+
|from_json(value)|value                  |
+----------------+-----------------------+
|{'TESTING'}     |{"data": "\'TESTING\'"}|
+----------------+-----------------------+


scala> Seq("""{"data": "\'TESTING\'"}""").toDF("value").repartition(1).selectExpr("from_json(value, 'data string', map('allowBackslashEscapingAnyCharacter', 'false', 'allowSingleQuotes', 'false'))", "*").show(100, false)
+----------------+-----------------------+
|from_json(value)|value                  |
+----------------+-----------------------+
|{null}          |{"data": "\'TESTING\'"}|
+----------------+-----------------------+

[revans2]

This depends on rapidsai/cudf#15303

[revans2]

This depends on rapidsai/cudf#15303

[revans2]

The CUDF code/dependency was fixed and this appears to be working fine now.