JSON becomes malformed when trying to write string containing \\u0000

[susisu]

If input JSON has a string containing a sequence \\u0000, due to #348, the part \u0000 is removed and the escape character \ is left, and this possibly causes error.

For example, when trying to write a JSON { "xxx": "***\\u0000***" }, I got the following error.

org.postgresql.util.PSQLException: ERROR: invalid input syntax for type json
  Detail: Escape sequence "\*" is invalid.
  Where: JSON data, line 1: {"xxx":"***\*...

[susisu]

If slick-pg can carefully remove null characters in strings, it would be nice; otherwise IMO it should not touch JSON after it is stringified.

[tminglei]

@susisu fixed, pls help check.

BTW, \\u0000 must be removed from inputing json string, otherwise it will encounter exception as

[info]   Cause: org.postgresql.util.PSQLException: ERROR: unsupported Unicode escape sequence
[info]   details: \u0000 cannot be converted to text.
[info]   position: JSON data, line 1: {"d":...

[susisu]

@tminglei Thank you for quick response.

BTW, \u0000 must be removed from inputing json string, otherwise it will encounter exception

I mean JSON string "\\u0000" should be saved as-is, because it is a normal sequence that does not contain null characters. Your fix seems not solving this...

[susisu]

In addition, string like "\\\u0000" will result in a malformed one.

[tminglei]

@susisu I did a re-fix try, can you help check again?

[susisu]

@tminglei Thanks! It is pretty good for "\\u0000".
However, it looks \u0000 in "\\\u0000" will not be removed now.

How about using regex like """\\+u0000""".r and removing \u0000 only if the number of \ is 2n+1?

[tminglei]

Regex will fail, I didn't find a useable regex.

[susisu]

Regex will fail, I didn't find a useable regex.

Yes, the exact regex that matches \u0000 in "\u0000" and "\\\u0000" but not matches it in "\\u0000" does not exist (or is non-standard).
But it is possible to find candidates that contain \u0000 to be removed, isn't it?

[susisu]

Maybe it becomes a bit complex if using regex etc., and I think it can be a reasonable way to not touch JSON after stringified.
But I don't know which is better...

[tminglei]

now, 123\u000045\\u00006\\\u00007 will be converted to 12345\\u00006\\7.
I didn't find a better way.

[susisu]

now, 123\u000045\\u00006\\\u00007 will be converted to 12345\\u00006\\7.

I expected this result, but d2731f4 will yield 12345\\u00006\\\u00007 which contains null character \u0000.

[tminglei]

@susisu re-fixed again. Your question should be answered by the test case.
Pls help check again.

[susisu]

@tminglei Looks good enough. Thanks!