Replace invalid unicode with replacement chars

[michalmuskala]

This could be an optional mode for the parser.

[vorce]

Hey @michalmuskala I'm interested in trying to create a PR for this. Do you have any implementation ideas or other pointers?

[michalmuskala]

I think the best way to handle this would be to expand the string_decode function we're passing around in the decoder. Right now it's only passed the final string at the end of everything, but we could be taking the entire string/7 function there and having couple versions of it. We'd probably need 4 versions - regular, copying, one that replaces unicode but does not copy when not necessary and one that does copy. It will be a bit of duplication, but it should be mostly copy-paste and that way should be the most performant. We should avoid additional conditionals in the main decode loop.

As to how to handle the replacement char, there are actually two ways - one is to replace every invalid byte with one replacement char, another to collect all invalid bytes and use just one replacement char in place of them. I've seen references on the internet to both methods. It would likely require some research to see which one is the most compliant with the standard. We should also probably treat invalid escapes as replacement chars and not errors in that case - basically turn every string decoding error into replacement chars.

[Moosieus]

This seems an infrequent but challenging issue people still encounter.

Shortly after the last post on this issue, the Unicode Standard was updated to promote W3C's standard for consistent substitution (Seen here, under the heading "U+FFFD Substitution of Maximal Subparts").

The basic gist is:

• Valid-but-truncated code sequences get replaced with one "�".
• All other illegal bytes are replaced with one "�" each.

I couldn't find anything in Elixir or Erlang that did this, so I wrote my own. Ideally though Elixir or OTP would provide a native solution (as other languages do).

[Moosieus]

Would it be possible to integrate the matches from String.replace_invalid/2 into the existing Jason.Decoder.string functions?