utf8proc does not correctly handle the 66 Unicode "noncharacters"

[ScottPJones]

utf8proc considers the 66 Unicode "noncharacters" to not be valid, however the Unicode standard specifically says that they are valid code points, and need to be handled correctly in conforming software.

[stevengj]

According to the Unicode FAQ they should still be category Cn, but we shouldn't return UTF8PROC_ERROR_INVALIDUTF8 for these.

[ScottPJones]

Yes, precisely!

[stevengj]

Quickly looking through the code, it seems like the only place this comes up is in utf8proc_iterate, where it seems like if (uc < 0 || ((uc & 0xFFFF) >= 0xFFFE)) should be replaced with if (uc < 0). And the (uc >= 0xFDD0 && uc < 0xFDF0) check should be removed in the lines above that.

Oh, and also in utf8proc_codepoint_valid.

[ScottPJones]

I think that's all I'd seen also... will fix as soon as I find that bloody "round tuit"!

[ScottPJones]

OK, I made the changes, testing them by rebuilding julia and running all the unit tests... but then it turns out that deps/utf8proc/utf8proc.c is not the same as in the JuliaLang/utf8proc repository... lots of simple difference, like UTF8PROC_DLLEXPORT vs. DLLEXPORT.
Can anybody please explain why there are differences, and how I put my changes in so they get used by Julia? Thanks!

[pao]

Julia doesn't track master of external dependencies, like utf8proc, except when absolutely necessary. I think you can modify your local build to use master (you can probably figure it out from deps/Makefile) for testing purposes.

[stevengj]

I would clone utf8proc into a separate repository before making changes. Editing git submodules is a recipe for trouble if you aren't a git guru.

[ScottPJones]

@stevengj Thanks, luckily, I'd already done that last night, after going crazy trying to figure out why deps/utf8proc didn't match my ScottPJones/utf8proc fork...