encoding of xquery modules is corrupted

[peetkes]

Hi,

I just upgraded from version 2.1.0 to the latest version 2.5.0 and I noticed something about the encoding of the xquery modules.

We have a module containing the following line:
let $returnValue := fn:replace($input,"Ä|ä|À|à|Á|á|Â|â|Ã|ã|Å|å|Ǎ|ǎ|Ą|ą|Ă|ă|Æ|æ","A")
When we upgraded to 2.5.0 the line ends up in the modules database like this:
let $returnValue := fn:replace($input,"Ä|ä|À|à |�|á|Â|â|Ã|ã|Å|å|�|ǎ|Ą|ą|Ă|ă|Æ|æ","A")
So I'm guessing something with the encoding of modules changed.
I also tried version 2.2.0 and that behaved the same as 2.5.0 so I think the issue is introduced in version 2.2.0

We are working with MarkLogic 8.0-6 on a Linux environment

Thanks,

Peter

[peetkes]

Most likely this is caused by the ml-javaclient-util where you added support for properties replacement ala roxy.

[rjrudin]

Was thinking the same thing - thanks for the line of code, I'll try to reproduce right now

[rjrudin]

Odd - I can reproduce on another project using ml-gradle 2.5.0, but I can't reproduce it within the ml-javaclient-util or ml-app-deployer projects. In both those projects, I can successfully replace tokens in a module, and that line of code is still stored correctly in ML. Seems like something specific to the Gradle runtime environment then, I'll keep digging.

[peetkes]

extra note: I'm running gradle from a Windows machine targeting a Linux machine

[rjrudin]

The recommendation at http://stackoverflow.com/questions/21267234/show-utf-8-text-properly-in-gradle worked for me.

Here's the task I tried out:

task info {
  doLast {
    println "prop: " + System.getProperty("file.encoding")
    println file("test.txt").text
  }
}

And test.txt contains:

"Hello Ä|ä|À|à|Á|á|Â|â|Ã|ã|Å|å|Ǎ|ǎ|Ą|ą|Ă|ă|Æ|æ"

And when I ran that without setting file.encoding properly, it printed out as:

prop: Cp1252
"Hello Ä|ä|À|à|Á|á|Â|â|Ã|ã|Å|å|?|?|?|?|?|?|Æ|æ"

And then I did "gradle --stop" to kill any Gradle daemons (though with Gradle 3.x, that no longer seems necessary - if you modify GRADLE_OPTS, Gradle seems to recognize that any existing daemons are "incompatible now"), and I then ran this:

export GRADLE_OPTS="-Dfile.encoding=utf-8"

And then I ran the task again, and it printed correctly:

prop: utf-8
"Hello Ã|ä|Ã|à|Ã|á|Ã|â|Ã|ã|Ã|Ã¥|Ç|Ç|Ä|Ä|Ä|Ä|Ã|æ"

Note - I tried some other ways of setting file.encoding with Gradle, even with "-D" on the command line, but only exporting it to GRADLE_OPTS did the trick.

[rjrudin]

Forgot to mention - once I set file.encoding in GRADLE_OPTS, my module with that text in it loaded correctly.

I'll add a Wiki page to touch on this, possibly add it to the FAQ.

[rjrudin]

https://github.com/rjrudin/ml-gradle/wiki/Encoding-issues

[mflatscher]

Another way that worked for me, when systemProp.file.encoding in gradle properties did not: Add org.gradle.jvmargs='-Dfile.encoding=UTF-8' to gradle.properties