Logs streaming API returns non escaped UTF-8 characters

[Settler]

Nomad version

Nomad v1.0.0 (cfca640)

Operating system and Environment details

Windows Server 2019

Issue

If the service writes non-ASCII logs into stdout, then frames returned from /v1/client/fs/logs will be decoded from Base64 in UI with invalid chars.
Looks like this: � � � � �

I have found this issue and pull request. It looks that fixes took place only in frontend part. They don't fix encoding part on server side which generates Base64 string. It seems that server encodes non-UTF8 chars into Base64 without escaping them and frontend, as a result, decodes them in wrong way.

[tgross]

Hi @Settler! Thanks for reporting this. I see that you're on Windows... do you know if the source data is UTF-16 encoded? That might help narrow down the behavior.

[Settler]

Hi @tgross.

Hmm. I saw Cyrillic OEM 866 encoding in log file xxx.stdout.0 where Russian letters were. But for pure English logs I saw UTF-8 encoding. But no one touched this log file except nomad.
Now I don't see logs with Cyrillic symbols, because our team has already "patched" our logs and removed Cyrillic symbols. Will try to make them again.

[tgross]

Ok, so just for clarity, the issue title says there are problems with utf-8 encoded characters but the problem is with non-utf-8 encoding? And is the terminal where you're reading the logs able to decode Cyrillic OEM 866?

[Settler]

Now when you asked, I'm not sure =)
Because I thought that nomad always write logs in UTF-8 and didn't check exact encoding in file :)
I'll try to reproduce that and write to you soon.

[Settler]

Yeah, I tested. It's strange. File encoding is Cyrillic OEM 866.
Let me dive you deeply inside our configuration. Maybe you'll get an idea of what happening.

We use .NET Core 3.1 and nlog to write logs.
For nlog we use this setup:

  <targets async="false">
    <target xsi:type="ColoredConsole" name="console" layout="${longdate} ${logger:shortName=true} ${message}" />
  </targets>

So, nlog writes to the Console (stdout).
To run processes we use Damon and raw_exec. So, damon catches stdout of our process and then sends logs to nomad (if I understand pipeline correctly).
Then nomad writes them to file with name {TASK_ID}.stdout.0.
And IF nomad receives Cyrillic characters, then he (maybe?) changes file encoding to Cyrillic OEM 866. If he receives only ANSI characters, then file encoding is kept as UTF-8.

That is all my guessings...

[Settler]

@tgross I have good news.

I have talked with our IT guys. And we found new localization settings in Windows Server 2019.
It sounds like this: Use Unicode UTF-8 for worldwide language support.
This setting could be set in Windows Region Settings:

If we enable this function then logs with Cyrillic symbols will be written with UTF-8 encoding.
So, I think this is an environment Issue and Windows "feature". We could close this Issue, because the solution has been found.
Hope this will help someone else.

[tgross]

Great! Glad to hear @Settler!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.