StringComparer.CurrentCultureIgnoreCase does not ignore case when LANG=C.UTF-8

[rjmholt]

See also: PowerShell/PowerShell#7761

For a small repro:

• Run dotnet new console
• Put the following in Program.cs:

  using System;
  using System.Collections;
  
  namespace LangExample
  {
      class Program
      {
          static void Main(string[] args)
          {
              if (StringComparer.CurrentCultureIgnoreCase.Compare("String", "string") != 0)
              {
                  throw new Exception("Hashtable key comparison is case-sensitive");
              }
  
              Console.WriteLine("Working as expected");
          }
      }
  }

• Run dotnet run and get output:

  Working as expected
  

• Run LANG=C.UTF-8 dotnet run and get output:

  
  Unhandled Exception: System.Exception: Hashtable key comparison is case-sensitive
     at LangExample.Program.Main(String[] args) in /home/rob/Documents/Dev/sandbox/LangExample/Program.cs:line 18
  

Expected Behaviour (with LANG=C.UTF-8):

StringComparer.CurrentCultureIgnoreCase.Compare("String", "string") == 0

Actual Behaviour (with LANG=C.UTF-8):

String.Comparer.CurrentCultureIgnoreCase.Compare("String", "string") != 0

Runtime information:

> dotnet --version
2.1.401

> uname -a
Linux chronos 4.15.0-33-generic dotnet/runtime#13856-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

> cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

Given that C.UTF-8 is a Debian locale, this is probably also the case on other Debian-based Linux distros.

[tarekgh]

"C" and "Posix" collations behavior is case sensitive. we encourage not using these locales if you are performing any sorting operations. you may look at https://github.com/dotnet/corefx/issues/28611 for more info.

You may look at the comment on the code https://github.com/dotnet/coreclr/blob/master/src/corefx/System.Globalization.Native/locale.cpp#L128 for Posix but it apply to C too.

Also, you may look at https://www.postgresql.org/docs/10/static/collation.html which is telling same things

23.2.2.1. Standard Collations
On all platforms, the collations named default, C, and POSIX are available. Additional collations may be available depending on operating system support. The default collation selects the LC_COLLATE and LC_CTYPE values specified at database creation time. The C and POSIX collations both specify “traditional C” behavior, in which only the ASCII letters “A” through “Z” are treated as letters, and sorting is done strictly by character code byte values.

[tarekgh]

dotnet/docs#8179

[wfurt]

This still seems wrong @tarekgh I could see that the lang could impact default comparison but I would expect functions where explicitly asked for "IgnoreCase" do case-insensitive comparison.

[tarekgh]

@wfurt did you read the comment https://github.com/dotnet/corefx/issues/32250#issuecomment-420749205? it has the explanation there. IF you don't like the behavior switch off from POSIX and C.

[wfurt]

Yes, I did read through and I think the argument is not correct.
I did also testing with native libc implementation and strcasecmp() keeps comparing in case insensitive matter regardless of LANG.
I did not find any documentation suggesting that CurrentCultureIgnoreCase may or may not perform case insensitive matching.

[wfurt]

https://wiki.musl-libc.org/functional-differences-from-glibc.html
musl-linux like Alpine is different from other distributions. The "normal" ways how to generate or changer locale does not quite work.
https://bugs.alpinelinux.org/issues/7374

[danmoseley]

@tarekgh what are your thoughts about @wfurt comment above?

i'm guessing most .NET code out there relies somewhere on case insensitive comparison, it seems quite odd if WSL is truly configured such that that is impossible. Should they set a different LANG? (Or maybe this is different in the new WSL2 )

[danmoseley]

I see @tarekgh already updated docs which is good 😄 dotnet/docs#8179

Just curious about @wfurt point.

[tarekgh]

To summarize/clarify,

When setting the culture to "C", ICU map it to en_us_posix. and the posix locale is for sure doesn't support case insensitive comparisons. We are just working as ICU. So, .NET Core behavior is 100% matching ICU which is the main globalization component for Linux. In short, we are not defining how "C" locale work, we just do whatever ICU does.

Now, if we need to do something here, we can just start changing the mapping of "C" to something else rather than POSIX locale. This is kind of breaking change as no idea if there is people already taking dependency in such behavior. POSIX locale just doing that because people wanted to compare strings as binaries (regardless of using case sensitive option or not).

Let me know what you think.

CC @jkotas @stephentoub

[danmoseley]

I see. In regular distros/shells, do they typically have different choices for LANG? I wonder why WSL does this, it seems like a large gotcha.

[tarekgh]

I can follow up with WSL team about that. but in Linux world think about "C" as invariant culture.

[wfurt]

Alpine is also like this. There really is no locale other than C. (see #962)

[danmoseley]

I am happy to close this and defer to experts. It would be great to circle back with WSL team's response. Thanks guys.