Dart String toUpperCase and toLowerCase methods are incorrect for Turkish.

[DartBot]

This issue was originally filed by @mdakin

---

String toLowerCase and toUpperCase does not work correctly for Turkish dotless i and capital dotted i.

Run this application (Unfortunately http://try-dart-lang.appspot.com/ loses Turkish characters after I tried to link it):

main() {

  // Expected conversions
  String trUpper = "A,B,C,Ç,D,E,F,G,Ğ,H,I,İ,J,K,L,M,N,O,Ö,P,R,S,Ş,T,U,Ü,V,Y,Z";
  String trLower = "a,b,c,ç,d,e,f,g,ğ,h,ı,i,j,k,l,m,n,o,ö,p,r,s,ş,t,u,ü,v,y,z";
  
  // Actual conversions
  String dartTrUpper = trLower.toUpperCase();
  String dartTrLower = trUpper.toLowerCase();
  
  if (dartTrUpper != trUpper) {
    print ("Incorrect Turkish toUpper conversion. \nExpected: ${trUpper} \nFound: ${dartTrUpper}");
  }
  if (dartTrLower != trLower) {
    print ("Incorrect Turkish toLower conversion. \nExpected: ${trLower} \nFound: ${dartTrLower}");
  }
}

Expected: Program does not print anything.
Actual: Prints 2 messages with outputs.

[DartBot]

This comment was originally written by @ahmetaa

---

I think without proper Locale (is there any?) settings 'I' cannot be converted to 'ı'.

[DartBot]

This comment was originally written by [email protected]

---

String.toLowerCase/toUpperCase will not be locale aware. The plan (which I do not personally agree with) is to provide a separate i18n library that has basically the important functionality of ICU (TBD).

---

Added WontFix label.

[DartBot]

This comment was originally written by @mdakin

---

So in order to make Dart applications that use Strings properly in Turkish (and Azeri etc.), programmers will have to use an external library? This is a huge disappointment.

[DartBot]

This comment was originally written by @ahmetaa

---

If it is well documented, I guess it is o.k. Otherwise it would cause subtle bugs.

[DartBot]

This comment was originally written by @mdakin

---

So in order to make Dart applications that use Strings properly in Turkish (and Azeri etc.), programmers will have to use an external library? This is a huge disappointment.

[DartBot]

This comment was originally written by [email protected]

---

The i18n library will be provided, but you will have to #import it, and use its methods rather than the ones on String.

[DartBot]

This comment was originally written by [email protected]

---

I agree personally - I think it would be better to leave off toLowerCase/toUpperCase from String than to provide one that produces unexpected behavior for non-ASCII characters and incorrect behavior in some locales even for ASCII characters, and we should be learning from i18n mistakes in Java and elsewhere rather than repeating them.

[DartBot]

This comment was originally written by [email protected]

---

If plan for Dart is to use an external library to support basic string case operations for Turkic languages with latin script, will it look like this: (lets assume an application that gets username and lowercases it before processing it further, a common use case)

name = get Name
locale = Get Users locale
if locale uses non Ascii Latin script
  normalized = i18n.toLowerCase(name, locale)
else
  normalized = name.toLowerCase()

I don't expect anybody to write code like this to convert cases, and as a result most applications written will be automatically broken for several Turkic languages.

I actually reported same error for go, http://code.google.com/p/go/issues/detail?id=703 they ended up adding toLowerSpecial and toUpperSpecial methods to Strings package (http://code.google.com/p/go/source/detail?r=477b3015c0 ) However I am not sure this solution is any better.

Would you consider reopening the bug, maybe in a different form? What can we do to fix this problem?

[DartBot]

This comment was originally written by [email protected]

---

Personally, I agree with you, but I think it is going to take more than a few people to change this plan, as the VM guys do not want to have to bundle something like ICU into the VM, as that would preclude implementation on very small platforms. Personally, I think any program that doesn't consider localization is broken at this day and age, and it shouldn't even be possible to write one that doesn't.

Regarding your example, the expectation is that anyone who does care about writing internationalized apps will always use the i18n library and not use String.toLowercase/toUpperCase, so it would look something like this:

normalized = I18n.toLowerCase(name);

There is work to be done on how locales are specified (on the browser it will almost certainly be just one locale at a time by default, the server gets a little messier but by default one locale per request), but the idea is if you care about localization you always use the localizable API.

[iposva-google]

Added Area-Library label.

[DartBot]

This comment was originally written by @mdakin

---

jat@, then, as you said before, maybe the safest path is to remove toLowerCase and toUpperCase methods from String class.

Would Dart team consider removing "WontFix" status for this bug?

[DartBot]

This comment was originally written by [email protected]

---

Set owner to @floitschG.
Added Area-Language label.

[DartBot]

This comment was originally written by @mdakin

---

After thinking about it, I have one last comment on this,

If toLowerCase and toUpperCase are by default locale sensitive, this would introduce a new class of terrible bugs, as Turkish case conversion i <-> İ and ı <-> I converts normal Ascii strings into non-Ascii and this breaks other things like URLs, database column names etc. Java's default locale sensitive case conversion approach caused tons of issues , most applications stopped working on Turkish locale.

The best scenario, as you already mentioned, would be providing both locale sensitive and insensitive case converson methods in a separate core library.

[floitschG]

The core Dart language avoids Unicode as much as possible. It has toLower/UpperCase but not much more.
toLowerCase and toUpperCase are using Unicode's (locale independent) lower and uppercase mappings (see first and fifth question at http://unicode.org/faq/casemap_charprop.html)
String.compareTo uses the unicode charCode and is hence locale independent, too.

As John said: for i18n support one will need to import the i18n library and not use the toLower/UpperCase functions for locale dependent strings.
Hopefully this will be as automatic as importing the html-library.

[DartBot]

This comment was originally written by @ahmetaa

---

Just to add some info,
There is a pub package for this now: http://pub.dartlang.org/packages/dotless_i
Also another issue is opened: https://code.google.com/p/dart/issues/detail?id=12721