ENH: Added str.normalize to use unicodedata.normalize

[sinhrks]

Derived from #9111. Can this be considered in v0.16.1? Otherwise will change the milestone.

unicodedata.normalize is quite useful to standardize multi-bytes characters. I think it is nice if StringMethods.normalize can perform this.

import pandas as pd
s = pd.Series([u'ＡＢＣＤＥ', u'１２３４５'])
s
#0    ＡＢＣＤＥ
#1    １２３４５
# dtype: object

s.str.normalize()
#0    ABCDE
#1    12345
# dtype: object

Another point I'd like to discuss here is the condition Index.str can be used. Currently, inferred_type must be string. I think the preferable condition is:

• Index must be normal Index, not MultiIndex.
• Its inferred_type should be either string, unicode or mixed.

This PR adds unicode currently, not mixed.

pd.Index([u'a', u'B']).inferred_type
# unicode
pd.Index(['a', u'B']).inferred_type
# mixed

# when we allow "mixed" to show str, we should exclude MultiIndex case.
pd.MultiIndex.from_tuples([('a', 'a'), ('a', 'b')]).inferred_type
# mixed

CC: @mortada

[mortada]

minor typo, should be "which behaves the same as"

[jorisvandenbossche]

I never used this, but I can see this could be a useful string method (and I think it is OK for 0.16.1)

And good catch for the unicode index! (for series this did work as it allows all object types)
Maybe add an explicit test for this? (it is now tested in your normalize tests, but for clarity?)

[jorisvandenbossche]

I don't know to what extent we want to really have such unicode characters in our source files?

[sinhrks]

This is an alternative expression of normal unicode string, such as "u'ＡＢＣ'" to work both on 2.x and 3.x. Unable to use six.u here, because it escapes unicode literal and change the result.

[shoyer]

Once we remove Python 3.2 support we can finally use u....

[jorisvandenbossche]

@shoyer 0.17? see #9118, maybe we should just decide when the 'finally' will be

@sinhrks sorry to be unclear, I just meant the unicode in the comment (what the unistr forms). I know our source files are unicode (or at least this one is), but I was just wondering to what extent we should also really use such characters (eg for when people with older or misconfigured editors looking at this file). But probably not a big deal

[sinhrks]

Maybe add an explicit test for this?

Yes, added tests for Index to confirm it should work as the same as Series.

[jreback]

I don't think you can accept mixed-integer, as these are in general python integer objects which need stringification first (which we could do, but that's a separate issue).

In [10]: pd.lib.infer_dtype([1,2,'a'])
Out[10]: 'mixed-integer'

[sinhrks]

@jreback It is to be compat with current Series behavior. In above case, Series.str is applied to all the elements and leave non-str as NaN.

s = pd.Series([1, 2, 'a'])
s.str.len()
# 0   NaN
# 1   NaN
# 2     1
# dtype: float64

[jreback]

i c...ok

[jreback]

@sinhrks thanks!