BUG: scalar assignment of a tz-aware is object dtype

[jreback]

[3] should be a datetime64[ns, UTC]

In [1]: df = pd.DataFrame({'A': [0, 1]})

In [3]: df['now'] = pd.Timestamp('20130101', tz='UTC')

In [4]: df
Out[4]: 
   A                        now
0  0  2013-01-01 00:00:00+00:00
1  1  2013-01-01 00:00:00+00:00

In [5]: df.dtypes
Out[5]: 
A       int64
now    object
dtype: object

In [6]: df['now2'] = pd.DatetimeIndex([pd.Timestamp('20130101', tz='UTC')]).repeat(len(df))

In [7]: df.dtypes
Out[7]: 
A                     int64
now                  object
now2    datetime64[ns, UTC]
dtype: object

[DylanDmitri]

I will try and fix this.

[jreback]

great!

[DylanDmitri]

Currently, infer_dtype_from_scalar (on datetimey/timestampy objects) returns a np.datetime64 if no timezone is given, and defaults to np.object_ on objects with timezones. Fixing this problem means returning something else, rather than np.object_.

Ideally return DatetimeTZDtypeType. However, this crashes on np.empty(shape, dtype=dtype) in cast_scalar_to_array. Seems like this should work, but it doesn't.

Quick fix is returning np.datetime64 rather than np.object_. You lose the timezone name, but numpy applies the correct offset before saving so the numbers are correct. This change doesn't break any tests, and results in the following behavior:

In [1]: df = pd.DataFrame({'A': [0, 1]})

In [3]: df['now'] = pd.Timestamp('20130101', tz='UTC')

In [5]: df.dtypes
Out[5]: 
A               int64
now    datetime64[ns]
dtype: object

In [6]: df['now2'] = pd.DatetimeIndex([pd.Timestamp('20130101', tz='UTC')]).repeat(len(df))

In [7]: df.dtypes
Out[7]: 
A                     int64
now          datetime64[ns]
now2    datetime64[ns, UTC]
dtype: object

Raises some inconsistencies, potentially problems with mixing in timezone-naive datetimes. Is the quick fix good enough?

[jreback]

@DylanDmitri you don't want to ever have numpy deal with timezones, they are completely wrong. infer_dtype_from_scalar has a pandas_dtype parameter that will make this work. We should actully just change this to do this by default (though this might break other things)

[DylanDmitri]

Been busy the last week, sorry. Here's the problem code (from line 2874 of frame.py)

# BEFORE
value = cast_scalar_to_array(len(self.index), value)
value = maybe_cast_to_datetime(value, value.dtype)

Main issue: cast_scalar_to_array defaults to dtype np.object_, which is then ignored by maybe_cast_to_datetime. Want to capture the real pandas dtype, and then pass that into maybe_cast_to_datetime, which then works properly.

# AFTER
from pandas.core.dtypes.cast import infer_dtype_from_scalar
pandas_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)

value = cast_scalar_to_array(len(self.index), value)
value = maybe_cast_to_datetime(value, pandas_dtype)

This fixes the problem. Will check tests, and have a PR soon.