API: ser[dt64].astype("string") vs ser[dt64]._values.astype("string")

[jbrockmendel]

ser = pd.Series([pd.NaT, pd.Timestamp.now()])

>>> ser.astype("string")
0                           NaT
1    2020-09-05 16:44:52.259773
dtype: string

>>> pd.Series(ser._values.astype("string"))
0                          <NA>
1    2020-09-05 16:44:52.259773
dtype: string

It would be nice to have the Series code (i.e. the Block.astype) dispatch to the array code.

Note also:

vals = ser._values.astype("string")
>>> vals.astype(ser.dtype)
ValueError: Could not convert object to NumPy datetime

>>> pd.Series(vals, dtype=ser.dtype)
ValueError: Could not convert object to NumPy datetime

>>> pd.to_datetime(vals)  # <-- works

[jorisvandenbossche]

It would be nice to have the Series code (i.e. the Block.astype) dispatch to the array code.

This already happens in general for extension dtypes:

pandas/pandas/core/internals/blocks.py

Lines 583 to 585 in aca77f7

	if self.is_extension:
	try:
	values = self.values.astype(dtype)

But DatetimeBlock (and probably TimedeltaBlock as well?) is the special case of not being an extension block, so taking a different path.

Now, I suppose that also the different path in theory should handle it correctly:

In [8]: pd.core.dtypes.cast.astype_nansafe(np.array(['NaT'], dtype="datetime64[ns]"), pd.StringDtype())                                                                                                            
Out[8]: 
<StringArray>
[<NA>]
Length: 1, dtype: string

so not really sure what is going wrong there.

[jbrockmendel]

@jorisvandenbossche could use your help medium-priority as this is becoming a blocker on ArrayManager work.

dta = pd.date_range("2016-01-01", periods=3)._data
dta[0] = pd.NaT
dta2 = dta.tz_localize("US/Pacific")
ser = pd.Series(dta)
ser2 = pd.Series(dta2)

assert (dta.astype(str) == ser.astype(str).values).all()
assert (dta2.astype(str) == ser2.astype(str)._values).all()

In [9]: ser2.astype("string")
Out[9]: 
0                         <NA>
1    2016-01-02 00:00:00-08:00
2    2016-01-03 00:00:00-08:00
dtype: string

In [10]: pd.Series(dta2.astype("string"))
Out[10]: 
0                         <NA>
1    2016-01-02 00:00:00-08:00
2    2016-01-03 00:00:00-08:00
dtype: string

In [11]: assert ser2.astype("string").equals(pd.Series(dta2.astype("string")))


In [12]: ser.astype("string")
Out[12]: 
0           NaT
1    2016-01-02
2    2016-01-03
dtype: string

In [13]: pd.Series(dta.astype("string"))
Out[13]: 
0                             <NA>
1    2016-01-02T00:00:00.000000000
2    2016-01-03T00:00:00.000000000
dtype: string

In [12] I think we want the first entry to be pd.NA, not "NaT". On the flip side, we probably want the latter two entries of [13] to match the latter two entries of [12].

i.e. dta.astype("string")[~dta.isna()] == dta.astype(str)[~dta.isna()] seems desirable.

This opens a new can of worms w/r/t consistency:

dta.astype(str)
dta.__array__(dtype=str)
dta.to_numpy(dtype=str)

StringArray.astype("datetime64[ns]")
StringArray._from_sequence(ndarray[dt64])
StringArray._from_sequence(DatetimeArray)

[jorisvandenbossche]

In [12] I think we want the first entry to be pd.NA, not "NaT". On the flip side, we probably want the latter two entries of [13] to match the latter two entries of [12].

Yes, agreed.

First question: why is it different?
So it seems that Series.astype for tz-naive data (case In[12]), goes through Block.astype, which ends up calling Block.to_native_types for this case which on its turn uses DatetimeArray._format_native_types:

pandas/pandas/core/internals/blocks.py

Lines 2211 to 2216 in 8dbb593

	def to_native_types(self, na_rep="NaT", **kwargs):
	""" convert to our native types format """
	arr = self.array_values()
	result = arr._format_native_types(na_rep=na_rep, **kwargs)
	return self.make_block(result)

The DatetimeArray.astype for tz-naive data (case In[13]), on the other hand, use StringArray._from_sequence for string dtype (but still _format_native_types to do astype(str), the non-extension string type):

pandas/pandas/core/arrays/datetimelike.py

Lines 346 to 351 in 8dbb593

	elif is_string_dtype(dtype) and not is_categorical_dtype(dtype):
	if is_extension_array_dtype(dtype):
	arr_cls = dtype.construct_array_type()
	return arr_cls._from_sequence(self, dtype=dtype)
	else:
	return self._format_native_types()

I suppose we can fix DatetimeArray.astype to also use _format_native_types for the extension string dtype, specifying na_rep=pd.NA to avoid converting pd.NaT to a string.

---

In the end, this also relates a lot to #22384 for more thoroughly clean this up (but I think we can certainly already fix the inconsistency above). I will try to carve out some time to look back at this issue.

[jorisvandenbossche]

@jbrockmendel this seems to be working correctly now (using your initial example):

In [25]: ser = pd.Series([pd.NaT, pd.Timestamp.now()])
    ...: 
    ...: ser.astype("string")
Out[25]: 
0                          <NA>
1    2022-12-09 13:00:34.091345
dtype: string

(not sure if it has done intentionally and is tested, or if we should still add tests for it)

[jbrockmendel]

best guess is it still merits a test