[ArrayManager] Ensure to store datetimelike data as DatetimeArray/TimedeltaArray (and not ndarray)

[jorisvandenbossche]

Pre-cursor for #39991

Currently we didn't really check that we were consistently storing datetimelike data as the EA (DatetimeArray, TimedeltaArray) or as ndarrray. Ensuring this in the ArrayManager constructor turns up a few failures.

I think it will be the easiest to always store them as EA and not as ndarray (eg for many other operations, we otherwise would wrap them in the EA anyway).

[jorisvandenbossche]

The reason I am doing this here is because this fails:

In [4]: np.array([pd.NaT], dtype="M8[ns]")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-99d42e913a1c> in <module>
----> 1 np.array([pd.NaT], dtype="M8[ns]")

ValueError: cannot convert float NaN to integer

(which is what the np.array([arr[loc] for arr in self.arrays], dtype=temp_dtype) above can be doing if the resulting dtype is M/m8)

@jbrockmendel do you know if that's something we do elsewhere as well / there is some existing code for this?

[jbrockmendel]

IIRC we can get BM to mess up if we get here with non-consolidated all-td64 blocks.

I think the thing to do (also similar change in the BM method) is to define result on L737 as a list, only wrap with np.array if none of these conditions hold

[jorisvandenbossche]

IIRC we can get BM to mess up if we get here with non-consolidated all-td64 blocks.

You can't get here in that case, though, since this is an ArrayManager method. And with the current BM you don't have this problem since it doesn't store DatetimeArray/TimedeltaArray with numpy dtypes.

I think the thing to do (also similar change in the BM method) is to define result on L737 as a list, only wrap with np.array if none of these conditions hold

Thanks, that's a good idea

[jorisvandenbossche]

Updated with this simplification.

[jorisvandenbossche]

@jbrockmendel more comments here?

[jbrockmendel]

we have a little-used is_ea_or_datetimelike_dtype, could use an analogous helper to get DatetimeArray/TimedeltaArray in these cases (not for this PR)

[jbrockmendel]

did you check if this is relevant for the BlockManager case?

[jorisvandenbossche]

Yes, see my (somewhat) answer at #40147 (comment).
But moreover, in the BlockManager method, it assigns slices from the Block values into the resulting array:

pandas/pandas/core/internals/managers.py

Lines 978 to 982 in b835ca2

	for blk in self.blocks:
	# Such assignment may incorrectly coerce NaT to None
	# result[blk.mgr_locs] = blk._slice((slice(None), loc))
	for i, rl in enumerate(blk.mgr_locs):
	result[rl] = blk.iget((i, loc))

So that's quite different as the code here, and the idea of first keeping it in a list doesn't really apply.

[jbrockmendel]

thanks

[jbrockmendel]

LGTM cc @jreback

[jorisvandenbossche]

@jbrockmendel thanks for the review

I updated #39991 now based on this