modin 'Series' object has no attribute 'asi8'

[litlep-nibbyt]

System information

OS X 11.6.4
Modin version '0.15.2'
Python 3.9.12

Describe the problem

Subtracting a DatetimeIndex and a modin Series fails since modin Series doesn't have asi8 attribute.

Source code / logs

>>> import modin.pandas as pd
>>> df = pd.DataFrame({'date':[1,2,3,4]})
UserWarning: Distributing <class 'dict'> object. This may take some time.
>>> df['dob'] = pd.date_range(start='1/1/1930', periods=len(df), freq='D')
>>> df['date'] = pd.date_range(start='1/1/1982', periods=len(df), freq='D')
>>> df.set_index('date', inplace=True)
>>> print(type(df.index))
<class 'pandas.core.indexes.datetimes.DatetimeIndex'>
>>> print(type(df.dob))
<class 'modin.pandas.series.Series'>
>>> df.index - df.dob
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/ops/common.py", line 70, in new_method
    return method(self, other)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/arraylike.py", line 108, in __sub__
    return self._arith_method(other, operator.sub)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 6717, in _arith_method
    return super()._arith_method(other, op)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/base.py", line 1295, in _arith_method
    result = ops.arithmetic_op(lvalues, rvalues, op)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/ops/array_ops.py", line 216, in arithmetic_op
    res_values = op(left, right)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/ops/common.py", line 70, in new_method
    return method(self, other)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py", line 1340, in __sub__
    result = self._sub_datetime_arraylike(other)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py", line 738, in _sub_datetime_arraylike
    other_i8 = other.asi8
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/modin/logging/logger_metaclass.py", line 68, in log_wrap
    return method(*args, **kwargs)
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/modin/pandas/series.py", line 335, in __getattr__
    raise e
  File "/usr/local/Caskroom/miniforge/base/envs/enobase3/lib/python3.9/site-packages/modin/pandas/series.py", line 331, in __getattr__
    return object.__getattribute__(self, key)
AttributeError: 'Series' object has no attribute 'asi8'

[pyrito]

Thanks for opening the issue @meijiu ! I was able to reproduce the bug you've just opened. It looks like df.index is being treated as a Series instead of an Index (since this is the class that actually contains the asi8 property). We'll take a look at the issue and merge in a fix.

[jbrockmendel]

FWIW Index.asi8 is deprecated in pandas

[pyrito]

FWIW Index.asi8 is deprecated in pandas

@jbrockmendel Correct me if I'm wrong here but it looks like it's still used in DateTimeIndex and a few others in that vein? That's what the change logs say at least.

[jbrockmendel]

DatetimeIndex, TimedeltaIndex, and PeriodIndex will still have .asi8, but pd.Index and the other subclasses will not.

I'm just starting to dig into the modin code base, will try to get a handle on what's going on with the arithmetic

[pyrito]

@jbrockmendel is this an issue you're interested in taking?

[jbrockmendel]

Looks like the fundamental problem here is that pd.Index.__sub__(ModinSeries) is not returning NotImplemented. The reversed op df["dob"].__rsub__(df.index) works as expected.

A hacky-but-feasible solution would be to implement Series.asi8 to return ser.to_numpy().view("i8").

The correct solution is going to have to live upstream, same underlying issue as pandas-dev/pandas#38946

[pyrito]

Does anyone from @modin-project/modin-contributors @modin-project/modin-core have any thoughts?

[jbrockmendel]

Another option to solve it entirely on modin's end would be for modin's Series to subclass pd.Series. im assuming this was considered and rejected though

[vnlitvinov]

I think we intentionally do not subclass pandas objects, as it would lead to passing isinstance() checks but failing the interop (without some serious efforts).

As for this case - we can try to subclass pd.Index instead...

[jbrockmendel]

we can try to subclass pd.Index instead

This is ostensibly supported but really not advisable pandas-dev/pandas#45289

[pyrito]

@vnlitvinov @jbrockmendel do either of you want to take on this issue?

[jbrockmendel]

but failing the interop (without some serious efforts).

can you expand on this?

[jbrockmendel]

Another option would be to set define Series._typ = "series". That would trick some of the isinstance checks within pandas into treating this like a pandas Series.

[pyrito]

@jbrockmendel @vnlitvinov do either of you have the bandwidth to make a PR for this?

[jbrockmendel]

Will be addressable following pandas-dev/pandas#48347

[jbrockmendel]

This should now be addressable by implementing __pandas_priority__ (in pandas 2.1+)

[vnlitvinov]

Adding blocked as we're waiting for pandas 2.1 release here.