BUG: MaskedArray does not seem to respect ufunc dispatch hierarchy

[jthielen]

When using a custom array container that implements __array_ufunc__ and is meant to be able to wrap MaskedArrays, non-commutativity occurs in binary operations with MaskedArrays. Despite what the below comment mentions:

https://github.com/numpy/numpy/blob/v1.17.3/numpy/ma/core.py#L3960-L3972

it does not properly defer, as seen in the example below. I am unfortunately not well-acquainted with the MaskedArray internals, so I don't know what would be a good way forward for a fix.

xref hgrecco/pint#633

Reproducing code example:

import numpy as np
import numpy.lib.mixins


class WrappedArray(numpy.lib.mixins.NDArrayOperatorsMixin):
    __array_priority__ = 20

    def __init__(self, array, **attrs):
        self._array = array
        self.attrs = attrs

    def __repr__(self):
        return f"{self.__class__.__name__}(\n{self._array}\n{self.attrs}\n)"

    def __array__(self):
        return np.asarray(self._array)

    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
        if method == '__call__':
            inputs = [arg._array if isinstance(arg, self.__class__) else arg
                      for arg in inputs]
            return self.__class__(ufunc(*inputs, **kwargs), **self.attrs)
        else:
            return NotImplemented


# Show basic wrap
w = WrappedArray(np.arange(3), test=1)
print(w)
print()

# Wrapping MaskedArrays works fine
wm = WrappedArray(np.ma.masked_array([1, 3, 5], mask=[False, True, False]),
                  test=2)
print(wm)
print()

# Operations with ndarrays work fine
a = np.array([2, 0, 1])
print(a * w)
print()

# Operations with masked arrays are not commutative
m = np.ma.masked_array([2, 0, 1], mask=[False, True, False])

# Good
print(w * m)
print()

# Bad
print(m * w)

Output:

WrappedArray(
[0 1 2]
{'test': 1}
)

WrappedArray(
[1 -- 5]
{'test': 2}
)

WrappedArray(
[0 0 2]
{'test': 1}
)

WrappedArray(
[0 -- 2]
{'test': 1}
)

[0 -- 2]

Numpy/Python version information:

1.17.3 3.6.7 | packaged by conda-forge | (default, Nov 6 2019, 16:19:42)
[GCC 7.3.0]

[shoyer]

ping @mhvk who may have some thoughts here.

I suspect the right fix is to define a __array_ufunc__ method on MaskedArray, and rewrite the arithmetic special methods from:

   def add(self, other):
        if self._delegate_binop(other):
            return NotImplemented
        return add(self, other)

to

   def add(self, other):
        if self._delegate_binop(other):
            return NotImplemented
        return np.add(self, other)

Currently MaskedArray overrides ufuncs via __array_finalize__, which is quite indirect. There is an __array_ufunc__ method defined, but only via inheritance from the base np.ndarray class.

[mhvk]

Yes, MaskedArray should gain a proper __array_ufunc__... Right now, you also have to use the "old" system of overrides, and write a custom __array_wrap__. I think this may be do-able, but obviously less clear if it is worth the effort.

[dopplershift]

This issue should be re-opened since #16022 was reverted.

[mattip]

Reopening, since the PR to use __array_ufunc__ was reverted.