BUG: NA does not propagate through mask and where/ raises AssertionError in 1.1.0rc

[johnflavin]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

>>> a = pd.DataFrame([{'col': 'a'}] * 3).convert_dtypes()
>>> a
  col
0   a
1   a
2   a
>>> tfna = pd.Series([True, False, pd.NA]).convert_dtypes()
>>> tfna
0     True
1    False
2     <NA>
dtype: boolean
>>> a.mask(tfna, 'b')
  col
0   b
1   a
2   b
>>> a.where(tfna, 'b')
  col
0   a
1   b
2   b

Problem description

When replacing values with mask or where, pd.NA values in a boolean series used for the cond input are always replaced. This means pd.NA gets treated as False by where but as True by mask.

Expected Output

I would expect that pd.NA values in cond would propagate to the output.

>>> a.mask(tfna, 'b')
    col
0     b
1     a
2  <NA>
>>> a.where(tfna, 'b')
    col
0     a
1     b
2  <NA>

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.3.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 49.2.0.post20200712
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.15.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.17.1
pytables : None
pytest : 5.4.3
pyxlsb : None
s3fs : 0.4.2
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[simonjayhawkins]

Thanks @johnflavin for the PR.

This now raises AssertionError on master (code change in #32962) cc @jbrockmendel

>>> pd.__version__
'1.1.0rc0+7.g04e9e0afd'
>>>
>>> a = pd.DataFrame([{"col": "a"}] * 3).convert_dtypes()
>>> a
  col
0   a
1   a
2   a
>>>
>>>
>>> a.dtypes
col    string
dtype: object
>>>
>>> tfna = pd.Series([True, False, pd.NA]).convert_dtypes()
>>> tfna
0     True
1    False
2     <NA>
dtype: boolean
>>>
>>>
>>> a.mask(tfna, "b")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas\pandas\core\generic.py", line 9022, in mask
    return self.where(
  File "C:\Users\simon\pandas\pandas\core\generic.py", line 8992, in where
    return self._where(
  File "C:\Users\simon\pandas\pandas\core\generic.py", line 8849, in _where
    new_data = self._mgr.where(
  File "C:\Users\simon\pandas\pandas\core\internals\managers.py", line 513, in where
    return self.apply(
  File "C:\Users\simon\pandas\pandas\core\internals\managers.py", line 396, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\Users\simon\pandas\pandas\core\internals\blocks.py", line 1826, in where
    cond = _extract_bool_array(cond)
  File "C:\Users\simon\pandas\pandas\core\internals\blocks.py", line 2864, in _extract_bool_array
    assert mask.dtype == bool, mask.dtype
AssertionError: object
>>>
>>> a.where(tfna, "b")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\pandas\pandas\core\generic.py", line 8992, in where
    return self._where(
  File "C:\Users\simon\pandas\pandas\core\generic.py", line 8849, in _where
    new_data = self._mgr.where(
  File "C:\Users\simon\pandas\pandas\core\internals\managers.py", line 513, in where
    return self.apply(
  File "C:\Users\simon\pandas\pandas\core\internals\managers.py", line 396, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\Users\simon\pandas\pandas\core\internals\blocks.py", line 1826, in where
    cond = _extract_bool_array(cond)
  File "C:\Users\simon\pandas\pandas\core\internals\blocks.py", line 2864, in _extract_bool_array
    assert mask.dtype == bool, mask.dtype
AssertionError: object
>>>

[jbrockmendel]

I would expect that pd.NA values in cond would propagate to the output.

I guess until that feature is implemented, this could raise a ValueError or NotImplementedError

[dsaxton]

I would expect that pd.NA values in cond would propagate to the output.

>>> a.mask(tfna, 'b')
    col
0     b
1     a
2  <NA>
>>> a.where(tfna, 'b')
    col
0     a
1     b
2  <NA>

I think this makes sense in theory but would be difficult / impossible in general since most types don't support NA. Treating NA as False seems like the simplest solution and would be consistent with how indexing works with nullable booleans.

[simonjayhawkins]

moved off 1.1.2 milestone (scheduled for this week) as no PRs to fix in the pipeline

[jreback]

no open PR moving off 1.2