Pandas any() returning false with true values present

[maanukuttan]

Code Sample

In [1]: from io import StringIO

In [2]: import pandas as pd

In [3]: data = StringIO("""issue_date,issue_date_dt
   ...: ,
   ...: ,
   ...: 19600215.0,1960-02-15
   ...: ,
   ...: ,""")

In [4]: df = pd.read_csv(data, parse_dates=[1])

In [5]: df
Out[5]:
   issue_date issue_date_dt
0         NaN           NaT
1         NaN           NaT
2  19600215.0    1960-02-15
3         NaN           NaT
4         NaN           NaT

In [6]: df.any(axis=0)
Out[6]:
issue_date       True
issue_date_dt    True
dtype: bool

In [7]: df.any(axis=1)
Out[7]:
0    False
1    False
2    False
3    False
4    False
dtype: bool

Problem description

df.any(axis=0) behaves as expected. It returns True for both the columns, but df.any(axis=1) returns False for all the rows.
Note: A question with similar issue can be found here

Note: If you use notnull then we are getting the required output

In [9]: df.notnull().any(1)
Out[9]:
0    False
1    False
2     True
3    False
4    False
dtype: bool

Expected Output

df.any(axis=1) should return True for those rows with True values.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.15.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

Simpler repro:

In [21]: df = pd.DataFrame({"A": [1.0], "B": [pd.Timestamp('1960-02-15')]})
    ...:
    ...:
    ...:

In [22]: df.any(1)
Out[22]:
0    False
dtype: bool

[TomAugspurger]

Hmm, this seems wrong. We have mixed dtypes in

pandas/pandas/core/frame.py

Lines 7142 to 7143 in 362f2e2

	if axis == 1 and self._is_mixed_type and self._is_datelike_mixed_type:
	numeric_only = True

so we set numeric_only=True.

Then we go down to

pandas/pandas/core/frame.py

Lines 7196 to 7199 in 362f2e2

	if filter_type is None or filter_type == 'numeric':
	data = self._get_numeric_data()
	elif filter_type == 'bool':
	data = self._get_bool_data()

and our filter_type is bool, so we select boolean data only.

(Pdb) l
7202            else:
7203                if numeric_only:
7204                    if filter_type is None or filter_type == 'numeric':
7205                        data = self._get_numeric_data()
7206                    elif filter_type == 'bool':
7207 ->                     data = self._get_bool_data()
7208                    else:  # pragma: no cover
7209                        msg = ("Generating numeric_only data with filter_type {f}"
7210                               "not supported.".format(f=filter_type))
7211                        raise NotImplementedError(msg)
7212                    values = data.values
(Pdb) n
> /Users/taugspurger/sandbox/pandas/pandas/core/frame.py(7212)_reduce()
-> values = data.values
(Pdb) data
Empty DataFrame
Columns: []
Index: [0]

And since that's empty, the .any(1) will be false. Off the top of my head, I'm not sure what the fix is right now.

[TomAugspurger]

Moving off of 0.24, but would certainly welcome a fix if anyone wants to work on this.