Selecting "subsets" of a MultiIndex DataFrame sometimes changes dtypes

[Mofef]

Code Sample, a copy-pastable example if possible

Definition of data and columns:

from numpy import nan
data = [['n',  1,  0,  False,  2,  1,  False,  0,  0,  False,  2,  0,  False,  0,  1,  False,  1,  1,  False,  'o',
  1521734085.289453,  'p',  3233,  1521734085.289494]]
columns = [('a', 'd', 'i', nan, nan),
 ('a', 'd', 'j', 0.0, 'k'),
 ('a', 'd', 'j', 0.0, 'l'),
 ('a', 'd', 'j', 0.0, 'm'),
 ('a', 'd', 'j', 1.0, 'k'),
 ('a', 'd', 'j', 1.0, 'l'),
 ('a', 'd', 'j', 1.0, 'm'),
 ('a', 'd', 'j', 2.0, 'k'),
 ('a', 'd', 'j', 2.0, 'l'),
 ('a', 'd', 'j', 2.0, 'm'),
 ('a', 'd', 'j', 3.0, 'k'),
 ('a', 'd', 'j', 3.0, 'l'),
 ('a', 'd', 'j', 3.0, 'm'),
 ('a', 'd', 'j', 4.0, 'k'),
 ('a', 'd', 'j', 4.0, 'l'),
 ('a', 'd', 'j', 4.0, 'm'),
 ('a', 'd', 'j', 5.0, 'k'),
 ('a', 'd', 'j', 5.0, 'l'),
 ('a', 'd', 'j', 5.0, 'm'),
 ('b', 'f', nan, nan, nan),
 ('b', 'h', nan, nan, nan),
 ('c', 'e', nan, nan, nan),
 ('c', 'g', nan, nan, nan),
 ('c', 'h', nan, nan, nan)]

pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns)).dtypes.a.d.i
# object

pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns)).a.d.i.dtypes
# float64

this causes for example:

pd.DataFrame(np.array(data), columns=pd.MultiIndex.from_tuples(columns)).a.d.i 
# "n", dtype: object

pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns)).a.d.i 
#  nan, dtype: float32

Problem description

I think the example is self explaining

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-119-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.utf8
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.23.0.dev0+38.g6552718
pytest: 2.8.7
pip: 9.0.1
setuptools: 20.7.0
Cython: 0.23.4
numpy: 1.14.2
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 5.5.0
sphinx: 1.3.6
patsy: 0.4.1
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.6.4
feather: None
matplotlib: 2.1.2
openpyxl: 2.3.0
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: 0.7.3
lxml: 3.5.0
bs4: 4.4.1
html5lib: 0.9999999
sqlalchemy: 1.0.11
pymysql: None
psycopg2: 2.6.1 (dt dec mx pq3 ext lo64)
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[mroeschke]

This looks fixed on master; could use a test:

In [110]: pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns)).dtypes.a.d.i
     ...:
Out[110]:
NaN  NaN    object
dtype: object

In [111]: pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns)).a.d.i.dtypes
     ...:
Out[111]:
NaN  NaN    object
dtype: object

[ganevgv]

Finally managed to reproduce the inconsistent behaviour on the specific pandas version ( pandas: 0.23.0.dev0+38.g6552718d2 on commit: 6552718d2a908a57b88354fe916609631793131e) using smaller input.

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: data = [['x',  1]]
In [4]: columns = [('a', 'b', np.nan), ('a', 'c', 0.0)]
In [5]: df = pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns))

In [6]: df.dtypes.a.b
Out[6]:
NaN    object
dtype: object

In [7]: df.a.b.dtypes
Out[7]:
NaN    float64
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: 6552718 python: 3.6.5.final.0 python-bits: 64 OS: Darwin OS-release: 17.7.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8

pandas: 0.23.0.dev0+38.g6552718d2
pytest: 5.2.2
pip: 19.3.1
setuptools: 39.0.1
Cython: 0.29.14
numpy: 1.17.3
scipy: 1.3.1
pyarrow: 0.15.1
xarray: 0.14.0
IPython: 7.9.0
sphinx: 2.2.1
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2019.3
blosc: 1.8.1
bottleneck: 1.2.1
tables: 3.6.1
numexpr: 2.7.0
feather: None
matplotlib: 3.1.1
openpyxl: 3.0.0
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.2.2
lxml: 4.4.1
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.3.10
pymysql: None
psycopg2: None
jinja2: 2.10.3
s3fs: 0.3.5
fastparquet: 0.3.2
pandas_gbq: None
pandas_datareader: None
None

This is the output on the current master (pandas: 0.26.0.dev0+734.g0de99558b):

In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: data = [['x',  1]]
In [4]: columns = [('a', 'b', np.nan), ('a', 'c', 0.0)]
In [5]: df = pd.DataFrame(data, columns=pd.MultiIndex.from_tuples(columns))

In [6]: df.dtypes.a.b
Out[6]:
NaN    object
dtype: object

In [7]: df.a.b.dtypes
Out[7]:
NaN    object
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : 0de9955 python : 3.6.5.final.0 python-bits : 64 OS : Darwin OS-release : 17.7.0 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8

pandas : 0.26.0.dev0+734.g0de99558b
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.0
pip : 19.3.1
setuptools : 39.0.1
Cython : 0.29.14
pytest : 5.2.2
hypothesis : 4.42.6
sphinx : 2.2.1
blosc : 1.8.1
feather : None
xlsxwriter : 1.2.2
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.9.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : 1.2.1
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.1.1
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
s3fs : 0.3.5
scipy : 1.3.1
sqlalchemy : 1.3.10
tables : 3.6.1
xarray : 0.14.0
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.2
None

PS: I found recovering this particular pandas dev version a bit annoying as the commit was not provided in the original issues so I'll quickly explain the method I used:

1. cd pandas to get into the pandas subfolder
2. git checkout git rev-list -1 --before="Jan 3 2018" master to checkout to master on the particular day
3. python -c 'from _version import get_versions; print(get_versions()["version"])' to check the pandas version on that branch

I had to iterate between steps 2. and 3. a few times until I got the correct pandas version. Please notice that I did 3. as a shortcut to get the current pandas version without the need to build the C extensions and install the requirements. Thanks for the help, @sofiane87!