errors and inconsistent behaviour when using a DataFrame or a boolean Series as an index

[robbuckley]

Code Sample, a copy-pastable example if possible

In [97]:  s1 = pd.Series([1,2,3], index=[4,5,6])

In [98]:  s2 = pd.Series([1,3,2], index=s1)

In [99]: s2
Out[101]: 
1    1
2    3
3    2
dtype: int64

so far so good, now lets try a boolean series

In [101]: s3 = pd.Series([1,3,2], index=(s1==2))

In [102]: s3
...
...
~/anaconda3/lib/python3.5/site-packages/pandas/core/indexes/base.py in _format_with_header(self, header, na_rep, **kwargs)
   1905             values = np.array(values)
   1906         elif is_object_dtype(values.dtype):
-> 1907             values = lib.maybe_convert_objects(values, safe=1)
   1908 
   1909         if is_object_dtype(values.dtype):

TypeError: Argument 'objects' has incorrect type (expected numpy.ndarray, got Series)

In [103]: s3.index
...
...
~/anaconda3/lib/python3.5/site-packages/pandas/core/indexes/base.py in inferred_type(self)
   1567     def inferred_type(self):
   1568         """ return a string of the type inferred from the values """
-> 1569         return lib.infer_dtype(self)
   1570 
   1571     def _is_memory_usage_qualified(self):

pandas/_libs/src/inference.pyx in pandas._libs.lib.infer_dtype (pandas/_libs/lib.c:47002)()

ValueError: cannot infer type for <class 'NoneType'>

now a dataframe

In [178]: df = pd.DataFrame([[1,2],[3,4],[5,6]], index=[3,6,9])

In [180]: s4 = pd.Series([1,3,2], index=df)

In [181]: s4
...
...
~/anaconda3/lib/python3.5/site-packages/pandas/io/formats/format.py in <lambda>(x)
   1967 
   1968     def _format_strings(self):
-> 1969         formatter = self.formatter or (lambda x: '% d' % x)
   1970         fmt_values = [formatter(x) for x in self.values]
   1971         return fmt_values

TypeError: %d format: a number is required, not numpy.ndarray

In [182]: s4.index
Out[182]: Int64Index([[1, 2], [3, 4], [5, 6]], dtype='int64')

Problem description

I would expect passing a boolean Series as the index= parameter to either act as Series.index (as in the example where the integer Series s1 is used as an index) or Series.values (as the docs seem imply http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html "index : array-like or Index (1d)")

For the DataFrame case, it could either use df.index, or fail early with a ValueError

[this should explain why the current behaviour is a problem and why the expected output is a better solution.]

Note: We receive a lot of issues on our GitHub tracker, so it is very possible that your issue has been posted before. Please check first before submitting so that we do not have to handle and close duplicates!

Note: Many problems can be resolved by simply upgrading pandas to the latest version. Before submitting, please check if that solution works for you. If possible, you may want to check if master addresses this issue, but that is not necessary.

For documentation-related issues, you can check the latest versions of the docs on master here:

https://pandas-docs.github.io/pandas-docs-travis/

If the issue has not been resolved there, go ahead and file it in the issue tracker.

Expected Output

either

In [101]: s3 = pd.Series([1,3,2], index=(s1==2))
In [102]: s3
Out[119]: 
4    1
5    3
6    2
dtype: int64

In [103]: s4 = pd.Series([1,2,3], index=df)
In [104]: s4
Out[104]: 
4    1
5    2
6    3
dtype: int64

OR

In [121]: s3
Out[121]: 
False    1
True     3
False    2
dtype: int64

s4 = pd.Series([1,3,2], index=df)
ValueError

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.5.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.4.0
Cython: 0.26
numpy: 1.12.1
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 0.9.8
lxml: 3.8.0
bs4: 4.6.0
html5lib: 0.9999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[jreback]

@robbuckley not really clear why you would do this, e.g. pass a DataFrame to an Index. I believe we are going to raise on this (just discussing with @toobaz can't quite find the issue).

And a boolean index is a quite strange beast.

[toobaz]

@robbuckley three things are sure:

• if you pass something as index=something, it should not go in the Series values
• whatever behavior we choose for index= for Series, DataFrame should behave the same.
• if an error must arise, it must be in the constructor call, not later

The rest is a bit less obvious. Boolean indexes are strange beasts at least for the general reason that non-unique indexes are strange beasts. But I personally think we will end up supporting them, for consistency (in particular with unstacking of boolean levels from MultiIndexes, which make sense instead). See #15890 (@jreback probably not what you were looking for however) and maybe this comment.

This said, even if not introducing a BooleanIndex (or until we do), my preference for the above is to cast to object, rather than to raise an error.

[jreback]

@toobaz yeah I'll repurpose this issue a bit to enhance error reporting w.r.t. nested-list likes to Index, e.g. if you pass a DataFrame I would expect an error, even though technically you could make a MI.

[toobaz]

Sorry, I totally misunderstood the original report, so my point "if you pass something as index=something, it should not go in the Series values" made no sense. Let me try again:

• if you pass a Series as index=s, it should use s.values, not s.index
• I'm +1 on allowing a DataFrame to be casted to MultiIndex , but I agree it's debatable. Related to Indexing a MultiIndex with a (Multi)Index #15472 I guess.
• if an error must arise, it must be in the constructor call, not later

the discussion on BooleanIndex still holds, I think.

[robbuckley]

@jreback i dont have a specific use case, I just stumbled across it. IMHO they should either raise early or be supported in a consistent way. It would be in the spirit of the docs to use s.values for all cases, casting df.values into a multiindex. But a raise on any attempt to use a Series or df as an index would be better than the current behaviour, i think

Currently the constructors succeed for all these examples, but various things fail down the line, in more or less confusing ways -- except for the integer Series example, where it behaves as if s.index were passed.

The dataframe case is especially confusing - at a first [careless] glance, it appears to have kind-of worked -- s.index looks as if it has 2x1 ndarrays as the keys. But in fact s.index.values is a 2x3 ndarray and bad things happen when i try to do other things with the series.

[mroeschke]

These looks reasonable on master. Could use a test

In [47]: s3 = pd.Series([1,3,2], index=(s1==2))

In [48]: s3
Out[48]:
False    1
True     3
False    2
dtype: int64

In [49]: df = pd.DataFrame([[1,2],[3,4],[5,6]], index=[3,6,9])

In [50]: s4 = pd.Series([1,3,2], index=df)
ValueError: Index data must be 1-dimensional