Enable label-based indexing with .loc even when DataFrame is not lexsorted

[dr-leo]

As shown in the following code example, .loc is more restrictive than the deprecated .ix. Label-based indexing without slicing should be possible with .loc even when the dataframe is not lexsorted. The docs could be clearer in this respect. In addition, performance warnings should ideally be suppressible through options.

In [1]: from pandasdmx import *

In [2]: estat = Request('estat')

In [3]: dsd_resp = estat.datastructure('DSD_une_rt_a')

In [4]: df = dsd_resp.write().codelist

In [5]: df.ix[['AGE', 'UNIT']]
C:\Users\stefan\Anaconda3\envs\py35\Scripts\ipython-script.py:1:
DeprecationWarn
ing:
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate_ix

if name == 'main':
Out[5]:
dim_or_attr name
AGE AGE D AGE
TOTAL D Total
Y25-74 D From 25 to 74 years
Y_LT25 D Less than 25 years
UNIT UNIT D UNIT
PC_ACT D Percentage of active population
PC_POP D Percentage of total population
THS_PER D Thousand persons

In [6]: df.loc[['AGE', 'UNIT']]

UnsortedIndexError Traceback (most recent call last)
in ()
----> 1 df.loc[['AGE', 'UNIT']]

C:\Users\stefan\Anaconda3\envs\py35\lib\site-packages\pandas\core\indexing.py
in
getitem(self, key)
1326 else:
1327 key = com._apply_if_callable(key, self.obj)
-> 1328 return self._getitem_axis(key, axis=0)
1329
1330 def _is_scalar_access(self, key):

C:\Users\stefan\Anaconda3\envs\py35\lib\site-packages\pandas\core\indexing.py
in
_getitem_axis(self, key, axis)
1543 # nested tuple slicing
1544 if is_nested_tuple(key, labels):
-> 1545 locs = labels.get_locs(key)
1546 indexer = [slice(None)] * self.ndim
1547 indexer[axis] = locs

C:\Users\stefan\Anaconda3\envs\py35\lib\site-packages\pandas\core\indexes\multi.
py in get_locs(self, tup)
2267 'to be fully lexsorted
tuple len ({
0}), '
2268 'lexsort depth ({1})'
-> 2269 .format(len(tup),
self.lexsort_dept
h))
2270
2271 # indexer

UnsortedIndexError: 'MultiIndex Slicing requires the index to be fully
lexsorted
tuple len (1), lexsort depth (0)'

I'd like to see .loc do the job as .ix did, maybe with a performance
warning. Ideally the latter should be suppressible by setting an option
rather than wrapping everything in a context manager from the warnings
stdlib module.

[toobaz]

Label-based indexing without slicing should be possible with .loc even when the dataframe is not lexsorted

Notice this is specific to indexing with lists (and booleans, but not single labels). Anyway, I'll push a PR in minutes.

[dr-leo]

Great! To add one perhaps self-evident corner case: in a multi-indexed df, ':' should be permissible as well. df.loc[['a', 'c'], :, ['two', 'one']] This is not a tested code example. But the idea should be clear: If the slice captures the entire row/col, sorting should not be required either. Am 20.06.2017 um 11:03 schrieb Pietro Battiston:

…

Label-based indexing without slicing should be possible with .loc even when the dataframe is not lexsorted Notice this is specific to indexing with lists (and booleans, but not single labels). Anyway, I'll push a PR in minutes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16734 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADA58tH0g-2wYxOZpP_7dFMyrQ35Nzbxks5sF4r4gaJpZM4N_QR5>.

[toobaz]

If the slice captures the entire row/col, sorting should not be required either.

Yes, the fix contemplates this use case.

[jreback]

well @toobaz pushed a fix, but ideally you would have a copy-pastable example in the top section. IOW is just code and not relying on external library.

[toobaz]

For future reference:

In [2]: _mklbl = lambda prefix, n: ["%s%s" % (prefix, i) for i in range(n)]

In [3]: index = MultiIndex.from_product([_mklbl('A', 4), _mklbl('B', 2),
   ...:                                  _mklbl('C', 4), _mklbl('D', 2)])

In [4]: columns = MultiIndex.from_tuples([('a', 'foo'), ('a', 'bar'),
   ...:                                   ('b', 'foo'), ('b', 'bah')],
   ...:                                  names=['lvl0', 'lvl1'])

In [5]: df = DataFrame(-1, index=index, columns=columns)

In [6]: df.loc['A1', ('a', slice(None))]
---------------------------------------------------------------------------
UnsortedIndexError                        Traceback (most recent call last)
<ipython-input-6-0160dbc496d0> in <module>()
----> 1 df.loc['A1', ('a', slice(None))]

/home/nobackup/repo/pandas/pandas/core/indexing.py in __getitem__(self, key)
   1327             except (KeyError, IndexError):
   1328                 pass
-> 1329             return self._getitem_tuple(key)
   1330         else:
   1331             key = com._apply_if_callable(key, self.obj)

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_tuple(self, tup)
    834     def _getitem_tuple(self, tup):
    835         try:
--> 836             return self._getitem_lowerdim(tup)
    837         except IndexingError:
    838             pass

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_lowerdim(self, tup)
    946         # we may have a nested tuples indexer here
    947         if self._is_nested_tuple_indexer(tup):
--> 948             return self._getitem_nested_tuple(tup)
    949 
    950         # we maybe be using a tuple to represent multiple dimensions here

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_nested_tuple(self, tup)
   1025 
   1026             current_ndim = obj.ndim
-> 1027             obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
   1028             axis += 1
   1029 

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1547             # nested tuple slicing
   1548             if is_nested_tuple(key, labels):
-> 1549                 locs = labels.get_locs(key)
   1550                 indexer = [slice(None)] * self.ndim
   1551                 indexer[axis] = locs

/home/nobackup/repo/pandas/pandas/core/indexes/multi.py in get_locs(self, tup)
   2267                                      'to be fully lexsorted tuple len ({0}), '
   2268                                      'lexsort depth ({1})'
-> 2269                                      .format(len(tup), self.lexsort_depth))
   2270 
   2271         # indexer

UnsortedIndexError: 'MultiIndex Slicing requires the index to be fully lexsorted tuple len (2), lexsort depth (1)'

In [7]: df.loc['A1', ('a', ['foo'])]
---------------------------------------------------------------------------
UnsortedIndexError                        Traceback (most recent call last)
<ipython-input-7-84c90c00aa28> in <module>()
----> 1 df.loc['A1', ('a', ['foo'])]

/home/nobackup/repo/pandas/pandas/core/indexing.py in __getitem__(self, key)
   1327             except (KeyError, IndexError):
   1328                 pass
-> 1329             return self._getitem_tuple(key)
   1330         else:
   1331             key = com._apply_if_callable(key, self.obj)

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_tuple(self, tup)
    834     def _getitem_tuple(self, tup):
    835         try:
--> 836             return self._getitem_lowerdim(tup)
    837         except IndexingError:
    838             pass

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_lowerdim(self, tup)
    946         # we may have a nested tuples indexer here
    947         if self._is_nested_tuple_indexer(tup):
--> 948             return self._getitem_nested_tuple(tup)
    949 
    950         # we maybe be using a tuple to represent multiple dimensions here

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_nested_tuple(self, tup)
   1025 
   1026             current_ndim = obj.ndim
-> 1027             obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
   1028             axis += 1
   1029 

/home/nobackup/repo/pandas/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1547             # nested tuple slicing
   1548             if is_nested_tuple(key, labels):
-> 1549                 locs = labels.get_locs(key)
   1550                 indexer = [slice(None)] * self.ndim
   1551                 indexer[axis] = locs

/home/nobackup/repo/pandas/pandas/core/indexes/multi.py in get_locs(self, tup)
   2267                                      'to be fully lexsorted tuple len ({0}), '
   2268                                      'lexsort depth ({1})'
-> 2269                                      .format(len(tup), self.lexsort_depth))
   2270 
   2271         # indexer

UnsortedIndexError: 'MultiIndex Slicing requires the index to be fully lexsorted tuple len (2), lexsort depth (1)'