Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Refines NDArray indexing and adds numpy ndarray indexing [READY FOR REVIEW] #15942

Merged
merged 3 commits into from
Aug 27, 2019
Merged

Refines NDArray indexing and adds numpy ndarray indexing [READY FOR REVIEW] #15942

merged 3 commits into from
Aug 27, 2019

Conversation

zoeygxy
Copy link
Contributor

@zoeygxy zoeygxy commented Aug 19, 2019

Description

  • Fixed NDArray indexing: newaxis and setitem.
  • Added support for numpy indexing, including basic indexing and advanced integer array indexing, None and Ellipsis.
    https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html
  • Added np.full operator interface; test cases not added.
  • Wrapped slice_assign and slice_assign_scalar method for both numpy.ndarray and NDArray, doc added.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Fixed NDArray indexing: newaxis and setitem.
  • Added support for numpy indexing, including basic and advanced indexing, None and Ellipsis.

Comments

  • Need to add test cases for np.full.
  • Boolean Array indexing not supported yet.

@zoeygxy
Copy link
Contributor Author

zoeygxy commented Aug 19, 2019

@haojin2 @reminisce

reminisce
reminisce reviewed Aug 19, 2019
View reviewed changes
python/mxnet/ndarray/numpy/_op.py Outdated Show resolved Hide resolved
python/mxnet/ndarray/ndarray.py
@@ -2508,6 +2619,88 @@ def copy(self):
"""
return self.copyto(self.context)

def slice_assign_scalar(self, value, begin, end, step):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why a wrapper on top of _internal._slice_assign_scalar is needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To enable polymorphism and reuse code as much as possible. I reused NDArray's _set_nd_basic_indexing function in ndarray class and wrapping slice_assign_scalar and slice_assign as class methods can let python automatically call NDArray operators in NDArray processing, and ndarray ones in ndarray processing. If you feel such handling confusing, I can also copy-paste the entire _set_nd_basic_indexing function to multiarray.py.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for some other operators like _full and scatter_set_nd.

python/mxnet/ndarray/ndarray.py
"""
return _internal._slice_assign_scalar(self, value, begin=begin, end=end, step=step, out=self)

def slice_assign(self, rhs, begin, end, step):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

python/mxnet/ndarray/ndarray.py
@@ -2678,20 +2871,32 @@ def to_dlpack_for_write(self):
"""
return to_dlpack_for_write(self)

def _full(self, value):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

python/mxnet/ndarray/ndarray.py
"""
return _internal._full(self.shape, value=value, ctx=self.context, dtype=self.dtype, out=self)

def _scatter_set_nd(self, value_nd, indices):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

python/mxnet/ndarray/numpy/_op.py Outdated Show resolved Hide resolved
python/mxnet/ndarray/ndarray.py Outdated Show resolved Hide resolved
python/mxnet/ndarray/numpy/_op.py Outdated Show resolved Hide resolved
python/mxnet/ndarray/numpy/_op.py Outdated Show resolved Hide resolved
@zoeygxy zoeygxy force-pushed the numpy_indexing_master branch from 1973c3a to e50c351 Compare August 20, 2019 05:37
@haojin2 haojin2 self-assigned this Aug 20, 2019
@haojin2 haojin2 added the Numpy label Aug 20, 2019
@zoeygxy zoeygxy marked this pull request as ready for review August 20, 2019 09:12
@zoeygxy zoeygxy requested a review from szha as a code owner August 20, 2019 09:12
@zoeygxy zoeygxy force-pushed the numpy_indexing_master branch from 7f71f00 to e17a563 Compare August 20, 2019 09:28
@zoeygxy
Copy link
Contributor Author

zoeygxy commented Aug 20, 2019

Waiting for CI result. Still fixing style.

@zoeygxy zoeygxy force-pushed the numpy_indexing_master branch 4 times, most recently from 2ab9211 to 40459bd Compare August 23, 2019 08:13
@zoeygxy
Copy link
Contributor Author

zoeygxy commented Aug 23, 2019

Fixed SliceAxisGrad_ and now supports subscription on zero-size shapes (eg. x = np.ones((3, 0)))

@zoeygxy zoeygxy force-pushed the numpy_indexing_master branch 2 times, most recently from 5f29819 to 9ff4d57 Compare August 26, 2019 03:01
@zoeygxy zoeygxy changed the title Refines NDArray indexing and adds numpy ndarray indexing [DO NOT MERGE YET] Refines NDArray indexing and adds numpy ndarray indexing [READY FOR REVIEW] Aug 26, 2019
reminisce
reminisce reviewed Aug 26, 2019
View reviewed changes
src/operator/tensor/matrix_op-inl.h Outdated Show resolved Hide resolved
src/operator/tensor/matrix_op-inl.h Outdated Show resolved Hide resolved
tests/python/unittest/test_numpy_ndarray.py Outdated Show resolved Hide resolved
@zoeygxy zoeygxy force-pushed the numpy_indexing_master branch from 9ff4d57 to 2052fb7 Compare August 27, 2019 08:50
@reminisce reminisce merged commit 8df9469 into apache:master Aug 27, 2019
This was referenced Sep 2, 2019
Closed
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@reminisce reminisce reminisce left review comments

@szha szha Awaiting requested review from szha szha is a code owner

Assignees

@haojin2 haojin2

Labels
Numpy
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zoeygxy @reminisce @haojin2