Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[MXNET-978] Fully connected, higher order grad #14779

Merged
merged 33 commits into from
Sep 30, 2019
Merged

[MXNET-978] Fully connected, higher order grad #14779

merged 33 commits into from
Sep 30, 2019

Conversation

larroy
Copy link
Contributor

@larroy larroy commented Apr 23, 2019 •
edited
Loading

Description

Add backward for fully connected backward.

This is not strictly correct if X or W depend on each other. After discussing with the community this a valid assumption that we can make (that the weights don't depend on the inputs).

This PR adds functionality required to get second order gradients of models using FullyConnected operator such as simple MLP.

#10002
#14619
Resolves #14991

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@larroy larroy force-pushed the fc_higher_order_grad branch from a6ca88a to f2a63de Compare April 23, 2019 21:49
@larroy
Copy link
Contributor Author

larroy commented Apr 23, 2019

@mxnet-label-bot add [Autograd, Backend, pr-work-in-progress, pr-awaiting-review]

@marcoabreu marcoabreu added Autograd Backend Issues related to the backend of MXNet pr-awaiting-review PR is waiting for code review pr-work-in-progress PR is still work in progress labels Apr 23, 2019
@apeforest
Copy link
Contributor

@larroy Thanks for the contribution. Could you please add unit test and change the title if it's ready for review?

@apeforest
Copy link
Contributor

Why are there changes in submodules?

@larroy
Copy link
Contributor Author

larroy commented Apr 24, 2019

Ignore submodules, I guess I was out of date. I will sync them on the next commits.

@larroy larroy force-pushed the fc_higher_order_grad branch 2 times, most recently from 01e6c92 to 25fda1d Compare April 26, 2019 21:00
@larroy larroy force-pushed the fc_higher_order_grad branch from 25fda1d to 684fac7 Compare May 6, 2019 06:34
@larroy larroy force-pushed the fc_higher_order_grad branch 2 times, most recently from bca77ba to 855e7d8 Compare May 24, 2019 21:49
@larroy larroy changed the title [WIP] Fully connected, higher order grad Fully connected, higher order grad May 28, 2019
@larroy larroy force-pushed the fc_higher_order_grad branch 2 times, most recently from c3326f1 to 7458af5 Compare June 5, 2019 18:50
@larroy
Copy link
Contributor Author

larroy commented Jun 5, 2019 •
edited
Loading

@apeforest @kshitij12345 please review

@larroy
Copy link
Contributor Author

larroy commented Jun 5, 2019

@mxnet-label-bot remove [pr-work-in-progress]

@marcoabreu marcoabreu removed the pr-work-in-progress PR is still work in progress label Jun 5, 2019
@larroy larroy changed the title Fully connected, higher order grad [Don't merge] Fully connected, higher order grad Jun 6, 2019
@larroy
Copy link
Contributor Author

larroy commented Jun 6, 2019 •
edited
Loading

Working on fixing this, added the differential calculus developments for review and reference:
image-b8b4fdcb-128a-462c-909d-4438a3639c3a

Left here just for reference. This development is incomplete and incorrect in some cases, I refined this further afterwards.

@larroy larroy force-pushed the fc_higher_order_grad branch from 36c6a73 to 256dfb2 Compare September 11, 2019 11:12
@sxjscience sxjscience merged commit 3ffd2c2 into apache:master Sep 30, 2019
@larroy larroy mentioned this pull request Oct 10, 2019
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@kshitij12345 kshitij12345 kshitij12345 left review comments

@sxjscience sxjscience sxjscience approved these changes

@apeforest apeforest apeforest approved these changes

@anirudh2290 anirudh2290 Awaiting requested review from anirudh2290

@eric-haibin-lin eric-haibin-lin Awaiting requested review from eric-haibin-lin eric-haibin-lin is a code owner

Assignees
No one assigned
Labels
Autograd Backend Issues related to the backend of MXNet pr-awaiting-merge Review and CI is complete. Ready to Merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Second order gradient wrt inputs, expected behaviour.
5 participants
@larroy @apeforest @kshitij12345 @sxjscience @marcoabreu