Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[operator] Add Mish Activation Function #20320

Merged
merged 11 commits into from
Jun 8, 2021
Merged

[operator] Add Mish Activation Function #20320

merged 11 commits into from
Jun 8, 2021

Conversation

Adnios
Copy link
Contributor

@Adnios Adnios commented May 28, 2021 •
edited
Loading

Description

Add Mish Activation Function.
Related issus: #16841
The pr(#17696) seem to be dead.

Checklist

Essentials

  • PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

Changes

  • Add native mish implementation
  • Integrate oneDNN implementation

Comments

  • This is mostly a copycat of log_sigmoid activation function. I referenced a lot of code during the implementation.

@Adnios Adnios requested a review from szha as a code owner May 28, 2021 04:58
@mxnet-bot
Copy link

Hey @Adnios , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [centos-gpu, unix-cpu, windows-gpu, miscellaneous, clang, website, windows-cpu, sanity, edge, centos-cpu, unix-gpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@Adnios Adnios marked this pull request as draft May 28, 2021 04:59
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels May 28, 2021
@Adnios
fix test_mish
2ff4d57 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix test_mish_gpu
3e9648c 
Signed-off-by: Adnios <[email protected]>
@Adnios Adnios force-pushed the mish branch 3 times, most recently from 3a11899 to 47a52f3 Compare May 29, 2021 07:12
@Adnios Adnios marked this pull request as ready for review May 29, 2021 10:42
@mseth10 mseth10 added pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress labels May 29, 2021
szha
szha reviewed Jun 1, 2021
View reviewed changes
Copy link
Member

@szha szha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Going forward in 2.0, as we will mainly use np/npx instead of sym, could you also include a test for it with npx.activation?

@Adnios
Copy link
Contributor Author

Adnios commented Jun 2, 2021

Thanks for the contribution! Going forward in 2.0, as we will mainly use np/npx instead of sym, could you also include a test for it with npx.activation?

Sure. Thanks for advice.

@Adnios Adnios closed this Jun 2, 2021
@Adnios Adnios reopened this Jun 2, 2021
@mseth10 mseth10 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-review PR is waiting for code review pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 2, 2021
@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 2, 2021
@Adnios Adnios marked this pull request as draft June 2, 2021 16:05
@Adnios
backward error test
8ea2560 
Signed-off-by: Adnios <[email protected]>
@Adnios
mv test_mish
aa40b43 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix backward bug
5bcd76a 
Signed-off-by: Adnios <[email protected]>
@Adnios
fix backward bug in cudnn
025f8de 
Signed-off-by: Adnios <[email protected]>
@Adnios Adnios marked this pull request as ready for review June 5, 2021 04:56
@Adnios
Copy link
Contributor Author

Adnios commented Jun 5, 2021

@mxnet-bot run ci [centos-gpu, unix-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [centos-gpu, unix-gpu]

@mseth10 mseth10 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jun 5, 2021
@Adnios Adnios requested a review from szha June 7, 2021 00:45
@Adnios
Copy link
Contributor Author

Adnios commented Jun 8, 2021

@szha Please help review

@szha szha merged commit 7740cca into apache:master Jun 8, 2021
@szha
Copy link
Member

szha commented Jun 8, 2021

@Adnios merged. Thank you for the contribution!

ptrendx
ptrendx reviewed Jun 9, 2021
View reviewed changes
src/common/cuda/rtc/forward_functions-inl.h
template <typename DType>
__device__ inline DType mish(const DType val) {
if (type_util::has_double_or_integral<DType>::value) {
return val * ::tanh(::log(1 + ::exp(val)));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that could be improved here (I did not notice this PR earlier, sorry for a late feedback) is the numerical stability of the softrelu part - see the implementation of the softrelu (it switches to softrelu(x) = x for large values of x to avoid overflow). @Adnios could you open another PR changing e.g. this function to

return val * op::tanh(op::softrelu(val));

(the double vs float is handled in op::tanh and op::softrelu anyway so this one will also be simpler as a result) and similarly backward?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed, usually Softplus has an upper bound of 20.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Thanks for your advice.

@Adnios Adnios mentioned this pull request Jun 11, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@digantamisra98 digantamisra98 digantamisra98 left review comments

@ptrendx ptrendx ptrendx left review comments

@szha szha Awaiting requested review from szha szha is a code owner

Assignees
No one assigned
Labels
pr-awaiting-review PR is waiting for code review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Adnios @mxnet-bot @szha @ptrendx @digantamisra98 @mseth10