Skip to content

Commit

Permalink
Merge branch 'master' into sf/overhaul
Browse files Browse the repository at this point in the history
  • Loading branch information
staticfloat authored Mar 28, 2019
2 parents ea6498e + 2bd7e8a commit 63dbfa2
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 1 deletion.
2 changes: 1 addition & 1 deletion src/activation.jl
Original file line number Diff line number Diff line change
Expand Up @@ -120,4 +120,4 @@ softsign(x) = x / (one(x) + abs(x))
See [Deep Sparse Rectifier Neural Networks](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf).
"""
softplus(x) = log1p(exp(x))
softplus(x) = ifelse(x > 0, x + log1p(exp(-x)), log1p(exp(x)))
2 changes: 2 additions & 0 deletions test/activation.jl
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ end
@test gelu(1.0) == 0.8411919906082768
@test swish(1.0) == 1.0 / (1.0 + exp(-1.0))
@test softplus(1.0) log(exp(1.0) + 1.0)
@test softplus(1e8) 1e8
@test softplus(-1e8) 0.0
@test softsign(1.0) == 0.5
@test selu(1.0) == 1.0507009873554804934193349852946

Expand Down

0 comments on commit 63dbfa2

Please sign in to comment.