Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slightly faster softplus #443

Merged
merged 1 commit into from
Nov 18, 2022
Merged

Slightly faster softplus #443

merged 1 commit into from
Nov 18, 2022

Conversation

Sleort
Copy link
Contributor

@Sleort Sleort commented Nov 18, 2022

A few percent faster softplus using relu(x) instead of max(x, 0). Ref. #347.

A few percent faster `softplus` using `relu(x)` instead of `max(x, 0)`. Ref. FluxML#347.
@CarloLucibello
Copy link
Member

makes sense, thanks!

@CarloLucibello CarloLucibello merged commit f7597d9 into FluxML:master Nov 18, 2022
@ToucheSir
Copy link
Member

It would've been nice to try the new benchmarking apparatus on this, but oh well. @Sleort if you notice any other low-hanging fruit, maybe we can try then.

@mcabbott
Copy link
Member

mcabbott commented Nov 20, 2022

BTW, it is possible to go much faster here:

julia> @btime y .= softplus.(x)   setup=(x=randn(Float32,1000); y=similar(x));
  19.375 μs (0 allocations: 0 bytes)

julia> @btime y .= g8.(x)   setup=(x=randn(Float32,1000); y=similar(x));
  9.541 μs (0 allocations: 0 bytes)

where it appears I used exp(x) for large negative x, then a polynomial correction, then a rational -- I don't remember but perhaps this seemed too complicated:

@inline f7(x) = evalpoly(x, 
    (0.6931472f0, 0.33953527f0, 0.09100373f0, 0.016568843f0, 0.0013059091f0, 1.8928275f-6)
)/evalpoly(x, 
    (1.0f0, 0.21118937f0, 0.11348734f0, 0.0145165585f0, 0.0014029884f0)
)

@inline f8(x) = evalpoly(x, 
    (0.99217963f0, 0.0027701536f0, -0.00039373388f0, 2.8039218f-5, -9.995612f-7, 1.4259708f-8)
)

@inline g8(x) = begin ax = abs(x)
    eax = @fastmath exp(-ax)
    ifelse(x < -15, eax, ifelse(x < -10, eax * f8(ax), eax * f7(ax) + relu(x)))
end

@Sleort Sleort deleted the patch-1 branch October 5, 2023 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants