The relu definition is a bit "scary" for array inputs.

[KristofferC]

The following seems like an easy mistake to make:

julia> x = randn(5)
5-element Array{Float64,1}:
 -1.0483606281411753
 -0.13524020825581384
 -2.184303973828124
  0.4308162306092796
 -0.673766167758703

julia> relu(x)
5-element Array{Float64,1}:
 0.0
 0.0
 0.0
 0.0
 0.0

julia> relu.(x)
5-element Array{Float64,1}:
 0.0
 0.0
 0.0
 0.4308162306092796
 0.0

[MikeInnes]

I agree but I'm not sure what to do about it. I guess we can just disable relu for arrays, and perhaps re-enable if someone complains about it.

[KristofferC]

Or add a vectorized definition for it.

relu(x::AbstractArray) = clamp.(x, 0, Inf)

or something.

[MikeInnes]

Strictly speaking you could argue that this changes the meaning of relu, and it's not consistent with the way that Base has gradually moved away from special-cased vectorised operations. But it's hard to imagine that being a big problem in practice.

Happy to take a PR for that fix.

[KristofferC]

Base has gradually moved away from special-cased vectorised operations.

Base typically have type annotations though

julia> sin(rand(5))
ERROR: MethodError: no method matching sin(::Array{Float64,1})

[KristofferC]

But yes, there are cases where f(x::AbstractArray) is not equal to f.(x), like for trigonometric functions for matrices.

[MikeInnes]

Yeah true. Maybe the better fix is to just define relu(::Real) only; it might make sense to do that across our activation functions.

[staticfloat]

I would support a help.jl file filled with things like:

relu(x::Array) = error("Use explicitly-broadcasted invocations such as `relu.(x)` to apply activation functions to tensors!")

These little gotchas can be very helpful to users coming from less-enlightened frameworks.

[staticfloat]

Fixed by #98