Leakyrelu

[ordicker]

Hi,

Sorry for disappearing, it was a bit crazy here for the last 7 months.
Anyway, I have added Leakyrelu support with the onnx_backend tests.

I have found that there are some discrepancies between pytorch and NNlib.jl (in about 1e-8), So I have changed == to \approx link
What do you think about that?

PR Checklist

• Tests are added
• Documentation, if applicable
  internal

[ordicker]

Not sure why julia-1.6 failed, do we have to support it?

[dfdx]

The test fails because in earlier versions Julia generates separate methods for functions with and without keywords. The type of leakyrelu with keywords is actually var"#leakyrelu##kw, which you do not cover in the save!() method. The simplest way to support kw version to is to use @opconfig_kw() macro - see save!() for convolution for an example. Would it be possible to make this little change in this PR?

[ordicker]

Ok that was super simple, sorry for the delay, I had problems reverting to julia1.6 on my computer.
As for the approx vs ==, do you think it's better to solve it in this package or in NNlib.jl?

BTW, yolo-v2 is loading and running. Hopefully, it will output the right results.

[dfdx]

I see it works now, thank you!

[dfdx]

I had problems reverting to julia1.6 on my computer

Usually, I simply remove ~/.julia and start over. But it may be painful sometimes.

As for the approx vs ==, do you think it's better to solve it in this package or in NNlib.jl?

Little inconsistencies may come from different implementations of the underlying stack such as BLAS. The difference in the order of 1e-8 is considered absolutely normal.