pReLU eta must be between -1 and 1

[cossio]

We can enforce this constraint by parameterizing eta, for example:

eta = tanh(xi)

where now xi is unbounded.

We can use something faster than tanh. Here are some examples:

This suggests that the faster alternative is:

eta = xi / (1 + abs(xi))

[cossio]

We don't have to parameterize eta. Rather, we can use a different parameter. So, instead of using

which is required to be between -1 and 1, we can use:

which is unbounded.

The relation between the two is:

[cossio]

Closed in e774670