ReLU activation function support

[hatsunearu]

Is there any chance that FANN will support the ReLU function as a possible activation function?

[mushketer888]

It is also strange to me that such popular library doesnt have ReLU.

[hatsunearu]

Well, I moved onto Tensorflow because FANN isn't quite flexible.
I did implement ReLU and LReLU on my fork so if you want to try it out, go ahead.

https://github.com/hatsunearu/fann

[mushketer888]

ReLU is like simple rectifier as we know. it would be too easy just to write it in fann_activation.h. I just dont know do i need to include derivative also (math stuff)

[hatsunearu]

Huh?
If you look at the diff on my fork, you can see what I did to include ReLU and LReLU. The derivatives are calculated with an IFDEF that computes the slope.

[hatsunearu]

master...hatsunearu:master

I need to fix the indents someday and I'll do a PR, but considering the original author hasn't pulled any of the PRs lately, I don't think it's worth.

[mushketer888]

wow thanks!

[geekgirljoy]

Steffen seems like a busy guy. Have you messaged him about it and asked him if he would review the code if you did a pull request? I agree that adding ReLU and LReLU would only improve FANN and make it more useful to people. I vote for a PR, FANN needs ReLU and LReLU 👍 :)

[hatsunearu]

well he'll get a notification for a PR if someone made one, but considering he hasn't replied to any of the recent ones I doubt he'll pull my change.

I'll make it anyways if you all want to. In the meantime, you can use my fork.

[hatsunearu]

pull request created: #74

[geekgirljoy]

Thanks for adding this :-)

[radioman]

when using these both or one, the training doesn't progress, it's stuck at one number usualy 0.5 ???

fann_set_activation_function_hidden(ann, FANN_RELU);
fann_set_activation_function_output(ann, FANN_RELU);

[hatsunearu]

try changing the weight initialization. you might also be running into the issue where the gradient "dies" because the ReLU has 0 gradient where x<0.

[MatthewScholefield]

Just wondering, isn't ReLU essentially the same as the LINEAR_PIECE activation function in FANN?

[geekgirljoy]

Well, I moved onto Tensorflow because FANN isn't quite flexible.
I did implement ReLU and LReLU on my fork so if you want to try it out, go ahead.

@hatsunearu By chance you wouldn't know where one might obtain a PHP port of Tensorflow would you? :-P

[hatsunearu]

@geekgirljoy Suuuuper late to the thread, but you need to make some program to interface your PHP webapp to Tensorflow yourself. I haven't done this before, but try something like this: https://www.tensorflow.org/serving/ and also try searching something like "serving tensorflow" on Google.

Depending on your actual model, it might be best to make PHP call a Tensorflow script written in Python, appropriately package the result in Python, then return that back to PHP which appropriately packages it up and serves to the user.

[mrrstrat]

Anyone know what the status is on ReLU: Is it close to being made live?

[hatsunearu]

I thought this was merged?

[mrrstrat]

I checked in the code and I do not see it (unless I am looking in the wrong place). I only see it in some unmerged branches. One of the authors did get back me and he is working on his dissertation and plans to get back to it in a couple months. I did remind him that he may have to get the latest code and re-implement his changes as the branch he has is out of date.

[hatsunearu]

If you can check out the code and such I highly suggest you fork and merge all the pending changes--probably a better idea for the community. IIRC the original author can take your git tree.

[mrrstrat]

I looked at the main branch to see if it was live in production and it did not appear to be. I was trying to not work from an un-merged branch or one pending changes as much as trying to not jump in at this time :-). If that's all that's needed is to take his changes and refresh them for submission then I am willing to do so. I have been working with this codebase since about 2004 and am pretty comfortable with it.
I would have to touch base with @troiganto and to see if he can get to it soon.

[hatsunearu]

Sounds like the repo owner needs to add people as co-maintainers :P

[mrrstrat]

I am up for it :-)

[mrrstrat]

Really?? Its been literally two years and there still is no ReLU introduced into the codebase. Is FANN a dead codebase or should there be a formal branch peel off to keep up with where deep learning is heading? I have used FANN since 2005 and would like to see it keep up with the times as much as possible.