Erlang Neural Networks (enn) is an application to implement artificial inteligence based on artificial neural networks (ANN).
Create your own project with rebar3.
$ rebar3 new app yourapp
Then in your project path find rebar.config file and add enn as dependency under the deps key:
{deps,
[
{enn, {git, "https://github.com/BorjaEst/enn.git", {tag, "<version>"}}}
]}.
Then using compile command, rebar3 will fetch the defined dependencies and compile them as well for your application.
$ rebar3 compile
At the end for making a release you first need to create your release structure and then making a release with following commands.
$ rebar3 new release yourrel
$ rebar3 release
You can find more information about dependencies in rebar3 - dependencies.
Load the app using your prefered method. For example in the project folder executing rebar3 shell:
$ rebar3 shell
===> Booted enn
All user functions are defined inside the module src/enn, however here is an example:
First of all I woudl initialize the observer, so you can see the loads of the system and the ETS tables:
1> observer:start().
ok
Here you can find a table nn_pool with all the currently enabled neural netwroks.
You can create a neural network simply with enn:start/1:
2> Network = enn:start(
2> #{inputs => layer:input(2, #{hidden1 => sequential}),
2> hidden1 => layer:sigmoid(4, #{hidden2 => sequential}),
2> hidden2 => layer:dense(3, #{outputs => sequential}),
2> outputs => layer:output(2, #{})}).
{network,#Ref<0.367976965.4190896130.201756>}
It is important to save the "Network id", you will need it to stop the network.
Another option is to first compile the model and run it after in 2 steps:
2> {atomic, Network} = mnesia:transaction(
2> fun() -> nnet:compile(
2> #{inputs => layer:input(2, #{hidden1 => sequential}),
2> hidden1 => layer:sigmoid(4, #{hidden2 => sequential}),
2> hidden2 => layer:dense(3, #{outputs => sequential}),
2> outputs => layer:output(2, #{})})
2> end).
{atomic,{network, #Ref<0.367976965.4190896130.204258>}}
3> Network = enn:start(Network).
{network,#Ref<0.367976965.4190896130.204258>}
Then you should gerenate your training, for example:
3> Loops = 8000,
3> Inputs = [[rand:uniform()-0.5, rand:uniform()-0.5] || _ <- lists:seq(1, Loops)],
3> Optima = [[I1+I2, I1-I2] || [I1, I2] <- Inputs],
3> ok.
ok
This operation is done by enn:fit/3:
4> enn:fit(Network, Inputs, Optima),
4> ok.
800 [==>.................] loss: 0.5324397056244214
1600 [====>...............] loss: 0.4869926512788517
2400 [======>.............] loss: 0.4189957818517215
3200 [========>...........] loss: 0.3559475178487766
4000 [==========>.........] loss: 0.3010031908365078
4800 [============>.......] loss: 0.2503973090555499
5600 [==============>.....] loss: 0.1866318856840655
6400 [================>...] loss: 0.1127141419963432
7200 [==================>.] loss: 0.0537037399650812
8000 [====================] loss: 0.0284808195771375
ok
This operation is done by enn:predict/2:
5> enn:predict(Network, [
5> [0.1, 0.6],
5> [0.3, 0.2],
5> [0.1, 0.1]
5> ]).
[[0.6172463551362715, -0.47078835817371445 ],
[0.49594263285192497, 0.0965897828109617 ],
[0.21946863843177294,-0.010693899601060347]]
For more options such log the cycles, or do not do not print the results, explore the options in the function enn:run/4.
You can easily do it with enn:stop/1:
6> enn:stop(Network).
ok
You will see in the observer window the network is gone.
You can resume your network with enn:start/1, the last network status is saved!
Just call enn:clone/1
7> {atomic, Clone_id} = mnesia:transaction(fun() -> nnet:clone(Network) end).
{atomic,{network,#Ref<0.3730030165.1951137794.240054>}}
Restart the network with enn:start/1:
8> enn:start(Network),
8> enn:predict(Network, [
8> [0.1, 0.6],
8> [0.3, 0.2],
8> [0.1, 0.1]
8> ]).
[[0.6172463551362715, -0.47078835817371445 ],
[0.49594263285192497, 0.0965897828109617 ],
[0.21946863843177294,-0.010693899601060347]]]
The folder test includes a module test_architectures.erl where you can find a lot of useful examples of models. Feel free to propose yours.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
In progress:
- Momentum, not well implemented on enn, review. This helps a lot when not using batch normalisation.
- Boolean activation function.
- Try which derivade behaves better (and if can be common to some activations).
Erlang performance:
- Use binary for long messages to speedup communications.
- Runtime connections change: ETS table with connections [{{in,N2},N1}, {{out,N1},N2}]
Speed-up training:
- ELU activation function seems by papers to perform better than Sigmoid and ReLU. Leaky ReLU is good as well.
Find correct solution:
- none for now
Importants to be clasified:
- Normalisation: Batch normalisation
- Regularization: dropout
- Optimizer: Adam
- Learning rate schedule: None
- Dropout, every training step (except output neurons) has a probability to be ignored during that training step. The weights have to be multiplied by this factor after training.
- Play with learning rate:
- After some trainings, reduce the value a defined value
- Reduce if the error grows / Proportional to error
- Exponential reduction
- Power scheduling
- Weights close to 0 must be set to 0, so in the next construction are removed
- Implement optimisers: -Nesterov Accelerated Gradient, performs better than momentum -AdaGrad, good idea but not efficient on the last steps -RMSProp, improvement of AdaGrad -Adam Optimization, adaptative moment estimation. This is the best in almost all cases.
Nice to try:
-
New Input attribute; Correlation, measures how much the input changed in relation with the error. For example a neuron with inputs A,B,C but optima is A+B, C would have a "correlation" near to 0 and should be deleted.
-
Residual Network?, add a transfer function to the activation function to make it time dependent
-
Max-Norm regularisation,. ||w||2 <= r
-
Training with Kalman filter
-
A DNN can be trained with unlabeled data, it is call unsupervised pertaining.
-
To integrate easily keras, use stochastic gradient descent (SGD) together with eager execution.
-
Nonsaturating activation functions.
-
Gradient Clipping.
Others:
- Implement reusage of networks for drivers etc. Normally the lower layers are the important.
- OPC UA client to get inputs from industry
This software is under GPL-3.0 license.