ElmanPattern null as activation function in output layer

[Dennis1111]

Hi

Once made a post about the ElmanPattern in the forums since I thaught it was strange that in the output layer null is used as the argument for the activation function.
I see it as a bug but apologize in advance if I'm wrong about moving the issue to here.
When I train the Elman network it produces extreme figures (I guess it depends on the learning task also).
I have added generate2() to the Elman Network which instead uses
network.addLayer(new BasicLayer(this.activation, false, this.outputNeurons));
and also made a small test program to show the differences.

Training Elman network without Sigmoid
Epoch 0 Error = 3.340547587759162
Epoch 1 Error = 12.093709207462359
Epoch 2 Error = 20.10574781179527
Epoch 3 Error = 192.05958976530124
Epoch 4 Error = 584.3769644076101
Epoch 5 Error = 7792.268927035693
Epoch 6 Error = 73795.51179330552
Epoch 7 Error = 686300.045126438
Epoch 8 Error = 3871554.818886862
Epoch 9 Error = 2.3696703827344127E7
Training Elman network with Sigmoid
Epoch 0 Error = 0.24208329326323957
Epoch 1 Error = 0.2252998899657926
Epoch 2 Error = 0.20599926542888564
Epoch 3 Error = 0.1845726317633126
Epoch 4 Error = 0.16181453130188853
Epoch 5 Error = 0.13884928778828928
Epoch 6 Error = 0.11686393694107555
Epoch 7 Error = 0.09679717323086938
Epoch 8 Error = 0.0791784626642417
Epoch 9 Error = 0.06417317172944495

The test code

package org.encog.neural.pattern;

import java.util.Random;

import org.encog.ml.data.basic.*;
import org.encog.neural.pattern.ElmanPattern;
import org.encog.neural.networks.BasicNetwork;
import org.encog.neural.networks.training.propagation.back.Backpropagation;
import org.encog.engine.network.activation.ActivationSigmoid;
import org.encog.util.arrayutil.NormalizeArray;

public class ElmanPatternTest
{

private BasicMLDataSet dataSet;
private BasicNetwork elmanNetwork1;
private BasicNetwork elmanNetwork2;

public ElmanPatternTest()
{
double noiseFactor=100;
int patterns=10000;
this.dataSet = createTestDataSet(patterns,noiseFactor);
int inputSize=1;
int outputSize=1;
int hiddens=10;
createElmanNetworks(inputSize,outputSize,hiddens);
}

public void train(int epochs)
{
System.out.println("Training Elman network without Sigmoid");
train(epochs,elmanNetwork1);
System.out.println("Training Elman network with Sigmoid");
train(epochs,elmanNetwork2);
}

private void train(int epochs,BasicNetwork network)
{
double learnRate = 0.0001;
double momentum= 0.001;
Backpropagation train = new Backpropagation(network,this.dataSet,learnRate,momentum);
for(int epoch=0;epoch<epochs;epoch++)
{
train.iteration();
System.out.println("Epoch "+epoch+" Error = "+train.getError());
}
}

private void createElmanNetworks(int inputSize,int outputSize,int hiddenNeurons)
{
ElmanPattern pattern = new ElmanPattern();
pattern.setActivationFunction(new ActivationSigmoid());
pattern.setInputNeurons(inputSize);
pattern.addHiddenLayer(hiddenNeurons);
pattern.setOutputNeurons(outputSize);
this.elmanNetwork1=(BasicNetwork)pattern.generate();
this.elmanNetwork2=(BasicNetwork)pattern.generate2();
}

private BasicMLDataSet createTestDataSet(int size,double noiseFactor)
{
Random rd = new Random();
BasicMLDataSet samples = new BasicMLDataSet();

double[] inputs = new double[size];
double[] ideals = new double[size];

for(int pattern=0;pattern<size;pattern++)
  {
double noise = noiseFactor*((rd.nextDouble()-0.5)/100);
double x = (2*((double)pattern)/size)-1;//-1..1
inputs[pattern] = x;
ideals[pattern] = Math.sin(x)+noise-(Math.cos(x)/3);    
  }
NormalizeArray norm = new NormalizeArray();

norm.setNormalizedLow(0.05);
norm.setNormalizedHigh(0.95);
ideals = norm.process(ideals); 

double[][] inputDataSet = new double[size][1];
double[][] idealDataSet = new double[size][1];
for(int pattern=0;pattern<size;pattern++)
  {
inputDataSet[pattern][0]=inputs[pattern];
idealDataSet[pattern][0]=ideals[pattern];
  }

BasicMLDataSet dataSet= new BasicMLDataSet(inputDataSet,idealDataSet);
return dataSet;

}

public static void main(String[] args)
{
ElmanPatternTest elmanTest= new ElmanPatternTest();
elmanTest.train(10);
}
}

[jeffheaton]

I will extend the class to support this.