Bug: overflow-error in Windows with 64-bit-fannfloat.dll

[Jens01]

I had a overflow-error with the 64bit fannfloat.dll.
rprop_increase_factor and rprop_decrease_factor were similar to 0
Fixed it with this issue from ache7: #85

fann_train.c

void fann_update_weights_irpropm(struct fann *ann, unsigned int first_weight, unsigned int past_end)
{
	fann_type *train_slopes = ann->train_slopes;
	fann_type *weights = ann->weights;
	fann_type *prev_steps = ann->prev_steps;
	fann_type *prev_train_slopes = ann->prev_train_slopes;

	double prev_step, next_step, slope, prev_slope, same_sign;

	float increase_factor = ann->rprop_increase_factor;	/*1.2; */
	float decrease_factor = ann->rprop_decrease_factor;	/*0.5; */
	float delta_min = ann->rprop_delta_min;	/*0.0; */
	float delta_max = ann->rprop_delta_max;	/*50.0; */

	unsigned int i = first_weight;

	for(; i != past_end; i++)
	{
 
		prev_step = fann_max(prev_steps[i], (fann_type) 0.0001);	/* prev_step may not be zero because then the training will stop */
    
    
    
		slope = train_slopes[i];
		prev_slope = prev_train_slopes[i];
    
     
		same_sign = prev_slope * slope;

// https://github.com/libfann/fann/issues/85
		next_step = prev_step;
		if (same_sign > 0)
			next_step = fann_min(prev_step * increase_factor, delta_max);
		else if (same_sign < 0) {
			next_step = fann_max(prev_step * decrease_factor, delta_min);
			slope = 0;
		}

		if (slope < 0) {
			weights[i] -= next_step;
			if (weights[i] < -1500) weights[i] = -1500;
		}
		else if (slope > 0) {
			weights[i] += next_step;
			if (weights[i] > 1500) weights[i] = 1500;
		}

    /*
		if(same_sign >= 0.0)
			next_step = fann_min(prev_step * increase_factor, delta_max);
		else
		{
			next_step = fann_max(prev_step * decrease_factor, delta_min);
			slope = 0;
		}
   
     
    
		if(slope < 0)
		{
			weights[i] -= next_step;
			if(weights[i] < -1500)
				weights[i] = -1500;
		}
		else
		{
			weights[i] += next_step;
			if(weights[i] > 1500)
				weights[i] = 1500;
		}
	*/
     
		/*if(i == 2){
		 * printf("weight=%f, slope=%f, next_step=%f, prev_step=%f\n", weights[i], slope, next_step, prev_step);
		 * } */

		/* update global data arrays */
		prev_steps[i] = next_step;
		prev_train_slopes[i] = slope;
		train_slopes[i] = 0.0;

	}
}

[andersfylling]

@troiganto did you update the code when you closed the previous issue?

[andersfylling]

Thanks for the issue btw. Could you also provide the code to replicate the issue? We would be happy if we could add it as a unit test for future changes/upgrades ^^,

[Jens01]

https://github.com/Jens01/NewFann/blob/cd8cce135b06cbd3d77d38fab10db5edfb2a70ae/FannSource/src/fann_train.c#L722
This is my project : https://github.com/Jens01/NewFann

The error comes during training (not always) when the desired error was very small 0.000001.
After the exception rprop_increase_factor and rprop_decrease_factor were similar to 0.

I tested it with the example parity8, cascade and then train in Delphi 64-bit compiler:

  try
    CTrain := TCascadeclass.Create(Fann);
    try
      CTrain.epochs_between_reports := 50;
      Fann.DefNeutronLayer(1, 0.75, FANN_SIGMOID_SYMMETRIC_STEPWISE);

      CTrain.desired_error          := 0.000001;
      CTrain.FannCascadeEvent       := TrainCasEvent;
      CTrain.training_algorithm     := FANN_TRAIN_RPROP;
      CTrain.train_error_function   := FANN_ERRORFUNC_LINEAR;
      CTrain.bit_fail_limit         := 0.5;
      CTrain.train_stop_function    := FANN_STOPFUNC_MSE;
      AcFunctions                   := [FANN_SIGMOID_STEPWISE];
      Steepness                     := [0.5, 1];
      CTrain.activation_functions   := AcFunctions;
      CTrain.activation_steepnesses := Steepness;

      if (CTrain.training_algorithm <> FANN_TRAIN_QUICKPROP) then
      begin
        CTrain.learning_rate    := 0.7;
        CTrain.WeigthsMax       := 5;
        CTrain.WeigthsMin       := -5;
        CTrain.UseRandomWeigths := False;
      end;
      CTrain.TrainCascadeFromFannFile(FNParityTrain);
    finally
      CTrain.Free;
    end;

    for i := 0 to Fann.LayerCount - 1 do
      mmoEvent.Lines.Add(Format('LayerIndx %d : Neurons : %d (%d)', [i, Fann.NeuronCount[i], Fann.NeuronandBiasCount[i]]));

    FannStream.Clear;
    Fann.SaveToStream(FannStream);
    FillNeuronsToList;
  finally
    Fann.Free;
  end;

P.S: Sometimes Github is not my friend...

[troiganto]

@andersfylling I have not. I mis-interpreted it as a misunderstanding about which variant of RProp FANN has implemented. However, I compared the FANN implementation with the original description of iRProp– and there is indeed a difference. I'll see to writing a PR tomorrow.

Writing a unit test about this might, however, be a bit difficult. If I'm understanding Jens01 correctly, they encountered the issue when running a Delphi version of cascade_train.c. I've tried this myself, but couldn't manage to reproduce it. It might be caused by differences in the parameter values, but I haven't been able to narrow it down so far.

[Jens01]

when running a Delphi version of cascade_train.c

First I used cascade after that I trained it. And during the training come the exception with a small desired error and 64bit.
I found the error-position with try except in the C++ code.