All inferences have the same predicted value

[wangjiawen2013]

Hi,
I perfromed the simple regression example and got a model. Then I infer the test data and find that when using batch data (, all the predicted values were the same.

pub fn infer<B: Backend>(artifact_dir: &str, device: B::Device, dataset_infer: DiabetesDataset) {
    let config = ExpConfig::load(format!("{artifact_dir}/config.json"))
        .expect("Config should exist for the model; run train first");
    let record: RegressionModelRecord<B> = NoStdTrainingRecorder::new()
        .load(format!("{artifact_dir}/model").into(), &device)
        .expect("Trained model should exist; run train first");
    let model = RegressionModelConfig::new(config.input_feature_len).init(&device).load_record(record);
    
    let batcher = DiabetesBatcher::new(device);
    // Infer all items simultaneously
    let items: Vec<DiabetesItem> = dataset_infer.iter().collect();
    let batch = batcher.batch(items);
    let outputs = model.forward(batch.inputs);
    let targets = batch.targets;
    let predicted = outputs;

    println!("Predicted {} Expected {}", predicted, targets);

here is the output:

Then I infer the items one by one. This time each item gave a different value. How this happened ?

[laggui]

In my tests the batch predictions don't give me the same results for all items in the batch, though I think there is still something wrong with the regression example.

I never realized that the normalization happens based on items in the batch. This might lead to issues, we should normalized based on some precomputed statistics instead.

[wangjiawen2013]

This happend when disabling the min_max_norm function. I disabled the normalization because the training results displayed NaN if I normalize the data. But when disabling the min_max_norm function, all the inference had the same value, as I said before.

Then I enable min_max_norm again, this time the infered values were different. However, the model didn't perform well because the infered value and the real value differed a lot. So I think we should use good dataset and model for an example.

But what did make the predicted values the same when disabling normalizaton ?

[wangjiawen2013]

In my tests the batch predictions don't give me the same results for all items in the batch, though I think there is still something wrong with the regression example.

I never realized that the normalization happens based on items in the batch. This might lead to issues, we should normalized based on some precomputed statistics instead.

I also find this problem.

[laggui]

This happend when disabling the min_max_norm function. I disabled the normalization because the training results displayed NaN if I normalize the data. But when disabling the min_max_norm function, all the inference had the same value, as I said before.

Then I enable min_max_norm again, this time the infered values were different. However, the model didn't perform well because the infered value and the real value differed a lot. So I think we should use good dataset and model for an example.

But what did make the predicted values the same when disabling normalizaton ?

Huh, that's interesting to say the least. My guess is that without normalization the input values are out of the expected range from the trained model (possibly by a lot), so the model degenerates to the same response.

So I think we should use good dataset and model for an example.

I actually updated the example to use a more representative dataset in #2405 with correct normalization. In my tests the model always had a MSE of ~0.55-0.6 and the predicted values were pretty close to the target (you can see an almost linear relationship in the predicted vs expected scatter plot). Let me know what you think :)

[wangjiawen2013]

@laggui
Hi,
I tested simple-regression according to the latest example. I shuffled the dataset and took 60% dataset as train data and 20% dataset as valid data. here is the result. Is it similar to your result ?

[laggui]

The example already has a train/valid/test split, but actually with your shuffling it seems you get an easier validation set because I had around ~0.59 MSE.

The results look pretty similar (though a bit better in your case) 🙂