Make sure python Inception and scala Inception examples have the same result

[jason-dai]

Especially for the training results.

[jason-dai]

Any update?

[yiheng]

@jenniew Can you provide the example code to reproduce this issue?

[jason-dai]

https://github.com/intel-analytics/BigDL/tree/master/pyspark/bigdl/models/inception

[yiheng]

@wzhongyuan please take a look at this issue

[wzhongyuan]

The python code and related parameters are same as scala ones, finally found the difference caused by preprocessing :) , Python uses OpenCV while Scala still uses legacy ones

 DataSet.SeqFileFolder.files(path, sc, classNumber).transform(
      MTLabeledBGRImgToBatch[ByteRecord](
        width = imageSize,
        height = imageSize,
        batchSize = batchSize,
        transformer = (BytesToBGRImg() -> BGRImgCropper(imageSize, imageSize)
          -> DatasetHFlip(0.5) -> BGRImgNormalizer(0.485, 0.456, 0.406, 0.229, 0.224, 0.225))
      ))

    train_transformer = Pipeline([PixelBytesToMat(),
                                  RandomCrop(image_size, image_size),
                                  RandomTransformer(HFlip(), 0.5),
                                  ChannelNormalize(0.485, 0.456, 0.406, 0.229, 0.224, 0.225),
                                  MatToTensor(to_rgb=True),
                                  ImageFrameToSample(input_keys=["imageTensor"], target_keys=["label"])
                                  ])

Will raise a PR to fix it

[wzhongyuan]

Need to implement a replacement of MTLabeledBGRImgToBatch compatible with OpenCV and then apply new Opencv transformers

[jason-dai]

I think the main issue is that the Top5 accuracy results of these two examples are different (the Scala version has higher accuracy). @jenniew can you provide the testing results?

[jenniew]

scala: top5: 88.2 top1: 68.2
python: top5: 87.4 top1: 67.1

[wzhongyuan]

Seems we cannot refactor scala code for now as python has worse performance... holding the change for now.
I will run a full training to re-produce the number and will try to find if any referenced caffe parameters and run accordingly with the same config to see how it works if caffe has better accuracy

[wzhongyuan]

@jenniew thanks for providing the data

[jenniew]

Python ChannelNormalize has an issue: #2544
Maybe it's related with the accuracy difference.

[jenniew]

caffe also uses opencv to transform, so I think using opencv would not decrease accuracy.

[wzhongyuan]

@jenniew I think that's the issue causing the discrepancy. The training result after the fix is much better now. I will put the fix in, quite simple change.

[wzhongyuan]

Will refactor the scala transformers as well

[jason-dai]

I believe caffe also use opencv based preprocessing? maybe we can just compared to caffe.

[wzhongyuan]

We mirrored intel caffe (please refer to #2555), but did not get compatible result, the final accuracy is below than what we've done with legacy transformers.

[wzhongyuan]

Closing this issue as it's fixed

[jason-dai]

Reopen - cannot reproduce the results

[wzhongyuan]

Probably caused by regression from commit ce7e573 , this commit is made after the convergence testing. we'll check and get back.