[tensorflow-pytorch-aarch64--r22.09] TensorFlow docker images are broken

[snadampal]

Issue Description
The tensorflow docker images built from r22.09 tag with onednn/acl as well as those available on docker hub (https://hub.docker.com/r/armswdev/tensorflow-arm-neoverse:r22.09-tf-2.10.0-onednn-acl or latest) are producing incorrect results for MLPerf resnet50 model.

the last working tag was: tensorflow-pytorch-aarch64--r22.08
TF2.10 official wheel works fine, so the issue is with one of the staging patches maintained on top of TF 2.10.
https://github.com/ARM-software/Tool-Solutions/tree/main/docker/tensorflow-aarch64/patches

How to reproduce
docker pull armswdev/tensorflow-arm-neoverse

follow this section to run MLPerf resnet50 inference with "--accuracy" option.
https://github.com/ARM-software/Tool-Solutions/blob/main/docker/tensorflow-aarch64/examples/README.md#mlcommons-tm-benchmarks

[nSircombe]

Thanks for the report @snadampal, we'll look into it.
Our builds include a few accuracy tests, on the Python examples, on the CPP examples, and on the MLCommons examples - however, the MLCommons accuracy test employs --count=1 expecting 100% accuracy.

Could you provide more details of the failure? Run lines, logs etc. and environment (what platform, is 'fast maths' enabled, etc.)

Many thanks.

[snadampal]

Hi @nSircombe , the issue was observed only for fast math enabled with bf16 kernels.

export DNNL_DEFAULT_FPMATH_MODE=BF16
run commands:
MLCommons/inference/vision/classification_detection: ./run_local.sh tf resnet50 cpu --accuracy

Failing scenario log: (r22.09 tag)
TestScenario.SingleStream qps=68.64, mean=0.0144, time=7.284, acc=2.600%, queries=500, tiles=50.0:0.0144,80.0:0.0145,90.0:0.0146,95.0:0.0147,99.0:0.0148,99.9:0.0151

Passing scenario logs: (r22.08 tag)
TestScenario.SingleStream qps=66.96, mean=0.0148, time=7.467, acc=76.200%, queries=500, tiles=50.0:0.0147,80.0:0.0149,90.0:0.0150,95.0:0.0151,99.0:0.0153,99.9:0.0159