Skip to content

Commit

Permalink
[QNN EP] Increase tolerance for ReduceProd test on x64 Windows (#17078)
Browse files Browse the repository at this point in the history
### Description
Slightly increases the allowable error tolerance for ReduceProd tests on
x64 Windows/Linux with the QNN CPU backend.


### Motivation and Context
A recent [PR](#16916)
updated the input range for ReduceProd tests, which uncovered an
inaccuracy for ReduceProd on x64 Windows/Linux with the QNN CPU backend.
This PR updates the allowable error tolerance and adds a TODO for
investigation.

This is needed to ensure the QNN_Nuget_Windows pipeline runs
successfully.
  • Loading branch information
adrianlizarraga authored Aug 9, 2023
1 parent 4bc2287 commit d793e23
Showing 1 changed file with 18 additions and 4 deletions.
22 changes: 18 additions & 4 deletions onnxruntime/test/providers/qnn/reduce_op_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ static void RunReduceOpCpuTest(const std::string& op_type,
const std::vector<int64_t>& axes,
bool keepdims,
int opset,
ExpectedEPNodeAssignment expected_ep_assignment) {
ExpectedEPNodeAssignment expected_ep_assignment,
float fp32_abs_err = 1e-5f) {
ProviderOptions provider_options;
#if defined(_WIN32)
provider_options["backend_path"] = "QnnCpu.dll";
Expand All @@ -92,7 +93,8 @@ static void RunReduceOpCpuTest(const std::string& op_type,
false), // noop_with_empty_axes
provider_options,
opset,
expected_ep_assignment);
expected_ep_assignment,
fp32_abs_err);
}

//
Expand Down Expand Up @@ -166,21 +168,33 @@ TEST_F(QnnCPUBackendTests, ReduceSumOpset11_Float) {
// - Uses opset 18, which has "axes" as an input.
TEST_F(QnnCPUBackendTests, ReduceProdOpset18) {
RunReduceOpCpuTest<float>("ReduceProd",
TestInputDef<float>({2, 2}, false, -10.0f, 10.0f),
TestInputDef<float>({2, 2}, false, {-10.0f, -8.2f, 0.0f, 10.0f}),
std::vector<int64_t>{0, 1},
true, // keepdims
18,
ExpectedEPNodeAssignment::All);
}

// TODO: Investigate slight inaccuracy. x64 Windows/Linux require a slightly larger error tolerance greater than 1.5e-5f.
// LOG: ... the value pair (208.881729, 208.881744) at index #0 don't match, which is 1.52588e-05 from 208.882
TEST_F(QnnCPUBackendTests, ReduceProdOpset18_SlightlyInaccurate_WindowsLinuxX64) {
RunReduceOpCpuTest<float>("ReduceProd",
TestInputDef<float>({2, 2}, false, {3.21289f, -5.9981f, -1.72799f, 6.27263f}),
std::vector<int64_t>{0, 1},
true, // keepdims
18,
ExpectedEPNodeAssignment::All,
2e-5f); // x64 Linux & Windows require larger tolerance.
}

// Test creates a graph with a ReduceProd node, and checks that all
// nodes are supported by the QNN EP (cpu backend), and that the inference results match the CPU EP results.
//
// - The input and output data type is float.
// - Uses opset 13, which has "axes" as an attribute.
TEST_F(QnnCPUBackendTests, ReduceProdOpset13) {
RunReduceOpCpuTest<float>("ReduceProd",
TestInputDef<float>({2, 2}, false, -10.0f, 10.0f),
TestInputDef<float>({2, 2}, false, {-10.0f, -8.2f, 0.0f, 10.0f}),
std::vector<int64_t>{0, 1},
true, // keepdims
13,
Expand Down

0 comments on commit d793e23

Please sign in to comment.