[CPU] Do not set lowering_config if they can't share common configs.

[hanhanW]

No description provided.

[github-actions]

Abbreviated Benchmark Summary

@ commit 7a98e96aafb73df1084d972e812a179be52a6d1a (vs. base 708cf77cdc5510616065355772ef797c8730da6c)

Regressed Latencies 🚩

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
EfficientNetV2SPT(linalg) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][experimental-flags,data-tiling,ukernel] local\_task(embedded\_elf)[8-thread,full-inference,default-flags] with zeros @ c2-standard-16[cpu]	52.477 (vs. 46.784, 12.17%↑)	53.168	3.547

Improved Latencies 🎉

Benchmark Name	Average Latency (ms)	Median Latency (ms)	Latency Standard Deviation (ms)
MobileNetV2\_fp32(tflite) [vmvx-generic-vmvx-vmvx][experimental-flags] local\_task(vmvx\_module)[4-thread,full-inference,system-scheduling] with zeros @ pixel-6-pro[big-core]	4170.617 (vs. 4921.248, 15.25%↓)	4184.884	100.144
EfficientNetV2SPT(linalg) [x86\_64-cascadelake-linux\_gnu-llvm\_cpu][default-flags] local\_task(embedded\_elf)[8-thread,full-inference,default-flags] with zeros @ c2-standard-16[cpu]	42.291 (vs. 47.687, 11.32%↓)	40.612	2.931
MobileNetV3Small\_fp32(tflite) [vmvx-generic-vmvx-vmvx][experimental-flags] local\_task(vmvx\_module)[4-thread,full-inference,system-scheduling] with zeros @ pixel-6-pro[big-core]	890.913 (vs. 987.439, 9.78%↓)	890.177	30.220

[Top 3 out of 5 results showed]

No improved or regressed compilation metrics 🏖️

For more information:

• Full benchmark result tables
• Build

Source Workflow Run

[dcaballe]

Can you add a test for this?

[hanhanW]

Can you add a test for this?

I tried when I created the PR, it is quite hard. I can try to prepare one before my vacation, so we can unblock the issue