Profile-Guided Optimization (PGO) benchmark results

[zamazan4ik]

Hi!

Recently I tested Profile-Guided Optimization (PGO) compiler optimization on different projects in different software domains - all the results are available at https://github.com/zamazan4ik/awesome-pgo . Since PGO shows measurable improvements in many cases, I decided to perform PGO benchmarks on zune-image. Here are my results.

Test environment

• Fedora 39
• Linux kernel 6.8.7
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.78
• zune-image version: the latest for now from the dev branch on commit 26df051313bb88ab89073fa1d8a6a0546a3b6a26
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench result I got with taskset -c 0 cargo bench command. The PGO training phase is done with taskset -c 0 cargo pgo bench, PGO optimization phase - with taskset -c 0 cargo pgo optimize bench. taskset -c 0 is used for reducing the OS scheduler influence on the results.

All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/155357b6c92fea0138e05c77c3afef07
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/04d5f78ac0c538f6b5eb6a6061bf678c
• (just for reference) PGO instrumentation compared to Release: https://gist.github.com/zamazan4ik/c45c1517f260c76e69d7c25b37f0d55a

According to the results, PGO measurably improves the library's performance in many cases.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks with other datasets. If it shows improvements - add a note to the documentation (README file?) about possible improvements in the library's performance with PGO.
• Probably you can try to get some insights about how the code can be optimized further based on the changes that compiler performed with PGO.

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

I would be happy to answer your questions about PGO.

[etemesi254]

Hi, thank you for taking time to do the benchmarks and reporting them.

That's some impressive perf gains, I'm impressed.

Perform more PGO benchmarks with other datasets. If it shows improvements - add a note to the documentation (README file?) about possible improvements in the library's performance with PGO

This is good and would love to see the results of this on the speed

Probably you can try to get some insights about how the code can be optimized further based on the changes that compiler performed with PGO.
If PGO shows gains across a diverse set of images, this would be good, but question is how do you compare a before and after? Do you put the whole project in compiler explorer or?

[zamazan4ik]

If PGO shows gains across a diverse set of images, this would be good, but question is how do you compare a before and after? Do you put the whole project in compiler explorer or?

The comparison above is done via cargo bench built-in comparison logic. It's possible to use it over a bigger set of images for more robust benchmarking.

If we are talking about "performance insights" - I didn't do that. I think you can try to gather some performance profiles before and after PGO and compare them to get some insights. Another thing - run a disassembler on the code before and after PGO to understand the difference in the assembly level.

Unfortunately, performing PGO optimization directly on the Compiler Explorer side is not possible for now: compiler-explorer/compiler-explorer#5414 . If you think that having PGO support is a useful thing to have - please vote somehow on this issue.