Increases benchmark warm-up and decreases the threshold value

[cheqianh]

This PR
(1) Addressed a memory leak issue - link
(2) Experimented the GHA runners' resource limitation and switched to MacOS for 1k iterations and warmups - link
(3) Added a method for Ion binary/text conversion - link
(4) Fixed a comparison result mismatch issue by sorting the combination list - link

The threshold is set to 0.2 for now, we will work on its enhancement in the future. The latest performance detection failed due to the variance.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[tgregg]

A threshold of 0.5 still allows a 50% performance regression, right? That's still not going to be low enough for our needs. How did you select 150 for warmups/iterations? Are you seeing diminishing returns in terms of reduced variance as you increase the warmups/iterations, or does the regression detection workflow just start taking too long?

[cheqianh]

If 0.5 doesn't satisfy our needs, we need to consider other improvements as well.

There is an error when running 200+ warm-ups - actions/runner-images#6680.

The Github runner only allocate fixed resource (e.g., memory) to jobs. Running over 200 times make the job cancel automatically. Additionally the results of relative-difference are usually < 0.3 but sometimes one sample file (real_worlds_data_1) fail with 0.5+ perf regressions (and if the workflow fails, it's always this file).

One possible solution to enable more iteration and warmups is using a larger runner - https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners

[cheqianh]

Found a small memory leak in each simpleion.dumps()/loads() call, which might be the root to cause the workflow exceeds the resource limit. Investigating the root.

[cheqianh]

Confirmed that execution_with_command(['read', file, '--api', 'load_dump', '--api', 'streaming']) has memory leak issues for simpleion load API when reading testStructs.10n. Didn't see any memory leak for dump API.

[cheqianh]

Fixed the memory leak for read and it worked - 9858b44

but still failed for large execution number. Investigating.

[popematt]

In general, we should only use MacOS runners if we need to test specifically that something works correctly on MacOS or we're distributing a MacOS-specific binary.

What's the rationale for running on MacOS here?

[cheqianh]

I'm trying to run more warm-ups and iterations for the benchmark-cli command in order to generate a consistent threshold value to help us identify the performance regression. However, command with 150+ executions will hit the resource limitation of the GHA linux runner.

[popematt]

Which resource limit? Is it causing the job to timeout?

[cheqianh]

One of the workflows run into this issue - actions/runner-images#6680

[cheqianh]

I found another issue that causes the workflow to generate a wrong threshold value. After fixing that issue, the threshold value seems to be more stable. But ideally we will eventually benchmark performance on all popular platforms.

[cheqianh]

This PR
(1) Addressed a memory leak issue - link
(2) Experimented the GHA runners' resource limitation and switched to MacOS for 1k iterations and warmups - link
(3) Added a method for Ion binary/text conversion - link
(4) Fixed a comparison result mismatch issue by sorting the combination list - link

The threshold is set to 0.2 for now, we will work on its enhancement in the future. An example performance detection failed due to variance.

[cheqianh]

I changed the target commit to the main branch so we don't have to create a new PR for that. In addition, I

• Opened an issue for avoiding file copy - ion-python-benchmark-cli's format conversion shouldn't always copy the input file #269
• Updated the GH issue for identifying the threshold values by results percentile - ion-python-benchmark-cli's threshold value need to be more accurate. #265