Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

have two main actions: with unroll and without, for no unroll icrease the step by 1 in pilot (not *2) #771

Merged
merged 2 commits into from
Jun 5, 2018

Conversation

adamsitnik
Copy link
Member

I am currently working on porting CoreCLR to BenchmarkDotNet.
CoreCLR contains a lot of "benchmark game" benchmarks. These are long-running benchmarks.

After #760 the time it takes to run the benchmarks has greatly improved (ML.NET benchmarks from 20 to 2 minutes). However, I have found two extra scenarios for improvement:

  1. The time to execute benchmark is slighly less than IterationTime. Let's say it's 0.4s when the IterationTime is 0.5s. In that case we invoke it 16 times per iteration (default unroll factor value). It's too much, we should invoke it just once.
  2. The time to execute benchmark is just few times less than IterationTime. Let's say it's 0.1s when the IterationTime is 0.5s. In that case we invoke it 16 times per iteration (default unroll factor value). It's too much, we should invoke it few times to fit within IterationTime .

My changes are quite simple:

  1. If the time to execute benchmark is slighly less than IterationTime, run it once per iteration. No Pilot needed.
  2. If the time to execute benchmark is just few times less than IterationTime, start the Pilot with MinInvokeCount = 2 (we know 1 is not enough), and increment it with every pilot iteration (not double, it's typically too much for these benchmarks).

Example:

[Benchmark]
public void Sleep100() => Thread.Sleep(100);
IdleJitting  1: 1 op, 803135.91 ns, 803.1359 us/op
MainJitting  1: 1 op, 100786001.20 ns, 100.7860 ms/op

Pilot        1: 2 op, 200567013.38 ns, 100.2835 ms/op
Pilot        2: 3 op, 301861923.92 ns, 100.6206 ms/op
Pilot        3: 4 op, 401721624.60 ns, 100.4304 ms/op
Pilot        4: 5 op, 503124019.07 ns, 100.6248 ms/op

MainWarmup   1: 5 op, 503016250.03 ns, 100.6033 ms/op

…ccording to IterationTime and first invocation time
@@ -67,7 +68,10 @@ private long RunAuto()
if (invokeCount >= MaxInvokeCount)
break;

invokeCount *= 2;
if (unrollFactor == 1 && invokeCount < EnvResolver.DefaultUnrollFactorForThroughput)

This comment was marked as spam.

This comment was marked as spam.

Copy link
Member

@AndreyAkinshin AndreyAkinshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adamsitnik adamsitnik merged commit 3c3b47f into master Jun 5, 2018
@adamsitnik adamsitnik deleted the useIterationTime branch June 5, 2018 11:31
@adamsitnik adamsitnik added this to the v0.11.0 milestone Jul 18, 2018
alinasmirnova pushed a commit to alinasmirnova/BenchmarkDotNet that referenced this pull request Sep 22, 2018
… the step by 1 in pilot (not *2) (dotnet#771)

* have two main actions: with unroll and without, choose the best one according to IterationTime and first invocation time

* require to run the pilot, don't assume that benchmark is stable by doing simple math based on first invocation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants