Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent results between BenchmarkSwitcher and BenchmarkRunner #1665

Closed
ProIcons opened this issue Mar 8, 2021 · 2 comments
Closed

Inconsistent results between BenchmarkSwitcher and BenchmarkRunner #1665

ProIcons opened this issue Mar 8, 2021 · 2 comments

Comments

@ProIcons
Copy link

ProIcons commented Mar 8, 2021

I have this benchmark class.

[MemoryDiagnoser]
//[SimpleJob(launchCount: 3, warmupCount: 10, targetCount: 30)]
[MinColumn, MaxColumn, MeanColumn, MedianColumn]
public class Benchmarks
{
    [Benchmark]
    public byte[] ListToArray()
    {
        byte[] send =
        {
            0x9D, 0x00, 0x00, 0x00, 0x99, 0x00, 0x00, 0x00, 0x03, 0x6D, 0x30, 0x00, 0x88, 0x00, 0x00, 0x00, 0x02, 0x49, 0x44, 0x00, 0x04, 0x00, 0x00, 0x00, 0x47, 0x50, 0x64, 0x00, 0x02, 0x43,
            0x6F, 0x49, 0x44, 0x00, 0x2F, 0x00, 0x00, 0x00
        };
        byte[] send1 = {0x00, 0x02, 0x54, 0x6B, 0x00, 0x2D, 0x00, 0x00, 0x00};
        byte[] send2 = {0x00, 0x10, 0x63, 0x67, 0x79, 0x00, 0x6D, 0x03, 0x00, 0x00, 0x00, 0x10, 0x6D, 0x63, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00};

        List<byte> bytes = new List<byte>(send1.Length + send.Length + send2.Length);
        bytes.AddRange(send);
        bytes.AddRange(send1);
        bytes.AddRange(send2);

        return bytes.ToArray();
    }

    [Benchmark]
    public byte[] ArrayWithCopyTo()
    {
        byte[] send =
        {
            0x9D, 0x00, 0x00, 0x00, 0x99, 0x00, 0x00, 0x00, 0x03, 0x6D, 0x30, 0x00, 0x88, 0x00, 0x00, 0x00, 0x02, 0x49, 0x44, 0x00, 0x04, 0x00, 0x00, 0x00, 0x47, 0x50, 0x64, 0x00, 0x02, 0x43,
            0x6F, 0x49, 0x44, 0x00, 0x2F, 0x00, 0x00, 0x00
        };
        byte[] send1 = {0x00, 0x02, 0x54, 0x6B, 0x00, 0x2D, 0x00, 0x00, 0x00};
        byte[] send2 = {0x00, 0x10, 0x63, 0x67, 0x79, 0x00, 0x6D, 0x03, 0x00, 0x00, 0x00, 0x10, 0x6D, 0x63, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00};

        byte[] mainsend = new byte[send.Length + send1.Length + send2.Length];


        send.CopyTo(mainsend, 0);
        send1.CopyTo(mainsend, send.Length);
        send2.CopyTo(mainsend, send1.Length);

        return mainsend;
    }

    [Benchmark]
    public byte[] BufferWithBlockCopy()
    {
        byte[] send =
        {
            0x9D, 0x00, 0x00, 0x00, 0x99, 0x00, 0x00, 0x00, 0x03, 0x6D, 0x30, 0x00, 0x88, 0x00, 0x00, 0x00, 0x02, 0x49, 0x44, 0x00, 0x04, 0x00, 0x00, 0x00, 0x47, 0x50, 0x64, 0x00, 0x02, 0x43,
            0x6F, 0x49, 0x44, 0x00, 0x2F, 0x00, 0x00, 0x00
        };
        byte[] send1 = {0x00, 0x02, 0x54, 0x6B, 0x00, 0x2D, 0x00, 0x00, 0x00};
        byte[] send2 = {0x00, 0x10, 0x63, 0x67, 0x79, 0x00, 0x6D, 0x03, 0x00, 0x00, 0x00, 0x10, 0x6D, 0x63, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00};

        byte[] mainsend = new byte[send.Length + send1.Length + send2.Length];

        Buffer.BlockCopy(send, 0, mainsend, 0, send.Length);
        Buffer.BlockCopy(send1, 0, mainsend, send.Length, send1.Length);
        Buffer.BlockCopy(send2, 0, mainsend, send.Length + send1.Length, send2.Length);

        return mainsend;
    }
}

And i get 2 different results if i run it with BenchmarkRunner vs BenchmarkSwitcher.

var summary = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
.NET Core SDK=5.0.103
  [Host]     : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT
  DefaultJob : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT

|              Method |     Mean |    Error |   StdDev |      Min |      Max |   Median |  Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------------- |---------:|---------:|---------:|---------:|---------:|---------:|-------:|------:|------:|----------:|
|         ListToArray | 79.40 ns | 0.878 ns | 0.821 ns | 78.25 ns | 80.84 ns | 79.18 ns | 0.0598 |     - |     - |     376 B |
|     ArrayWithCopyTo | 37.63 ns | 0.460 ns | 0.407 ns | 37.09 ns | 38.34 ns | 37.65 ns | 0.0395 |     - |     - |     248 B |
| BufferWithBlockCopy | 32.44 ns | 0.651 ns | 0.577 ns | 31.72 ns | 33.35 ns | 32.18 ns | 0.0395 |     - |     - |     248 B |
BenchmarkDotNet.Reports.Summary[] summary = BenchmarkRunner.Run(typeof(Program).Assembly);
.NET Core SDK=5.0.103
  [Host]     : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT
  DefaultJob : .NET Core 5.0.3 (CoreCLR 5.0.321.7212, CoreFX 5.0.321.7212), X64 RyuJIT

|              Method |      Mean |    Error |   StdDev |       Min |       Max |    Median |  Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------------- |----------:|---------:|---------:|----------:|----------:|----------:|-------:|------:|------:|----------:|
|         ListToArray | 102.94 ns | 1.195 ns | 1.118 ns | 101.29 ns | 104.97 ns | 102.53 ns | 0.0598 |     - |     - |     376 B |
|     ArrayWithCopyTo |  53.23 ns | 1.073 ns | 1.192 ns |  51.76 ns |  55.78 ns |  53.06 ns | 0.0395 |     - |     - |     248 B |
| BufferWithBlockCopy |  46.87 ns | 0.849 ns | 0.753 ns |  46.04 ns |  48.82 ns |  46.74 ns | 0.0395 |     - |     - |     248 B |

Any idea why might that happening?

@adamsitnik
Copy link
Member

Hi @ProIcons

Any idea why might that happening?

There are many reasons why it might be happening. Some of them:

  • code|memory alignment changes
  • various heuristics (GC, Tiered JIT etc)
  • OS (other busy processes taking over the resources, CPU scheduling)
  • hardware (disk, CPU cache)

To find out why, you need to use a profiler, capture two trace files and compare them. You can do that by using PerfView. If this won't give you the answer you might need to use VTune or uProf to get the disassembly.

Most likely in this particular case, it's memory alignment. Here you can see how it affects copying one array into another: #1587 (comment). Here you can read more on this subject: dotnet/performance#1602

Since there is nothing actionable from the BDN perspective (BenchmarkSwitcher and BenchmarkRunner use the same code to run benchmarks), I am going to close the issue

@ProIcons
Copy link
Author

ProIcons commented Mar 9, 2021

Well i mean yeah it seemed odd to me. But i ran these over 5 tjmes each and i got the same results. The same deviations i don't know. It might nit actually matter but i was a bit curious

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants