Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilized perf results for KMeansAndLogisticRegression benchmark #10

Open
briancylui opened this issue Aug 30, 2018 · 4 comments
Open

Stabilized perf results for KMeansAndLogisticRegression benchmark #10

briancylui opened this issue Aug 30, 2018 · 4 comments
Assignees
@briancylui

Comments

@briancylui
Copy link
Owner 

// * Legends *
  Mean         : Arithmetic mean of all measurements
  Error        : Half of 99.9% confidence interval
  StdDev       : Standard deviation of all measurements
  Extra Metric : Value of the provided extra metric
  Gen 0        : GC Generation 0 collects per 1k Operations
  Gen 1        : GC Generation 1 collects per 1k Operations
  Gen 2        : GC Generation 2 collects per 1k Operations
  Allocated    : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 s          : 1 Second (1 sec)

netcoreapp3.0 with my change:

Iteration 1:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-YPIYKX : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.781 s 0.0976 s 0.1044 s - 16000.0000 8000.0000 3000.0000 47.57 MB

Iteration 2:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Max: 1.90GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-GREASD : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.781 s 0.1050 s 0.1209 s - 16000.0000 8000.0000 3000.0000 47.27 MB

Iteration 3:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-OHSMOJ : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.853 s 0.0926 s 0.1067 s - 16000.0000 8000.0000 3000.0000 47.17 MB

Iteration 4:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-MIAXEM : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 1.797 s 0.1104 s 0.1271 s - 16000.0000 8000.0000 3000.0000 47.27 MB

netcoreapp3.0 without my change:

Iteration 1:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-TIGZKX : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.094 s 0.0440 s 0.0489 s - 17000.0000 8000.0000 3000.0000 47.13 MB

Iteration 2:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Max: 1.90GHz) (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-LMZGNT : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.062 s 0.0828 s 0.0920 s - 17000.0000 8000.0000 3000.0000 47.17 MB

Iteration 3:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-NAKODA : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.139 s 0.0677 s 0.0780 s - 17000.0000 9000.0000 3000.0000 47.12 MB

Iteration 4:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.2.100-refac-20180613-1
  [Host]     : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT
  Job-QVPJCB : .NET Core 3.0.0-preview1-26822-01 (CoreCLR 4.6.26817.05, CoreFX 4.6.26817.04), 64bit RyuJIT

Toolchain=netcoreapp3.0  MaxIterationCount=20  WarmupCount=1
Method Mean Error StdDev Extra Metric Gen 0 Gen 1 Gen 2 Allocated
TrainKMeansAndLR 2.079 s 0.0778 s 0.0896 s - 17000.0000 9000.0000 3000.0000 47.36 MB
@briancylui
Copy link
Owner Author

@danmosemsft @eerhardt @safern @tannergooding: In netcoreapp3.0, perf results with my change are at least 10% better than those without my change. Will present these.

@briancylui briancylui self-assigned this Aug 30, 2018
@eerhardt
Copy link
Collaborator

This is awesome results, @briancylui! I'm excited to see the presentation tomorrow.

@safern
Copy link

safern commented Aug 31, 2018 •
edited
Loading

This is great!! I love the fact that ML.NET can be used in different architectures, by building on one of them and getting perf benefits!

@helloguo
Copy link

@briancylui can you share how to build and run ML.NET and Benchmark KMeansAndLogisticRegression with .NET Core 3.0 with your change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@briancylui briancylui

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@eerhardt @briancylui @helloguo @safern