Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks and compare to other libraries #2

Closed
gfoidl opened this issue May 17, 2021 · 7 comments · Fixed by #10
Closed

Add benchmarks and compare to other libraries #2

gfoidl opened this issue May 17, 2021 · 7 comments · Fixed by #10
Assignees

Comments

@gfoidl
Copy link
Contributor

gfoidl commented May 17, 2021

We should a add benchmark project and also run a comparison with other libraries.

Maybe we can run the benchmarks as part of CI (once public we get "unlimited" time)?
(But TBH I don't know if this is worth it).

@gfoidl gfoidl self-assigned this May 17, 2021
@gfoidl
Copy link
Contributor Author

gfoidl commented May 17, 2021

WTF??? I just started on adding some benchmarks, then

// Validating benchmarks:
Assembly MyCSharp.HttpUserAgentParser.Benchmarks which defines benchmarks references non-optimized DeviceDetector.NET
        If you own this dependency, please, build it in RELEASE.
        If you don't, you can disable this policy by using 'config.With(ConfigOptions.DisableOptimizationsValidator)'.

They seriously publish a debug-build of DeviceDetector.NET.
grafik

--> totpero/DeviceDetector.NET#44 (well if the project is stale...but to be fair).

@gfoidl
Copy link
Contributor Author

gfoidl commented May 17, 2021

Did a first comparison (and re-checked the numbers, as it's impressive even for the non-cached usage 🚀):

|             Method | Categories |         Data |            Mean |           Error |          StdDev |          Median |     Ratio |  RatioSD |     Gen 0 |    Gen 1 |    Gen 2 |  Allocated |
|------------------- |----------- |------------- |----------------:|----------------:|----------------:|----------------:|----------:|---------:|----------:|---------:|---------:|-----------:|
|           MyCSharp |      Basic | Chrome Win10 |     47,462.7 ns |       941.89 ns |     1,986.76 ns |     47,093.3 ns |      1.00 |     0.00 |    0.1221 |        - |        - |      432 B |
|           UAParser |      Basic | Chrome Win10 | 37,696,940.5 ns |   749,889.55 ns | 1,371,215.45 ns | 37,386,471.4 ns |    792.13 |    41.17 | 1785.7143 | 785.7143 | 214.2857 | 11231570 B |
| DeviceDetector.NET |      Basic | Chrome Win10 | 39,702,615.3 ns | 1,498,312.53 ns | 4,370,647.97 ns | 38,021,100.0 ns |    857.95 |    95.42 | 2000.0000 |        - |        - |  9081624 B |
|                    |            |              |                 |                 |                 |                 |           |          |           |          |          |            |
|           MyCSharp |     Cached | Chrome Win10 |        145.4 ns |         2.49 ns |         3.05 ns |        145.6 ns |      1.00 |     0.00 |         - |        - |        - |          - |
|           UAParser |     Cached | Chrome Win10 |  1,525,259.0 ns |   148,158.46 ns |   436,848.58 ns |  1,454,550.0 ns | 12,200.55 | 2,523.15 |         - |        - |        - |    37824 B |
|                    |            |              |                 |                 |                 |                 |           |          |           |          |          |            |
|           MyCSharp |      Basic |   Google-Bot |      4,877.1 ns |        97.09 ns |       122.79 ns |      4,842.1 ns |      1.00 |     0.00 |         - |        - |        - |          - |
|           UAParser |      Basic |   Google-Bot | 38,654,491.5 ns |   952,744.94 ns | 2,733,603.83 ns | 37,739,792.9 ns |  8,102.49 |   690.32 | 1642.8571 | 714.2857 | 214.2857 | 10720175 B |
| DeviceDetector.NET |      Basic |   Google-Bot |  2,534,810.6 ns |   211,591.74 ns |   603,683.14 ns |  2,400,050.0 ns |    517.98 |   108.59 |         - |        - |        - |  1058504 B |
|                    |            |              |                 |                 |                 |                 |           |          |           |          |          |            |
|           MyCSharp |     Cached |   Google-Bot |        108.1 ns |         2.15 ns |         2.87 ns |        107.4 ns |      1.00 |     0.00 |         - |        - |        - |          - |
|           UAParser |     Cached |   Google-Bot |    898,076.5 ns |    48,038.92 ns |   129,876.01 ns |    851,500.0 ns |  8,278.14 |   828.82 |         - |        - |        - |    45192 B |

Benchmark-code: ee2368a
(will do a PR after the init-change is in)

@BenjaminAbt
Copy link
Member

I didn't look into the issues when I started the benchmarks :-(

@gfoidl
Copy link
Contributor Author

gfoidl commented May 17, 2021

No problem. I submit a PR with the benchs for the comparison, then I hope we both are happy 😃

@gfoidl gfoidl mentioned this issue May 17, 2021
@gfoidl
Copy link
Contributor Author

gfoidl commented May 17, 2021

Should we add these numbers from above to the ReadMe or link from the ReadMe to here?

Due the comments it should be separate. Maybe check the results file in and link from the ReadMe?

@BenjaminAbt
Copy link
Member

Yes, we should add them to the readme

@BenjaminAbt
Copy link
Member

.NET 6 Benchmark

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1348 (21H1/May2021Update)
AMD Ryzen 9 5950X, 1 CPU, 32 logical and 16 physical cores
.NET SDK=6.0.100
  [Host]   : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT
  ShortRun : .NET 6.0.0 (6.0.21.52210), X64 RyuJIT

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3  

|             Method | Categories |         Data |             Mean |             Error |         StdDev |    Ratio | RatioSD |     Gen 0 |     Gen 1 |    Gen 2 |    Allocated |
|------------------- |----------- |------------- |-----------------:|------------------:|---------------:|---------:|--------:|----------:|----------:|---------:|-------------:|
|           MyCSharp |      Basic | Chrome Win10 |     25,791.72 ns |      1,468.128 ns |      80.473 ns |     1.00 |    0.00 |         - |         - |        - |        432 B |
|           UAParser |      Basic | Chrome Win10 | 14,845,743.23 ns |  2,080,079.003 ns | 114,016.149 ns |   575.61 |    6.21 |  656.2500 |  343.7500 | 156.2500 | 11,449,360 B |
| DeviceDetector.NET |      Basic | Chrome Win10 | 45,725,500.00 ns | 10,948,162.399 ns | 600,105.724 ns | 1,772.84 |   17.86 | 2000.0000 | 1000.0000 |        - | 41,508,464 B |
|                    |            |              |                  |                   |                |          |         |           |           |          |              |
|           MyCSharp |     Cached | Chrome Win10 |         79.33 ns |          0.921 ns |       0.050 ns |     1.00 |    0.00 |         - |         - |        - |            - |
|           UAParser |     Cached | Chrome Win10 |    748,833.33 ns |  1,056,427.069 ns |  57,906.332 ns | 9,439.05 |  729.85 |         - |         - |        - |     38,208 B |
|                    |            |              |                  |                   |                |          |         |           |           |          |              |
|           MyCSharp |      Basic |   Google-Bot |      2,399.84 ns |        423.008 ns |      23.186 ns |     1.00 |    0.00 |         - |         - |        - |            - |
|           UAParser |      Basic |   Google-Bot | 14,320,067.71 ns |    213,954.583 ns |  11,727.573 ns | 5,967.46 |   58.88 |  625.0000 |  328.1250 | 140.6250 | 10,946,582 B |
| DeviceDetector.NET |      Basic |   Google-Bot |  3,118,447.79 ns |    100,384.159 ns |   5,502.394 ns | 1,299.52 |   12.94 |  156.2500 |   39.0625 |        - |  2,628,468 B |
|                    |            |              |                  |                   |                |          |         |           |           |          |              |
|           MyCSharp |     Cached |   Google-Bot |         57.21 ns |          5.526 ns |       0.303 ns |     1.00 |    0.00 |         - |         - |        - |            - |
|           UAParser |     Cached |   Google-Bot |    416,898.34 ns |     10,947.707 ns |     600.081 ns | 7,287.14 |   49.12 |    2.4414 |         - |        - |     45,856 B |

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants