Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stop intalling all the .NET SDK versions in benchmark agents #6612

Merged
merged 1 commit into from
Jan 31, 2025
Merged

Stop intalling all the .NET SDK versions in benchmark agents #6612

merged 1 commit into from
Jan 31, 2025

Conversation

andrewlock
Copy link
Member

Summary of changes

Only install the SDK version specified in the global.json

Reason for change

Currently the benchmark agents are installing every patch release of every TFM we ever test against. That means they slowly run out of space. We're only (currently) running benchmarks against EOL TFMs anyway, so there aren't new versions of these to test against anyway, so this is purely just a maintenance burden.

Implementation details

Use install-latest-dotnet-sdk.yml instead of install-dotnet-sdks.yml

Test coverage

Meh, I'm sure it'll be fine.

Other details

Yeah "latest" is not a great description for the task but ah well 🙈

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 31, 2025
@andrewlock andrewlock requested a review from a team as a code owner January 31, 2025 09:18
@andrewlock andrewlock changed the title Stop intalling _all_ the .NET SDK versions in benchmark agents Stop intalling all the .NET SDK versions in benchmark agents Jan 31, 2025
gleocadie
gleocadie approved these changes Jan 31, 2025
View reviewed changes
Copy link
Collaborator

@gleocadie gleocadie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks a lot :)

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jan 31, 2025 •
edited
Loading

Datadog Report

Branch report: andrew/ci/stop-benchmarks-filling-up
Commit report: af4ddff
Test service: dd-trace-dotnet

❌ 2 Failed (0 Known Flaky), 242124 Passed, 2061 Skipped, 19h 9m 2.34s Total Time

❌ Failed Tests (2)

  • Baseline - Samples.HttpMessageHandler.windows.net80.json.scenarios - Details

    Expand for error 
     
 <null>rd Output: tnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.0&arch=x64&rid=win-x64&os=win10

  • CallTarget+Inlining+NGEN - Samples.HttpMessageHandler.windows.net80.json.scenarios - Details

    Expand for error 
     
 <null>rd Output: tnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.0&arch=x64&rid=win-x64&os=win10

@andrewlock
Copy link
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #6612 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.133
  • 1 benchmarks are slower, with geometric mean 1.189
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.06μs 46.3ns 355ns 0.00852 0.00426 0 5.6 KB
master StartStopWithChild netcoreapp3.1 10.2μs 56.3ns 328ns 0.02 0.01 0 5.79 KB
master StartStopWithChild net472 16μs 42.1ns 163ns 1.05 0.333 0.103 6.21 KB
#6612 StartStopWithChild net6.0 7.94μs 39ns 161ns 0.0187 0.0075 0 5.61 KB
#6612 StartStopWithChild netcoreapp3.1 10.1μs 52.7ns 279ns 0.0197 0.00985 0 5.81 KB
#6612 StartStopWithChild net472 16.1μs 44.5ns 172ns 1.03 0.304 0.0959 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 517μs 436ns 1.57μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 653μs 203ns 731ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 852μs 720ns 2.6μs 0.425 0 0 3.3 KB
#6612 WriteAndFlushEnrichedTraces net6.0 484μs 202ns 784ns 0 0 0 2.7 KB
#6612 WriteAndFlushEnrichedTraces netcoreapp3.1 681μs 538ns 2.01μs 0 0 0 2.7 KB
#6612 WriteAndFlushEnrichedTraces net472 845μs 611ns 2.29μs 0.422 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 128μs 577ns 2.24μs 0.194 0 0 14.47 KB
master SendRequest netcoreapp3.1 143μs 248ns 896ns 0.216 0 0 17.27 KB
master SendRequest net472 0.00198ns 0.0014ns 0.00506ns 0 0 0 0 b
#6612 SendRequest net6.0 129μs 598ns 2.39μs 0.194 0 0 14.47 KB
#6612 SendRequest netcoreapp3.1 146μs 185ns 692ns 0.219 0 0 17.27 KB
#6612 SendRequest net472 0.0111ns 0.00392ns 0.0152ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 547μs 2.07μs 7.46μs 0.556 0 0 41.59 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 676μs 3.54μs 18.4μs 0.359 0 0 41.79 KB
master WriteAndFlushEnrichedTraces net472 855μs 4.06μs 15.7μs 8.28 2.48 0.414 53.29 KB
#6612 WriteAndFlushEnrichedTraces net6.0 563μs 2.69μs 10.4μs 0.558 0 0 41.63 KB
#6612 WriteAndFlushEnrichedTraces netcoreapp3.1 670μs 3.6μs 20.3μs 0.324 0 0 41.99 KB
#6612 WriteAndFlushEnrichedTraces net472 838μs 4.37μs 21.9μs 8.39 2.52 0.419 53.33 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.3μs 1.04ns 4.03ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.71μs 0.993ns 3.85ns 0.0137 0 0 1.02 KB
master ExecuteNonQuery net472 2.16μs 2.92ns 11.3ns 0.156 0.00107 0 987 B
#6612 ExecuteNonQuery net6.0 1.29μs 1ns 3.75ns 0.0142 0 0 1.02 KB
#6612 ExecuteNonQuery netcoreapp3.1 1.74μs 0.856ns 3.31ns 0.013 0 0 1.02 KB
#6612 ExecuteNonQuery net472 2.09μs 1.8ns 6.96ns 0.156 0.00104 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6612

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.133 1,299.02 1,146.98

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.3μs 0.614ns 2.3ns 0.0137 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.5μs 0.968ns 3.75ns 0.0128 0 0 976 B
master CallElasticsearch net472 2.53μs 1.18ns 4.41ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.29μs 0.82ns 2.96ns 0.0129 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.6μs 0.834ns 3.12ns 0.0136 0 0 1.02 KB
master CallElasticsearchAsync net472 2.71μs 1.11ns 4.16ns 0.166 0 0 1.05 KB
#6612 CallElasticsearch net6.0 1.15μs 0.868ns 3.25ns 0.0138 0 0 976 B
#6612 CallElasticsearch netcoreapp3.1 1.57μs 1.04ns 3.9ns 0.0133 0 0 976 B
#6612 CallElasticsearch net472 2.55μs 1.15ns 4.47ns 0.158 0 0 995 B
#6612 CallElasticsearchAsync net6.0 1.24μs 0.42ns 1.57ns 0.013 0 0 952 B
#6612 CallElasticsearchAsync netcoreapp3.1 1.69μs 1.59ns 5.95ns 0.0136 0 0 1.02 KB
#6612 CallElasticsearchAsync net472 2.62μs 0.953ns 3.57ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.28μs 0.464ns 1.74ns 0.0133 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.65μs 1.13ns 4.38ns 0.013 0 0 952 B
master ExecuteAsync net472 1.88μs 0.642ns 2.49ns 0.145 0 0 915 B
#6612 ExecuteAsync net6.0 1.3μs 0.651ns 2.35ns 0.0137 0 0 952 B
#6612 ExecuteAsync netcoreapp3.1 1.64μs 0.647ns 2.33ns 0.0131 0 0 952 B
#6612 ExecuteAsync net472 1.81μs 0.455ns 1.76ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.45μs 1.6ns 6.2ns 0.0311 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.25μs 3.75ns 13ns 0.0384 0 0 2.85 KB
master SendAsync net472 7.42μs 1.81ns 7.02ns 0.493 0 0 3.12 KB
#6612 SendAsync net6.0 4.47μs 1.81ns 6.79ns 0.0314 0 0 2.31 KB
#6612 SendAsync netcoreapp3.1 5.29μs 2.01ns 7.23ns 0.0371 0 0 2.85 KB
#6612 SendAsync net472 7.41μs 1.44ns 5.59ns 0.496 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.53μs 0.66ns 2.47ns 0.0229 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.26μs 2.07ns 7.48ns 0.0225 0 0 1.64 KB
master EnrichedLog net472 2.64μs 1.6ns 6.2ns 0.249 0 0 1.57 KB
#6612 EnrichedLog net6.0 1.45μs 1.02ns 3.93ns 0.0233 0 0 1.64 KB
#6612 EnrichedLog netcoreapp3.1 2.31μs 0.925ns 3.46ns 0.0218 0 0 1.64 KB
#6612 EnrichedLog net472 2.61μs 1.3ns 5.02ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 116μs 113ns 437ns 0.058 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 120μs 257ns 994ns 0.0603 0 0 4.28 KB
master EnrichedLog net472 150μs 211ns 819ns 0.674 0.225 0 4.46 KB
#6612 EnrichedLog net6.0 113μs 160ns 618ns 0.0564 0 0 4.28 KB
#6612 EnrichedLog netcoreapp3.1 115μs 134ns 519ns 0.0576 0 0 4.28 KB
#6612 EnrichedLog net472 150μs 136ns 527ns 0.677 0.226 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.01μs 1.42ns 5.49ns 0.0302 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.43μs 1.5ns 5.62ns 0.0289 0 0 2.2 KB
master EnrichedLog net472 4.86μs 1.3ns 5.02ns 0.319 0 0 2.02 KB
#6612 EnrichedLog net6.0 3.06μs 0.645ns 2.41ns 0.0306 0 0 2.2 KB
#6612 EnrichedLog netcoreapp3.1 4.13μs 0.654ns 2.45ns 0.0289 0 0 2.2 KB
#6612 EnrichedLog net472 4.88μs 0.974ns 3.65ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.33μs 0.963ns 3.73ns 0.0161 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.69μs 0.587ns 2.27ns 0.0152 0 0 1.14 KB
master SendReceive net472 2.03μs 0.608ns 2.27ns 0.183 0 0 1.16 KB
#6612 SendReceive net6.0 1.36μs 0.551ns 2.13ns 0.0163 0 0 1.14 KB
#6612 SendReceive netcoreapp3.1 1.8μs 0.909ns 3.52ns 0.0153 0 0 1.14 KB
#6612 SendReceive net472 2.14μs 1.36ns 5.25ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.9μs 1.35ns 5.07ns 0.0216 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.89μs 1.76ns 6.59ns 0.0215 0 0 1.65 KB
master EnrichedLog net472 4.49μs 1.74ns 6.52ns 0.323 0 0 2.04 KB
#6612 EnrichedLog net6.0 2.78μs 1.29ns 4.99ns 0.0222 0 0 1.6 KB
#6612 EnrichedLog netcoreapp3.1 3.87μs 9.84ns 38.1ns 0.021 0 0 1.65 KB
#6612 EnrichedLog net472 4.38μs 2.46ns 9.51ns 0.324 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6612

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.189 481.51 572.61

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 402ns 0.695ns 2.6ns 0.00802 0 0 576 B
master StartFinishSpan netcoreapp3.1 557ns 0.836ns 3.24ns 0.00777 0 0 576 B
master StartFinishSpan net472 662ns 1.28ns 4.95ns 0.0915 0 0 578 B
master StartFinishScope net6.0 482ns 0.804ns 3.11ns 0.00985 0 0 696 B
master StartFinishScope netcoreapp3.1 716ns 1.08ns 4.19ns 0.00961 0 0 696 B
master StartFinishScope net472 836ns 2.03ns 7.87ns 0.105 0 0 658 B
#6612 StartFinishSpan net6.0 408ns 0.21ns 0.813ns 0.00816 0 0 576 B
#6612 StartFinishSpan netcoreapp3.1 557ns 0.628ns 2.43ns 0.00782 0 0 576 B
#6612 StartFinishSpan net472 605ns 0.422ns 1.63ns 0.0916 0 0 578 B
#6612 StartFinishScope net6.0 573ns 0.193ns 0.749ns 0.00974 0 0 696 B
#6612 StartFinishScope netcoreapp3.1 719ns 0.31ns 1.2ns 0.00937 0 0 696 B
#6612 StartFinishScope net472 852ns 0.986ns 3.82ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 660ns 1.17ns 4.54ns 0.00963 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 893ns 1.34ns 5.2ns 0.00952 0 0 696 B
master RunOnMethodBegin net472 1.03μs 2.37ns 9.17ns 0.104 0 0 658 B
#6612 RunOnMethodBegin net6.0 719ns 0.381ns 1.38ns 0.00969 0 0 696 B
#6612 RunOnMethodBegin netcoreapp3.1 900ns 0.679ns 2.63ns 0.00937 0 0 696 B
#6612 RunOnMethodBegin net472 1.12μs 0.566ns 2.19ns 0.105 0 0 658 B

@andrewlock andrewlock merged commit 6ab8186 into master Jan 31, 2025
93 of 97 checks passed
@andrewlock andrewlock deleted the andrew/ci/stop-benchmarks-filling-up branch January 31, 2025 12:15
@github-actions github-actions bot added this to the vNext-v3 milestone Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gleocadie gleocadie gleocadie approved these changes

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.10.0
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @gleocadie