Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run load and startup benchmarks in different jobs #6920

Merged
merged 1 commit into from
Apr 16, 2024
Merged

Run load and startup benchmarks in different jobs #6920

merged 1 commit into from
Apr 16, 2024

Conversation

manuel-alvarez-alvarez
Copy link
Member

@manuel-alvarez-alvarez manuel-alvarez-alvarez commented Apr 16, 2024 •
edited
Loading

What Does This Do

Separates load and startup benchmarks in different jobs.

Motivation

The main reason for the separation is to improve the time it takes to run the benchmaks

Additional Notes

Jira ticket: [PROJ-IDENT]

@manuel-alvarez-alvarez manuel-alvarez-alvarez added the comp: tooling Build & Tooling label Apr 16, 2024
@pr-commenter
Copy link

pr-commenter bot commented Apr 16, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master malvarez/separate-benchmarks-jobs
git_commit_date 1713248465 1713257235
git_commit_sha 4c97fc1 e179e7d
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~e179e7d539
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713259814 1713259814
ci_job_id 487515903 487515903
ci_pipeline_id 32173618 32173618
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 14 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1073811
Total [baseline] (8.545 s) : 0, 8544861
Agent [candidate] (1.083 s) : 0, 1082750
Total [candidate] (8.569 s) : 0, 8569126
section iast
Agent [baseline] (1.197 s) : 0, 1197411
Total [baseline] (9.032 s) : 0, 9031741
Agent [candidate] (1.196 s) : 0, 1196036
Total [candidate] (8.981 s) : 0, 8980607
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.206 s) : 0, 1206060
Total [baseline] (9.024 s) : 0, 9024002
Agent [candidate] (1.204 s) : 0, 1204457
Total [candidate] (9.012 s) : 0, 9011673
section iast_TELEMETRY_OFF
Agent [baseline] (1.192 s) : 0, 1192032
Total [baseline] (8.985 s) : 0, 8985418
Agent [candidate] (1.196 s) : 0, 1195582
Total [candidate] (8.977 s) : 0, 8977075
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent iast 1.197 s 123.6 ms (11.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.206 s 132.25 ms (12.3%)
Agent iast_TELEMETRY_OFF 1.192 s 118.222 ms (11.0%)
Total tracing 8.545 s -
Total iast 9.032 s 486.88 ms (5.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.024 s 479.141 ms (5.6%)
Total iast_TELEMETRY_OFF 8.985 s 440.557 ms (5.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.083 s -
Agent iast 1.196 s 113.286 ms (10.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.204 s 121.707 ms (11.2%)
Agent iast_TELEMETRY_OFF 1.196 s 112.832 ms (10.4%)
Total tracing 8.569 s -
Total iast 8.981 s 411.481 ms (4.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.012 s 442.547 ms (5.2%)
Total iast_TELEMETRY_OFF 8.977 s 407.949 ms (4.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (672.541 ms) : 0, 672541
BytebuddyAgent [candidate] (678.172 ms) : 0, 678172
GlobalTracer [baseline] (308.829 ms) : 0, 308829
GlobalTracer [candidate] (311.593 ms) : 0, 311593
AppSec [baseline] (49.773 ms) : 0, 49773
AppSec [candidate] (49.988 ms) : 0, 49988
Remote Config [baseline] (671.608 µs) : 0, 672
Remote Config [candidate] (671.131 µs) : 0, 671
Telemetry [baseline] (7.626 ms) : 0, 7626
Telemetry [candidate] (7.671 ms) : 0, 7671
section iast
BytebuddyAgent [baseline] (793.587 ms) : 0, 793587
BytebuddyAgent [candidate] (792.522 ms) : 0, 792522
GlobalTracer [baseline] (287.926 ms) : 0, 287926
GlobalTracer [candidate] (287.413 ms) : 0, 287413
AppSec [baseline] (52.311 ms) : 0, 52311
AppSec [candidate] (50.306 ms) : 0, 50306
IAST [baseline] (21.937 ms) : 0, 21937
IAST [candidate] (22.788 ms) : 0, 22788
Remote Config [baseline] (571.827 µs) : 0, 572
Remote Config [candidate] (593.811 µs) : 0, 594
Telemetry [baseline] (6.591 ms) : 0, 6591
Telemetry [candidate] (8.097 ms) : 0, 8097
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (799.867 ms) : 0, 799867
BytebuddyAgent [candidate] (797.821 ms) : 0, 797821
GlobalTracer [baseline] (289.482 ms) : 0, 289482
GlobalTracer [candidate] (289.434 ms) : 0, 289434
AppSec [baseline] (52.61 ms) : 0, 52610
AppSec [candidate] (49.111 ms) : 0, 49111
IAST [baseline] (22.19 ms) : 0, 22190
IAST [candidate] (24.042 ms) : 0, 24042
Remote Config [baseline] (595.428 µs) : 0, 595
Remote Config [candidate] (585.235 µs) : 0, 585
Telemetry [baseline] (6.669 ms) : 0, 6669
Telemetry [candidate] (8.891 ms) : 0, 8891
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (789.892 ms) : 0, 789892
BytebuddyAgent [candidate] (791.964 ms) : 0, 791964
GlobalTracer [baseline] (286.713 ms) : 0, 286713
GlobalTracer [candidate] (287.649 ms) : 0, 287649
AppSec [baseline] (50.745 ms) : 0, 50745
AppSec [candidate] (48.627 ms) : 0, 48627
IAST [baseline] (21.644 ms) : 0, 21644
IAST [candidate] (24.986 ms) : 0, 24986
Remote Config [baseline] (589.916 µs) : 0, 590
Remote Config [candidate] (580.688 µs) : 0, 581
Telemetry [baseline] (8.128 ms) : 0, 8128
Telemetry [candidate] (7.376 ms) : 0, 7376
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.075 s) : 0, 1074698
Total [baseline] (10.365 s) : 0, 10364574
Agent [candidate] (1.088 s) : 0, 1088128
Total [candidate] (10.356 s) : 0, 10355682
section appsec
Agent [baseline] (1.196 s) : 0, 1195850
Total [baseline] (10.462 s) : 0, 10461945
Agent [candidate] (1.205 s) : 0, 1204665
Total [candidate] (10.48 s) : 0, 10479814
section iast
Agent [baseline] (1.205 s) : 0, 1205071
Total [baseline] (10.779 s) : 0, 10779085
Agent [candidate] (1.203 s) : 0, 1202780
Total [candidate] (10.718 s) : 0, 10717689
section profiling
Agent [baseline] (1.267 s) : 0, 1267018
Total [baseline] (10.564 s) : 0, 10563521
Agent [candidate] (1.267 s) : 0, 1267407
Total [candidate] (10.598 s) : 0, 10598428
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent appsec 1.196 s 121.152 ms (11.3%)
Agent iast 1.205 s 130.373 ms (12.1%)
Agent profiling 1.267 s 192.32 ms (17.9%)
Total tracing 10.365 s -
Total appsec 10.462 s 97.372 ms (0.9%)
Total iast 10.779 s 414.512 ms (4.0%)
Total profiling 10.564 s 198.947 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.088 s -
Agent appsec 1.205 s 116.537 ms (10.7%)
Agent iast 1.203 s 114.652 ms (10.5%)
Agent profiling 1.267 s 179.279 ms (16.5%)
Total tracing 10.356 s -
Total appsec 10.48 s 124.132 ms (1.2%)
Total iast 10.718 s 362.007 ms (3.5%)
Total profiling 10.598 s 242.746 ms (2.3%) 
gantt
    title petclinic - break down per module: candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.026 ms) : 0, 673026
BytebuddyAgent [candidate] (682.784 ms) : 0, 682784
GlobalTracer [baseline] (309.519 ms) : 0, 309519
GlobalTracer [candidate] (312.043 ms) : 0, 312043
AppSec [baseline] (49.559 ms) : 0, 49559
AppSec [candidate] (50.056 ms) : 0, 50056
Remote Config [baseline] (657.761 µs) : 0, 658
Remote Config [candidate] (671.318 µs) : 0, 671
Telemetry [baseline] (7.585 ms) : 0, 7585
Telemetry [candidate] (7.63 ms) : 0, 7630
section appsec
BytebuddyAgent [baseline] (693.887 ms) : 0, 693887
BytebuddyAgent [candidate] (699.377 ms) : 0, 699377
GlobalTracer [baseline] (290.388 ms) : 0, 290388
GlobalTracer [candidate] (292.733 ms) : 0, 292733
AppSec [baseline] (149.74 ms) : 0, 149740
AppSec [candidate] (150.587 ms) : 0, 150587
IAST [baseline] (18.768 ms) : 0, 18768
IAST [candidate] (19.048 ms) : 0, 19048
Remote Config [baseline] (601.18 µs) : 0, 601
Remote Config [candidate] (611.968 µs) : 0, 612
Telemetry [baseline] (8.169 ms) : 0, 8169
Telemetry [candidate] (7.564 ms) : 0, 7564
section iast
BytebuddyAgent [baseline] (798.607 ms) : 0, 798607
BytebuddyAgent [candidate] (798.201 ms) : 0, 798201
GlobalTracer [baseline] (289.369 ms) : 0, 289369
GlobalTracer [candidate] (288.847 ms) : 0, 288847
AppSec [baseline] (49.751 ms) : 0, 49751
AppSec [candidate] (51.787 ms) : 0, 51787
IAST [baseline] (24.033 ms) : 0, 24033
IAST [candidate] (22.157 ms) : 0, 22157
Remote Config [baseline] (571.927 µs) : 0, 572
Remote Config [candidate] (587.978 µs) : 0, 588
Telemetry [baseline] (8.142 ms) : 0, 8142
Telemetry [candidate] (6.646 ms) : 0, 6646
section profiling
BytebuddyAgent [baseline] (677.561 ms) : 0, 677561
BytebuddyAgent [candidate] (677.281 ms) : 0, 677281
GlobalTracer [baseline] (379.166 ms) : 0, 379166
GlobalTracer [candidate] (379.797 ms) : 0, 379797
AppSec [baseline] (50.111 ms) : 0, 50111
AppSec [candidate] (49.957 ms) : 0, 49957
Remote Config [baseline] (723.064 µs) : 0, 723
Remote Config [candidate] (721.382 µs) : 0, 721
Telemetry [baseline] (7.441 ms) : 0, 7441
Telemetry [candidate] (7.466 ms) : 0, 7466
ProfilingAgent [baseline] (95.712 ms) : 0, 95712
ProfilingAgent [candidate] (95.768 ms) : 0, 95768
Profiling [baseline] (95.735 ms) : 0, 95735
Profiling [candidate] (95.792 ms) : 0, 95792
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-16T09:00:41 2024-04-16T09:07:30
git_branch master malvarez/separate-benchmarks-jobs
git_commit_date 1713248465 1713257235
git_commit_sha 4c97fc1 e179e7d
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~e179e7d539
start_time 2024-04-16T09:00:27 2024-04-16T09:07:16
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713258793 1713258793
ci_job_id 487515904 487515904
ci_pipeline_id 32173618 32173618
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 16 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling better
[-85.950µs; -32.009µs] or [-5.563%; -2.072%]		 unstable
[-323.244op/s; +797.318op/s] or [-10.909%; +26.909%]		 1.486ms 3200.000op/s 1.545ms 2962.963op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.331 ms) : 1311, 1350
.   : milestone, 1331,
appsec (1.726 ms) : 1702, 1750
.   : milestone, 1726,
appsec_no_iast (1.73 ms) : 1707, 1754
.   : milestone, 1730,
iast (1.483 ms) : 1460, 1505
.   : milestone, 1483,
profiling (1.545 ms) : 1519, 1571
.   : milestone, 1545,
tracing (1.466 ms) : 1441, 1491
.   : milestone, 1466,
section candidate
no_agent (1.346 ms) : 1327, 1365
.   : milestone, 1346,
appsec (1.71 ms) : 1686, 1734
.   : milestone, 1710,
appsec_no_iast (1.725 ms) : 1700, 1750
.   : milestone, 1725,
iast (1.499 ms) : 1476, 1521
.   : milestone, 1499,
profiling (1.486 ms) : 1462, 1510
.   : milestone, 1486,
tracing (1.479 ms) : 1455, 1503
.   : milestone, 1479,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.311 ms, 1.35 ms] -
appsec 1.726 ms [1.702 ms, 1.75 ms] 395.031 µs (29.7%)
appsec_no_iast 1.73 ms [1.707 ms, 1.754 ms] 399.51 µs (30.0%)
iast 1.483 ms [1.46 ms, 1.505 ms] 152.039 µs (11.4%)
profiling 1.545 ms [1.519 ms, 1.571 ms] 214.369 µs (16.1%)
tracing 1.466 ms [1.441 ms, 1.491 ms] 135.71 µs (10.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.346 ms [1.327 ms, 1.365 ms] -
appsec 1.71 ms [1.686 ms, 1.734 ms] 364.145 µs (27.1%)
appsec_no_iast 1.725 ms [1.7 ms, 1.75 ms] 379.485 µs (28.2%)
iast 1.499 ms [1.476 ms, 1.521 ms] 152.736 µs (11.3%)
profiling 1.486 ms [1.462 ms, 1.51 ms] 140.191 µs (10.4%)
tracing 1.479 ms [1.455 ms, 1.503 ms] 133.011 µs (9.9%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~e179e7d539, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.213 µs) : 345, 386
.   : milestone, 365,
iast (479.889 µs) : 458, 502
.   : milestone, 480,
iast_FULL (537.588 µs) : 516, 559
.   : milestone, 538,
iast_GLOBAL (506.361 µs) : 484, 528
.   : milestone, 506,
iast_HARDCODED_SECRET_DISABLED (477.084 µs) : 455, 499
.   : milestone, 477,
iast_INACTIVE (441.825 µs) : 421, 462
.   : milestone, 442,
iast_TELEMETRY_OFF (472.423 µs) : 451, 494
.   : milestone, 472,
tracing (449.3 µs) : 429, 470
.   : milestone, 449,
section candidate
no_agent (367.431 µs) : 347, 388
.   : milestone, 367,
iast (475.736 µs) : 455, 497
.   : milestone, 476,
iast_FULL (537.79 µs) : 517, 559
.   : milestone, 538,
iast_GLOBAL (494.221 µs) : 473, 516
.   : milestone, 494,
iast_HARDCODED_SECRET_DISABLED (468.477 µs) : 447, 490
.   : milestone, 468,
iast_INACTIVE (447.348 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (469.845 µs) : 449, 490
.   : milestone, 470,
tracing (442.573 µs) : 422, 463
.   : milestone, 443,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.213 µs [344.743 µs, 385.684 µs] -
iast 479.889 µs [457.834 µs, 501.943 µs] 114.675 µs (31.4%)
iast_FULL 537.588 µs [516.204 µs, 558.973 µs] 172.375 µs (47.2%)
iast_GLOBAL 506.361 µs [484.41 µs, 528.312 µs] 141.148 µs (38.6%)
iast_HARDCODED_SECRET_DISABLED 477.084 µs [455.312 µs, 498.856 µs] 111.871 µs (30.6%)
iast_INACTIVE 441.825 µs [421.353 µs, 462.297 µs] 76.611 µs (21.0%)
iast_TELEMETRY_OFF 472.423 µs [451.343 µs, 493.502 µs] 107.209 µs (29.4%)
tracing 449.3 µs [428.638 µs, 469.963 µs] 84.087 µs (23.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.431 µs [346.815 µs, 388.046 µs] -
iast 475.736 µs [454.698 µs, 496.773 µs] 108.305 µs (29.5%)
iast_FULL 537.79 µs [516.969 µs, 558.611 µs] 170.359 µs (46.4%)
iast_GLOBAL 494.221 µs [472.879 µs, 515.563 µs] 126.79 µs (34.5%)
iast_HARDCODED_SECRET_DISABLED 468.477 µs [446.917 µs, 490.036 µs] 101.046 µs (27.5%)
iast_INACTIVE 447.348 µs [426.352 µs, 468.343 µs] 79.917 µs (21.8%)
iast_TELEMETRY_OFF 469.845 µs [449.259 µs, 490.431 µs] 102.414 µs (27.9%)
tracing 442.573 µs [422.058 µs, 463.088 µs] 75.142 µs (20.5%)

@manuel-alvarez-alvarez manuel-alvarez-alvarez marked this pull request as ready for review April 16, 2024 09:34
@manuel-alvarez-alvarez manuel-alvarez-alvarez requested a review from a team as a code owner April 16, 2024 09:34
smola
smola approved these changes Apr 16, 2024
View reviewed changes
.gitlab/benchmarks.yml Outdated
- ./steps/analyze-results.sh
- ./steps/upload-results-to-s3.sh
- ./steps/post-pr-comment.sh
- git clone --branch malvarez/dd-trace-java-separate-tracer-benchmarks https://github.com/DataDog/benchmarking-platform.git /platform && cd /platform
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure you fix the branch name before merging.

@manuel-alvarez-alvarez manuel-alvarez-alvarez force-pushed the malvarez/separate-benchmarks-jobs branch from e179e7d to 6c6188e Compare April 16, 2024 10:52
@manuel-alvarez-alvarez manuel-alvarez-alvarez merged commit d9850dd into master Apr 16, 2024
77 of 78 checks passed
@manuel-alvarez-alvarez manuel-alvarez-alvarez deleted the malvarez/separate-benchmarks-jobs branch April 16, 2024 11:33
@github-actions github-actions bot added this to the 1.33.0 milestone Apr 16, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog added the tag: no release notes Changes to exclude from release notes label Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smola smola smola approved these changes

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer PerfectSlayer is a code owner automatically assigned from DataDog/apm-java

@jandro996 jandro996 Awaiting requested review from jandro996

Assignees
No one assigned
Labels
comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.33.0
Development

Successfully merging this pull request may close these issues.
3 participants
@manuel-alvarez-alvarez @smola @nikita-tkachenko-datadog