Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add smoke tests for telemetry #7955

Merged
merged 9 commits into from
Nov 21, 2024
Merged

Add smoke tests for telemetry #7955

merged 9 commits into from
Nov 21, 2024

Conversation

smola
Copy link
Member

@smola smola commented Nov 14, 2024 •
edited
Loading

What Does This Do

Add basic smoke tests for telemetry, so far testing:

  • app-started
  • app-dependencies-loaded

Adds a groovy extension to smoke tests, @RunLast, to force these tests to run at the end. This improves testing time, since telemetry tests need to wait for telemetry being sent in the background, so other tests can run meanwhile. Without this, given that these are defined in a parent class, they would run first.

Motivation

We have almost no tests doing integration testing of telemetry dependencies. System-tests do not provide enough coverage here, while our unit tests can miss all kinds of issues.

Additional Notes

Contributor Checklist

Jira ticket: APPSEC-55773 (related)

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes comp: telemetry Telemetry labels Nov 14, 2024
@pr-commenter
Copy link

pr-commenter bot commented Nov 14, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/telemetry-smoke-tests2
git_commit_date 1732122216 1732175565
git_commit_sha df51f63 5f1a938
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~5f1a938f02
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732178037 1732178037
ci_job_id 714936100 714936100
ci_pipeline_id 49573425 49573425
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 55 metrics, 8 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.097 s) : 0, 1096697
Total [baseline] (10.444 s) : 0, 10443682
Agent [candidate] (1.091 s) : 0, 1090546
Total [candidate] (10.531 s) : 0, 10530694
section appsec
Agent [baseline] (1.224 s) : 0, 1224265
Total [baseline] (10.694 s) : 0, 10693696
Agent [candidate] (1.227 s) : 0, 1227465
Total [candidate] (10.752 s) : 0, 10752433
section iast
Agent [baseline] (1.215 s) : 0, 1214648
Total [baseline] (10.946 s) : 0, 10946460
Agent [candidate] (1.214 s) : 0, 1214221
Total [candidate] (10.922 s) : 0, 10922134
section profiling
Agent [baseline] (1.291 s) : 0, 1291400
Total [baseline] (10.799 s) : 0, 10799453
Agent [candidate] (1.302 s) : 0, 1301820
Total [candidate] (10.777 s) : 0, 10777384
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.097 s -
Agent appsec 1.224 s 127.569 ms (11.6%)
Agent iast 1.215 s 117.951 ms (10.8%)
Agent profiling 1.291 s 194.704 ms (17.8%)
Total tracing 10.444 s -
Total appsec 10.694 s 250.014 ms (2.4%)
Total iast 10.946 s 502.778 ms (4.8%)
Total profiling 10.799 s 355.771 ms (3.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.091 s -
Agent appsec 1.227 s 136.919 ms (12.6%)
Agent iast 1.214 s 123.675 ms (11.3%)
Agent profiling 1.302 s 211.274 ms (19.4%)
Total tracing 10.531 s -
Total appsec 10.752 s 221.739 ms (2.1%)
Total iast 10.922 s 391.44 ms (3.7%)
Total profiling 10.777 s 246.69 ms (2.3%) 
gantt
    title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (696.468 ms) : 0, 696468
BytebuddyAgent [candidate] (692.867 ms) : 0, 692867
GlobalTracer [baseline] (318.72 ms) : 0, 318720
GlobalTracer [candidate] (318.064 ms) : 0, 318064
AppSec [baseline] (55.019 ms) : 0, 55019
AppSec [candidate] (55.209 ms) : 0, 55209
Remote Config [baseline] (687.796 µs) : 0, 688
Remote Config [candidate] (686.994 µs) : 0, 687
Telemetry [baseline] (12.033 ms) : 0, 12033
Telemetry [candidate] (10.047 ms) : 0, 10047
section appsec
BytebuddyAgent [baseline] (710.763 ms) : 0, 710763
BytebuddyAgent [candidate] (711.356 ms) : 0, 711356
GlobalTracer [baseline] (314.331 ms) : 0, 314331
GlobalTracer [candidate] (315.1 ms) : 0, 315100
AppSec [baseline] (166.473 ms) : 0, 166473
AppSec [candidate] (168.72 ms) : 0, 168720
IAST [baseline] (20.153 ms) : 0, 20153
IAST [candidate] (19.325 ms) : 0, 19325
Remote Config [baseline] (642.943 µs) : 0, 643
Remote Config [candidate] (645.334 µs) : 0, 645
Telemetry [baseline] (8.301 ms) : 0, 8301
Telemetry [candidate] (8.266 ms) : 0, 8266
section iast
BytebuddyAgent [baseline] (807.963 ms) : 0, 807963
BytebuddyAgent [candidate] (808.177 ms) : 0, 808177
GlobalTracer [baseline] (306.225 ms) : 0, 306225
GlobalTracer [candidate] (305.727 ms) : 0, 305727
AppSec [baseline] (56.381 ms) : 0, 56381
AppSec [candidate] (57.872 ms) : 0, 57872
IAST [baseline] (22.377 ms) : 0, 22377
IAST [candidate] (20.687 ms) : 0, 20687
Remote Config [baseline] (619.904 µs) : 0, 620
Remote Config [candidate] (600.741 µs) : 0, 601
Telemetry [baseline] (7.429 ms) : 0, 7429
Telemetry [candidate] (7.503 ms) : 0, 7503
section profiling
BytebuddyAgent [baseline] (688.566 ms) : 0, 688566
BytebuddyAgent [candidate] (695.466 ms) : 0, 695466
GlobalTracer [baseline] (401.696 ms) : 0, 401696
GlobalTracer [candidate] (403.961 ms) : 0, 403961
AppSec [baseline] (55.705 ms) : 0, 55705
AppSec [candidate] (55.766 ms) : 0, 55766
Remote Config [baseline] (682.332 µs) : 0, 682
Remote Config [candidate] (689.766 µs) : 0, 690
Telemetry [baseline] (10.894 ms) : 0, 10894
Telemetry [candidate] (13.69 ms) : 0, 13690
ProfilingAgent [baseline] (94.816 ms) : 0, 94816
ProfilingAgent [candidate] (92.73 ms) : 0, 92730
Profiling [baseline] (94.842 ms) : 0, 94842
Profiling [candidate] (92.754 ms) : 0, 92754
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.087 s) : 0, 1087429
Total [baseline] (8.683 s) : 0, 8682569
Agent [candidate] (1.089 s) : 0, 1089207
Total [candidate] (8.648 s) : 0, 8647552
section iast
Agent [baseline] (1.215 s) : 0, 1215183
Total [baseline] (9.208 s) : 0, 9207935
Agent [candidate] (1.223 s) : 0, 1223466
Total [candidate] (9.217 s) : 0, 9216961
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.215 s) : 0, 1214811
Total [baseline] (9.178 s) : 0, 9178027
Agent [candidate] (1.224 s) : 0, 1224238
Total [candidate] (9.148 s) : 0, 9147652
section iast_TELEMETRY_OFF
Agent [baseline] (1.22 s) : 0, 1220238
Total [baseline] (9.179 s) : 0, 9179450
Agent [candidate] (1.214 s) : 0, 1213695
Total [candidate] (9.166 s) : 0, 9165516
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.087 s -
Agent iast 1.215 s 127.754 ms (11.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.215 s 127.383 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.22 s 132.809 ms (12.2%)
Total tracing 8.683 s -
Total iast 9.208 s 525.366 ms (6.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.178 s 495.458 ms (5.7%)
Total iast_TELEMETRY_OFF 9.179 s 496.881 ms (5.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.089 s -
Agent iast 1.223 s 134.259 ms (12.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.224 s 135.032 ms (12.4%)
Agent iast_TELEMETRY_OFF 1.214 s 124.488 ms (11.4%)
Total tracing 8.648 s -
Total iast 9.217 s 569.409 ms (6.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.148 s 500.1 ms (5.8%)
Total iast_TELEMETRY_OFF 9.166 s 517.963 ms (6.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (692.17 ms) : 0, 692170
BytebuddyAgent [candidate] (691.558 ms) : 0, 691558
GlobalTracer [baseline] (316.958 ms) : 0, 316958
GlobalTracer [candidate] (316.412 ms) : 0, 316412
AppSec [baseline] (54.602 ms) : 0, 54602
AppSec [candidate] (54.794 ms) : 0, 54794
Remote Config [baseline] (686.808 µs) : 0, 687
Remote Config [candidate] (678.701 µs) : 0, 679
Telemetry [baseline] (9.299 ms) : 0, 9299
Telemetry [candidate] (12.079 ms) : 0, 12079
section iast
BytebuddyAgent [baseline] (808.804 ms) : 0, 808804
BytebuddyAgent [candidate] (813.274 ms) : 0, 813274
GlobalTracer [baseline] (306.025 ms) : 0, 306025
GlobalTracer [candidate] (309.01 ms) : 0, 309010
AppSec [baseline] (57.176 ms) : 0, 57176
AppSec [candidate] (58.379 ms) : 0, 58379
IAST [baseline] (21.405 ms) : 0, 21405
IAST [candidate] (20.924 ms) : 0, 20924
Remote Config [baseline] (620.958 µs) : 0, 621
Remote Config [candidate] (607.411 µs) : 0, 607
Telemetry [baseline] (7.478 ms) : 0, 7478
Telemetry [candidate] (7.525 ms) : 0, 7525
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (808.156 ms) : 0, 808156
BytebuddyAgent [candidate] (814.83 ms) : 0, 814830
GlobalTracer [baseline] (306.769 ms) : 0, 306769
GlobalTracer [candidate] (308.175 ms) : 0, 308175
AppSec [baseline] (57.626 ms) : 0, 57626
AppSec [candidate] (58.194 ms) : 0, 58194
IAST [baseline] (20.541 ms) : 0, 20541
IAST [candidate] (21.005 ms) : 0, 21005
Remote Config [baseline] (618.65 µs) : 0, 619
Remote Config [candidate] (622.833 µs) : 0, 623
Telemetry [baseline] (7.439 ms) : 0, 7439
Telemetry [candidate] (7.584 ms) : 0, 7584
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (811.536 ms) : 0, 811536
BytebuddyAgent [candidate] (807.424 ms) : 0, 807424
GlobalTracer [baseline] (308.17 ms) : 0, 308170
GlobalTracer [candidate] (306.081 ms) : 0, 306081
AppSec [baseline] (58.319 ms) : 0, 58319
AppSec [candidate] (58.071 ms) : 0, 58071
IAST [baseline] (20.376 ms) : 0, 20376
IAST [candidate] (20.37 ms) : 0, 20370
Remote Config [baseline] (613.72 µs) : 0, 614
Remote Config [candidate] (618.466 µs) : 0, 618
Telemetry [baseline] (7.445 ms) : 0, 7445
Telemetry [candidate] (7.369 ms) : 0, 7369
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-11-21T08:04:06 2024-11-21T08:11:03
git_branch master smola/telemetry-smoke-tests2
git_commit_date 1732122216 1732175565
git_commit_sha df51f63 5f1a938
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~5f1a938f02
start_time 2024-11-21T08:03:53 2024-11-21T08:10:49
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732177014 1732177014
ci_job_id 714936101 714936101
ci_pipeline_id 49573425 49573425
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.339 ms) : 1319, 1358
.   : milestone, 1339,
appsec (1.76 ms) : 1735, 1784
.   : milestone, 1760,
appsec_no_iast (1.756 ms) : 1732, 1781
.   : milestone, 1756,
iast (1.505 ms) : 1482, 1528
.   : milestone, 1505,
profiling (1.482 ms) : 1459, 1504
.   : milestone, 1482,
tracing (1.514 ms) : 1490, 1539
.   : milestone, 1514,
section candidate
no_agent (1.352 ms) : 1333, 1371
.   : milestone, 1352,
appsec (1.727 ms) : 1702, 1752
.   : milestone, 1727,
appsec_no_iast (1.755 ms) : 1730, 1779
.   : milestone, 1755,
iast (1.49 ms) : 1468, 1513
.   : milestone, 1490,
profiling (1.527 ms) : 1503, 1551
.   : milestone, 1527,
tracing (1.468 ms) : 1442, 1494
.   : milestone, 1468,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.339 ms [1.319 ms, 1.358 ms] -
appsec 1.76 ms [1.735 ms, 1.784 ms] 420.934 µs (31.4%)
appsec_no_iast 1.756 ms [1.732 ms, 1.781 ms] 417.299 µs (31.2%)
iast 1.505 ms [1.482 ms, 1.528 ms] 166.336 µs (12.4%)
profiling 1.482 ms [1.459 ms, 1.504 ms] 142.798 µs (10.7%)
tracing 1.514 ms [1.49 ms, 1.539 ms] 175.557 µs (13.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.352 ms [1.333 ms, 1.371 ms] -
appsec 1.727 ms [1.702 ms, 1.752 ms] 374.652 µs (27.7%)
appsec_no_iast 1.755 ms [1.73 ms, 1.779 ms] 402.578 µs (29.8%)
iast 1.49 ms [1.468 ms, 1.513 ms] 138.38 µs (10.2%)
profiling 1.527 ms [1.503 ms, 1.551 ms] 174.917 µs (12.9%)
tracing 1.468 ms [1.442 ms, 1.494 ms] 116.168 µs (8.6%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.943 µs) : 351, 391
.   : milestone, 371,
iast (478.937 µs) : 458, 500
.   : milestone, 479,
iast_FULL (642.553 µs) : 621, 664
.   : milestone, 643,
iast_GLOBAL (508.372 µs) : 487, 530
.   : milestone, 508,
iast_HARDCODED_SECRET_DISABLED (486.314 µs) : 465, 508
.   : milestone, 486,
iast_INACTIVE (453.52 µs) : 433, 474
.   : milestone, 454,
iast_TELEMETRY_OFF (471.958 µs) : 451, 493
.   : milestone, 472,
tracing (448.974 µs) : 428, 470
.   : milestone, 449,
section candidate
no_agent (375.205 µs) : 355, 396
.   : milestone, 375,
iast (485.88 µs) : 464, 507
.   : milestone, 486,
iast_FULL (640.589 µs) : 619, 662
.   : milestone, 641,
iast_GLOBAL (517.902 µs) : 496, 540
.   : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (485.416 µs) : 464, 507
.   : milestone, 485,
iast_INACTIVE (453.299 µs) : 432, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (472.521 µs) : 451, 494
.   : milestone, 473,
tracing (441.104 µs) : 420, 462
.   : milestone, 441,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.943 µs [351.081 µs, 390.804 µs] -
iast 478.937 µs [457.649 µs, 500.226 µs] 107.994 µs (29.1%)
iast_FULL 642.553 µs [621.081 µs, 664.025 µs] 271.61 µs (73.2%)
iast_GLOBAL 508.372 µs [487.089 µs, 529.655 µs] 137.429 µs (37.0%)
iast_HARDCODED_SECRET_DISABLED 486.314 µs [464.591 µs, 508.036 µs] 115.371 µs (31.1%)
iast_INACTIVE 453.52 µs [432.617 µs, 474.423 µs] 82.578 µs (22.3%)
iast_TELEMETRY_OFF 471.958 µs [450.658 µs, 493.257 µs] 101.015 µs (27.2%)
tracing 448.974 µs [427.621 µs, 470.326 µs] 78.031 µs (21.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 375.205 µs [354.572 µs, 395.837 µs] -
iast 485.88 µs [464.388 µs, 507.371 µs] 110.675 µs (29.5%)
iast_FULL 640.589 µs [619.098 µs, 662.08 µs] 265.385 µs (70.7%)
iast_GLOBAL 517.902 µs [496.173 µs, 539.631 µs] 142.697 µs (38.0%)
iast_HARDCODED_SECRET_DISABLED 485.416 µs [463.865 µs, 506.968 µs] 110.212 µs (29.4%)
iast_INACTIVE 453.299 µs [432.32 µs, 474.279 µs] 78.095 µs (20.8%)
iast_TELEMETRY_OFF 472.521 µs [450.721 µs, 494.321 µs] 97.316 µs (25.9%)
tracing 441.104 µs [420.325 µs, 461.882 µs] 65.899 µs (17.6%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/telemetry-smoke-tests2
git_commit_date 1732122216 1732175565
git_commit_sha df51f63 5f1a938
release_version 1.43.0-SNAPSHOT~df51f63bab 1.43.0-SNAPSHOT~5f1a938f02
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1732177566 1732177566
ci_job_id 714936102 714936102
ci_pipeline_id 49573425 49573425
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.464 ms) : 1452, 1475
.   : milestone, 1464,
appsec (2.331 ms) : 2290, 2373
.   : milestone, 2331,
iast (2.075 ms) : 2023, 2127
.   : milestone, 2075,
iast_GLOBAL (2.123 ms) : 2071, 2176
.   : milestone, 2123,
profiling (1.944 ms) : 1902, 1986
.   : milestone, 1944,
tracing (1.925 ms) : 1885, 1965
.   : milestone, 1925,
section candidate
no_agent (1.462 ms) : 1451, 1473
.   : milestone, 1462,
appsec (2.327 ms) : 2285, 2368
.   : milestone, 2327,
iast (2.077 ms) : 2024, 2129
.   : milestone, 2077,
iast_GLOBAL (2.125 ms) : 2072, 2177
.   : milestone, 2125,
profiling (1.933 ms) : 1892, 1975
.   : milestone, 1933,
tracing (1.924 ms) : 1884, 1964
.   : milestone, 1924,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.464 ms [1.452 ms, 1.475 ms] -
appsec 2.331 ms [2.29 ms, 2.373 ms] 867.748 µs (59.3%)
iast 2.075 ms [2.023 ms, 2.127 ms] 611.409 µs (41.8%)
iast_GLOBAL 2.123 ms [2.071 ms, 2.176 ms] 659.678 µs (45.1%)
profiling 1.944 ms [1.902 ms, 1.986 ms] 479.891 µs (32.8%)
tracing 1.925 ms [1.885 ms, 1.965 ms] 461.012 µs (31.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.462 ms [1.451 ms, 1.473 ms] -
appsec 2.327 ms [2.285 ms, 2.368 ms] 864.668 µs (59.1%)
iast 2.077 ms [2.024 ms, 2.129 ms] 614.563 µs (42.0%)
iast_GLOBAL 2.125 ms [2.072 ms, 2.177 ms] 662.803 µs (45.3%)
profiling 1.933 ms [1.892 ms, 1.975 ms] 471.525 µs (32.3%)
tracing 1.924 ms [1.884 ms, 1.964 ms] 462.231 µs (31.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~5f1a938f02, baseline=1.43.0-SNAPSHOT~df51f63bab
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.493 s) : 15493000, 15493000
.   : milestone, 15493000,
appsec (14.916 s) : 14916000, 14916000
.   : milestone, 14916000,
iast (18.616 s) : 18616000, 18616000
.   : milestone, 18616000,
iast_GLOBAL (18.345 s) : 18345000, 18345000
.   : milestone, 18345000,
profiling (14.733 s) : 14733000, 14733000
.   : milestone, 14733000,
tracing (14.87 s) : 14870000, 14870000
.   : milestone, 14870000,
section candidate
no_agent (15.484 s) : 15484000, 15484000
.   : milestone, 15484000,
appsec (15.124 s) : 15124000, 15124000
.   : milestone, 15124000,
iast (18.966 s) : 18966000, 18966000
.   : milestone, 18966000,
iast_GLOBAL (17.835 s) : 17835000, 17835000
.   : milestone, 17835000,
profiling (14.698 s) : 14698000, 14698000
.   : milestone, 14698000,
tracing (14.703 s) : 14703000, 14703000
.   : milestone, 14703000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.493 s [15.493 s, 15.493 s] -
appsec 14.916 s [14.916 s, 14.916 s] -577.0 ms (-3.7%)
iast 18.616 s [18.616 s, 18.616 s] 3.123 s (20.2%)
iast_GLOBAL 18.345 s [18.345 s, 18.345 s] 2.852 s (18.4%)
profiling 14.733 s [14.733 s, 14.733 s] -760.0 ms (-4.9%)
tracing 14.87 s [14.87 s, 14.87 s] -623.0 ms (-4.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.484 s [15.484 s, 15.484 s] -
appsec 15.124 s [15.124 s, 15.124 s] -360.0 ms (-2.3%)
iast 18.966 s [18.966 s, 18.966 s] 3.482 s (22.5%)
iast_GLOBAL 17.835 s [17.835 s, 17.835 s] 2.351 s (15.2%)
profiling 14.698 s [14.698 s, 14.698 s] -786.0 ms (-5.1%)
tracing 14.703 s [14.703 s, 14.703 s] -781.0 ms (-5.0%)

@smola smola force-pushed the smola/telemetry-smoke-tests2 branch from bf4901e to 5bb82b8 Compare November 15, 2024 07:31
@smola smola force-pushed the smola/telemetry-smoke-tests2 branch from 7c77c8c to e077ee9 Compare November 15, 2024 12:00
@smola smola marked this pull request as ready for review November 21, 2024 07:52
@smola smola requested a review from a team as a code owner November 21, 2024 07:52
@smola smola requested a review from PerfectSlayer November 21, 2024 07:52
PerfectSlayer
PerfectSlayer reviewed Nov 21, 2024
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great addition! 👍

Quick question about the RunLast thing: rather than introducing a custom mechanism, what about using cleanup or cleanupSpec? Would it work?

@smola
Copy link
Member Author

smola commented Nov 21, 2024

@PerfectSlayer what about using cleanup or cleanupSpec? Would it work?

It should work, but then these wouldn't be represented as tests. If they failed, they would be represented as failed clean up. If they succeeded, they would be sort of silent. Unless there's some way to nest a test case within cleanupSpec?

@PerfectSlayer
Copy link
Contributor

Agreed, I was just checking if there was some other way to do it with some Spock capabilities.
In addition, parent spec seems to be run first, which is the worst case scenario in your case.

You're good to merge then!

@smola smola merged commit cd1b746 into master Nov 21, 2024
102 checks passed
@smola smola deleted the smola/telemetry-smoke-tests2 branch November 21, 2024 13:38
@github-actions github-actions bot added this to the 1.43.0 milestone Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

Assignees
No one assigned
Labels
comp: telemetry Telemetry comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.43.0
Development

Successfully merging this pull request may close these issues.
2 participants
@smola @PerfectSlayer