Fix tracing JUnit5 tests in Maven projects with multiple forks #8086
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikita-tkachenko-datadog
added
type: bug
comp: ci visibility
nikita-tkachenko-datadog
force-pushed
the
nikita-tkachenko/fix-maven-multiple-forks
branch
from
d01bb7a to
56d64ad
Compare
anmarchenko
approved these changes
Dec 13, 2024
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 11 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.093 s) : 0, 1093141
Total [baseline] (8.707 s) : 0, 8706796
Agent [candidate] (1.105 s) : 0, 1104933
Total [candidate] (8.683 s) : 0, 8683163
section iast
Agent [baseline] (1.222 s) : 0, 1221501
Total [baseline] (9.217 s) : 0, 9216591
Agent [candidate] (1.22 s) : 0, 1220245
Total [candidate] (9.214 s) : 0, 9214421
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.22 s) : 0, 1220421
Total [baseline] (9.173 s) : 0, 9172708
Agent [candidate] (1.237 s) : 0, 1236759
Total [candidate] (9.248 s) : 0, 9247854
section iast_TELEMETRY_OFF
Agent [baseline] (1.226 s) : 0, 1225819
Total [baseline] (9.262 s) : 0, 9262163
Agent [candidate] (1.216 s) : 0, 1215588
Total [candidate] (9.193 s) : 0, 9192630
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (696.032 ms) : 0, 696032
BytebuddyAgent [candidate] (705.466 ms) : 0, 705466
GlobalTracer [baseline] (316.748 ms) : 0, 316748
GlobalTracer [candidate] (319.03 ms) : 0, 319030
AppSec [baseline] (54.6 ms) : 0, 54600
AppSec [candidate] (55.202 ms) : 0, 55202
Remote Config [baseline] (677.28 µs) : 0, 677
Remote Config [candidate] (693.757 µs) : 0, 694
Telemetry [baseline] (11.37 ms) : 0, 11370
Telemetry [candidate] (10.62 ms) : 0, 10620
section iast
BytebuddyAgent [baseline] (814.173 ms) : 0, 814173
BytebuddyAgent [candidate] (813.284 ms) : 0, 813284
GlobalTracer [baseline] (305.684 ms) : 0, 305684
GlobalTracer [candidate] (306.127 ms) : 0, 306127
AppSec [baseline] (56.775 ms) : 0, 56775
AppSec [candidate] (57.858 ms) : 0, 57858
IAST [baseline] (22.896 ms) : 0, 22896
IAST [candidate] (21.101 ms) : 0, 21101
Remote Config [baseline] (627.548 µs) : 0, 628
Remote Config [candidate] (632.274 µs) : 0, 632
Telemetry [baseline] (7.618 ms) : 0, 7618
Telemetry [candidate] (7.494 ms) : 0, 7494
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (813.875 ms) : 0, 813875
BytebuddyAgent [candidate] (823.295 ms) : 0, 823295
GlobalTracer [baseline] (305.571 ms) : 0, 305571
GlobalTracer [candidate] (310.301 ms) : 0, 310301
AppSec [baseline] (57.168 ms) : 0, 57168
AppSec [candidate] (58.364 ms) : 0, 58364
IAST [baseline] (21.946 ms) : 0, 21946
IAST [candidate] (22.656 ms) : 0, 22656
Remote Config [baseline] (624.897 µs) : 0, 625
Remote Config [candidate] (642.654 µs) : 0, 643
Telemetry [baseline] (7.474 ms) : 0, 7474
Telemetry [candidate] (7.62 ms) : 0, 7620
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (816.579 ms) : 0, 816579
BytebuddyAgent [candidate] (809.945 ms) : 0, 809945
GlobalTracer [baseline] (307.422 ms) : 0, 307422
GlobalTracer [candidate] (305.362 ms) : 0, 305362
AppSec [baseline] (58.061 ms) : 0, 58061
AppSec [candidate] (57.134 ms) : 0, 57134
IAST [baseline] (21.815 ms) : 0, 21815
IAST [candidate] (21.382 ms) : 0, 21382
Remote Config [baseline] (637.322 µs) : 0, 637
Remote Config [candidate] (635.745 µs) : 0, 636
Telemetry [baseline] (7.473 ms) : 0, 7473
Telemetry [candidate] (7.309 ms) : 0, 7309
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.089 s) : 0, 1089312
Total [baseline] (10.418 s) : 0, 10418143
Agent [candidate] (1.09 s) : 0, 1089514
Total [candidate] (10.47 s) : 0, 10469598
section appsec
Agent [baseline] (1.227 s) : 0, 1227306
Total [baseline] (10.709 s) : 0, 10708629
Agent [candidate] (1.227 s) : 0, 1226906
Total [candidate] (10.725 s) : 0, 10725049
section iast
Agent [baseline] (1.231 s) : 0, 1231114
Total [baseline] (10.994 s) : 0, 10993550
Agent [candidate] (1.224 s) : 0, 1223617
Total [candidate] (10.989 s) : 0, 10989231
section profiling
Agent [baseline] (1.316 s) : 0, 1316329
Total [baseline] (10.778 s) : 0, 10777897
Agent [candidate] (1.322 s) : 0, 1322479
Total [candidate] (10.812 s) : 0, 10812460
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.635 ms) : 0, 693635
BytebuddyAgent [candidate] (695.143 ms) : 0, 695143
GlobalTracer [baseline] (316.26 ms) : 0, 316260
GlobalTracer [candidate] (316.458 ms) : 0, 316458
AppSec [baseline] (54.499 ms) : 0, 54499
AppSec [candidate] (54.887 ms) : 0, 54887
Remote Config [baseline] (694.094 µs) : 0, 694
Remote Config [candidate] (706.451 µs) : 0, 706
Telemetry [baseline] (10.548 ms) : 0, 10548
Telemetry [candidate] (8.592 ms) : 0, 8592
section appsec
BytebuddyAgent [baseline] (713.463 ms) : 0, 713463
BytebuddyAgent [candidate] (712.957 ms) : 0, 712957
GlobalTracer [baseline] (314.271 ms) : 0, 314271
GlobalTracer [candidate] (314.47 ms) : 0, 314470
AppSec [baseline] (167.642 ms) : 0, 167642
AppSec [candidate] (167.331 ms) : 0, 167331
IAST [baseline] (19.719 ms) : 0, 19719
IAST [candidate] (19.918 ms) : 0, 19918
Remote Config [baseline] (644.224 µs) : 0, 644
Remote Config [candidate] (645.875 µs) : 0, 646
Telemetry [baseline] (7.722 ms) : 0, 7722
Telemetry [candidate] (7.837 ms) : 0, 7837
section iast
BytebuddyAgent [baseline] (820.588 ms) : 0, 820588
BytebuddyAgent [candidate] (815.652 ms) : 0, 815652
GlobalTracer [baseline] (308.931 ms) : 0, 308931
GlobalTracer [candidate] (306.785 ms) : 0, 306785
AppSec [baseline] (57.601 ms) : 0, 57601
AppSec [candidate] (57.272 ms) : 0, 57272
IAST [baseline] (22.016 ms) : 0, 22016
IAST [candidate] (22.036 ms) : 0, 22036
Remote Config [baseline] (633.273 µs) : 0, 633
Remote Config [candidate] (626.542 µs) : 0, 627
Telemetry [baseline] (7.527 ms) : 0, 7527
Telemetry [candidate] (7.5 ms) : 0, 7500
section profiling
BytebuddyAgent [baseline] (688.571 ms) : 0, 688571
BytebuddyAgent [candidate] (693.251 ms) : 0, 693251
GlobalTracer [baseline] (432.849 ms) : 0, 432849
GlobalTracer [candidate] (434.191 ms) : 0, 434191
AppSec [baseline] (53.472 ms) : 0, 53472
AppSec [candidate] (53.811 ms) : 0, 53811
Remote Config [baseline] (668.258 µs) : 0, 668
Remote Config [candidate] (703.491 µs) : 0, 703
Telemetry [baseline] (7.623 ms) : 0, 7623
Telemetry [candidate] (7.668 ms) : 0, 7668
ProfilingAgent [baseline] (93.846 ms) : 0, 93846
ProfilingAgent [candidate] (93.455 ms) : 0, 93455
Profiling [baseline] (93.869 ms) : 0, 93869
Profiling [candidate] (93.479 ms) : 0, 93479
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section baseline
no_agent (374.545 µs) : 354, 395
. : milestone, 375,
iast (488.724 µs) : 467, 510
. : milestone, 489,
iast_FULL (651.365 µs) : 630, 673
. : milestone, 651,
iast_GLOBAL (519.641 µs) : 498, 541
. : milestone, 520,
iast_HARDCODED_SECRET_DISABLED (493.397 µs) : 472, 515
. : milestone, 493,
iast_INACTIVE (455.184 µs) : 434, 476
. : milestone, 455,
iast_TELEMETRY_OFF (480.479 µs) : 459, 502
. : milestone, 480,
tracing (454.371 µs) : 433, 475
. : milestone, 454,
section candidate
no_agent (376.948 µs) : 357, 397
. : milestone, 377,
iast (496.18 µs) : 475, 518
. : milestone, 496,
iast_FULL (653.959 µs) : 632, 676
. : milestone, 654,
iast_GLOBAL (529.558 µs) : 507, 552
. : milestone, 530,
iast_HARDCODED_SECRET_DISABLED (488.618 µs) : 467, 510
. : milestone, 489,
iast_INACTIVE (450.317 µs) : 429, 471
. : milestone, 450,
iast_TELEMETRY_OFF (481.742 µs) : 460, 503
. : milestone, 482,
tracing (455.848 µs) : 435, 477
. : milestone, 456,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section baseline
no_agent (1.35 ms) : 1331, 1368
. : milestone, 1350,
appsec (1.753 ms) : 1730, 1776
. : milestone, 1753,
appsec_no_iast (1.776 ms) : 1751, 1800
. : milestone, 1776,
iast (1.493 ms) : 1471, 1515
. : milestone, 1493,
profiling (1.535 ms) : 1510, 1559
. : milestone, 1535,
tracing (1.494 ms) : 1469, 1519
. : milestone, 1494,
section candidate
no_agent (1.355 ms) : 1336, 1374
. : milestone, 1355,
appsec (1.759 ms) : 1735, 1783
. : milestone, 1759,
appsec_no_iast (1.766 ms) : 1739, 1793
. : milestone, 1766,
iast (1.511 ms) : 1488, 1533
. : milestone, 1511,
profiling (1.561 ms) : 1536, 1586
. : milestone, 1561,
tracing (1.505 ms) : 1480, 1529
. : milestone, 1505,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section baseline
no_agent (1.477 ms) : 1466, 1489
. : milestone, 1477,
appsec (2.365 ms) : 2323, 2407
. : milestone, 2365,
iast (2.096 ms) : 2043, 2149
. : milestone, 2096,
iast_GLOBAL (2.148 ms) : 2094, 2202
. : milestone, 2148,
profiling (1.965 ms) : 1922, 2007
. : milestone, 1965,
tracing (1.943 ms) : 1903, 1984
. : milestone, 1943,
section candidate
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (2.358 ms) : 2316, 2400
. : milestone, 2358,
iast (2.086 ms) : 2033, 2138
. : milestone, 2086,
iast_GLOBAL (2.145 ms) : 2092, 2198
. : milestone, 2145,
profiling (1.963 ms) : 1921, 2006
. : milestone, 1963,
tracing (1.935 ms) : 1894, 1975
. : milestone, 1935,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~56d64adbf0, baseline=1.45.0-SNAPSHOT~baedf8dd57
dateFormat X
axisFormat %s
section baseline
no_agent (15.488 s) : 15488000, 15488000
. : milestone, 15488000,
appsec (15.034 s) : 15034000, 15034000
. : milestone, 15034000,
iast (18.684 s) : 18684000, 18684000
. : milestone, 18684000,
iast_GLOBAL (18.066 s) : 18066000, 18066000
. : milestone, 18066000,
profiling (15.316 s) : 15316000, 15316000
. : milestone, 15316000,
tracing (14.861 s) : 14861000, 14861000
. : milestone, 14861000,
section candidate
no_agent (15.59 s) : 15590000, 15590000
. : milestone, 15590000,
appsec (14.911 s) : 14911000, 14911000
. : milestone, 14911000,
iast (19.173 s) : 19173000, 19173000
. : milestone, 19173000,
iast_GLOBAL (18.321 s) : 18321000, 18321000
. : milestone, 18321000,
profiling (15.038 s) : 15038000, 15038000
. : milestone, 15038000,
tracing (14.837 s) : 14837000, 14837000
. : milestone, 14837000,
|
nikita-tkachenko-datadog
deleted the
nikita-tkachenko/fix-maven-multiple-forks
branch
nikita-tkachenko-datadog
added
the
tag: no release notes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Fixes a regression introduced in release v1.44.0 (#8055).
When tracing JUnit 5 tests in Maven projects with multiple forks the following exception was seen:
Maven Surefire plugin can be configured to fork multiple JVMs per test task in order to parallelise tests execution in multiple processes:
When configure like this, Maven Surefire will invoke
AbstractPlatformProvider.getSuites()in order to list available test cases and divide them between the forks.If the tests are in JUnit 5, the platform provide will in turn invoke
JupiterTestEngine.discover()to get the list of tests.Release v1.44.0 introduced some changes in the way JUnit 5 is instrumented.
Among them was early initialisation of
TestEventHandler(needed to have the handler available by the time class and method orderers were requested from JUnit).Early initialisation happens at
JupiterTestEngine.discover()- the assumption was that this method was only executed in child JVMs.Since in the parent JVM (Maven build process)
TestEventHandlerfactory is not configured, early initialisation caused NPEs.The fix is to create a no-op TestEventHandler implementation in the parent process as a stub, so that the instrumentation does not fail.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: SDTEST-1336