Reset CI Visibility after DuckTypingTests #6509

kevingosse · 2025-01-08T13:31:07Z

Summary of changes

Stop and reset CI Visibility after DuckTypingTests.CanDuckTypeManualTestSessionAsISession

Reason for change

When CI visibility is enabled, the origin of the new SpanContext instances is automatically set, which can cause some unit tests to fail if they happen to be executed afterwards.

Other details

Discovered while trying to randomize the order of the tests.

datadog-ddstaging · 2025-01-08T13:53:55Z

Datadog Report

Branch report: kevin/civisibility_tests
Commit report: de494e9
Test service: dd-trace-dotnet

✅ 0 Failed, 242796 Passed, 2133 Skipped, 19h 2m 57.18s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

EarlyFlakeDetection - Datadog.Trace.ClrProfiler.IntegrationTests.CI.TcpXUnitEvpTests - Last Failure
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```

andrewlock · 2025-01-08T14:10:50Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (69ms)  : 64, 74
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (982ms)  : 956, 1007
     .   : milestone, 982,
    master - mean (981ms)  : 954, 1008
     .   : milestone, 981,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (108ms)  : 105, 111
     .   : milestone, 108,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (678ms)  : 662, 694
     .   : milestone, 678,
    master - mean (678ms)  : 662, 695
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 90, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (634ms)  : 615, 652
     .   : milestone, 634,
    master - mean (633ms)  : 616, 651
     .   : milestone, 633,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (194ms)  : 189, 199
     .   : milestone, 194,
    master - mean (195ms)  : 189, 202
     .   : milestone, 195,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (1,101ms)  : 1070, 1132
     .   : milestone, 1101,
    master - mean (1,104ms)  : 1076, 1132
     .   : milestone, 1104,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (279ms)  : 274, 283
     .   : milestone, 279,
    master - mean (278ms)  : 273, 282
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (873ms)  : 845, 900
     .   : milestone, 873,
    master - mean (874ms)  : 847, 901
     .   : milestone, 874,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6509) - mean (267ms)  : 263, 270
     .   : milestone, 267,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6509) - mean (853ms)  : 820, 885
     .   : milestone, 853,
    master - mean (857ms)  : 829, 885
     .   : milestone, 857,

andrewlock · 2025-01-08T15:04:02Z

Benchmarks Report for tracer 🐌

Benchmarks for #6509 compared to master:

1 benchmarks are faster, with geometric mean 1.155
2 benchmarks are slower, with geometric mean 1.165
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.11μs	45.6ns	329ns	0.0166	0.0083	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	50ns	320ns	0.0157	0.00524	0	5.8 KB
master	`StartStopWithChild`	net472	16.6μs	67.3ns	261ns	1.06	0.331	0.0889	6.21 KB
#6509	`StartStopWithChild`	net6.0	7.8μs	43.5ns	275ns	0.0195	0.00779	0	5.61 KB
#6509	`StartStopWithChild`	netcoreapp3.1	9.82μs	45.6ns	188ns	0.0197	0.00987	0	5.8 KB
#6509	`StartStopWithChild`	net472	16.3μs	35ns	126ns	1.04	0.304	0.0822	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	480μs	324ns	1.26μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	654μs	328ns	1.18μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	855μs	841ns	3.26μs	0.422	3.3 KB
#6509	`WriteAndFlushEnrichedTraces`	net6.0	486μs	241ns	934ns	0	2.7 KB
#6509	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	616ns	2.39μs	0	2.7 KB
#6509	`WriteAndFlushEnrichedTraces`	net472	849μs	686ns	2.66μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	156μs	957ns	9.52μs	0.152	14.47 KB
master	`SendRequest`	netcoreapp3.1	172μs	1.01μs	9.35μs	0.165	17.27 KB
master	`SendRequest`	net472	0.00178ns	0.000725ns	0.00281ns	0	0 b
#6509	`SendRequest`	net6.0	155μs	903ns	7.97μs	0.137	14.47 KB
#6509	`SendRequest`	netcoreapp3.1	162μs	761ns	3.04μs	0.19	17.27 KB
#6509	`SendRequest`	net472	0.00157ns	0.000789ns	0.00306ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	581μs	3.27μs	22μs	0.587	0	0	41.6 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	671μs	3.72μs	22μs	0.322	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	867μs	3.9μs	15.1μs	8.19	2.59	0.431	53.33 KB
#6509	`WriteAndFlushEnrichedTraces`	net6.0	575μs	3.3μs	25.8μs	0.592	0	0	41.55 KB
#6509	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	682μs	2.99μs	11.2μs	0.327	0	0	41.81 KB
#6509	`WriteAndFlushEnrichedTraces`	net472	830μs	3.84μs	14.9μs	8.33	2.5	0.417	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.29μs	1.29ns	4.84ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.33ns	4.97ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.13μs	2.18ns	8.44ns	0.157	0.00106	987 B
#6509	`ExecuteNonQuery`	net6.0	1.31μs	2.55ns	9.86ns	0.0145	0	1.02 KB
#6509	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	2.86ns	10.7ns	0.0137	0	1.02 KB
#6509	`ExecuteNonQuery`	net472	2.1μs	1.14ns	4.28ns	0.156	0.00106	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.16μs	0.665ns	2.49ns	0.0139	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.66ns	6.21ns	0.0132	976 B
master	`CallElasticsearch`	net472	2.53μs	2.45ns	9.16ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.23μs	0.618ns	2.31ns	0.013	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	0.65ns	2.43ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	1.79ns	6.94ns	0.167	1.05 KB
#6509	`CallElasticsearch`	net6.0	1.12μs	0.651ns	2.44ns	0.0138	976 B
#6509	`CallElasticsearch`	netcoreapp3.1	1.61μs	0.841ns	3.26ns	0.0128	976 B
#6509	`CallElasticsearch`	net472	2.6μs	2.61ns	10.1ns	0.158	995 B
#6509	`CallElasticsearchAsync`	net6.0	1.24μs	0.99ns	3.84ns	0.0131	952 B
#6509	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.818ns	3.06ns	0.0139	1.02 KB
#6509	`CallElasticsearchAsync`	net472	2.55μs	1.39ns	5.21ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.28μs	0.864ns	3.23ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.65μs	0.581ns	2.18ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.82μs	1.09ns	3.78ns	0.145	915 B
#6509	`ExecuteAsync`	net6.0	1.38μs	0.744ns	2.68ns	0.0132	952 B
#6509	`ExecuteAsync`	netcoreapp3.1	1.61μs	1.16ns	4.34ns	0.0128	952 B
#6509	`ExecuteAsync`	net472	1.77μs	0.515ns	1.99ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.45μs	1.73ns	6.68ns	0.0315	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.29μs	2.82ns	10.9ns	0.0371	2.85 KB
master	`SendAsync`	net472	7.36μs	3.05ns	11ns	0.494	3.12 KB
#6509	`SendAsync`	net6.0	4.39μs	2.17ns	8.13ns	0.0307	2.31 KB
#6509	`SendAsync`	netcoreapp3.1	5.32μs	2.27ns	8.49ns	0.0371	2.85 KB
#6509	`SendAsync`	net472	7.17μs	2.16ns	8.38ns	0.494	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.54μs	0.653ns	2.45ns	0.0231	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.09μs	0.997ns	3.73ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.69μs	2.29ns	8.57ns	0.249	1.57 KB
#6509	`EnrichedLog`	net6.0	1.59μs	0.748ns	2.8ns	0.023	1.64 KB
#6509	`EnrichedLog`	netcoreapp3.1	2.16μs	1.29ns	4.84ns	0.0215	1.64 KB
#6509	`EnrichedLog`	net472	2.7μs	1.77ns	6.85ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	195ns	756ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	164ns	635ns	0.0603	0	4.28 KB
master	`EnrichedLog`	net472	151μs	173ns	648ns	0.671	0.224	4.46 KB
#6509	`EnrichedLog`	net6.0	118μs	391ns	1.51μs	0.0586	0	4.28 KB
#6509	`EnrichedLog`	netcoreapp3.1	120μs	203ns	787ns	0	0	4.28 KB
#6509	`EnrichedLog`	net472	153μs	265ns	991ns	0.682	0.227	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.92μs	1.13ns	4.38ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.2μs	1.84ns	6.88ns	0.0294	2.2 KB
master	`EnrichedLog`	net472	4.88μs	1.69ns	6.54ns	0.319	2.02 KB
#6509	`EnrichedLog`	net6.0	3.04μs	0.855ns	3.31ns	0.0307	2.2 KB
#6509	`EnrichedLog`	netcoreapp3.1	4.25μs	1.44ns	4.99ns	0.0297	2.2 KB
#6509	`EnrichedLog`	net472	4.99μs	1.55ns	6ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.818ns	3.17ns	0.0162	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.633ns	2.28ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.13μs	1.58ns	5.92ns	0.184	1.16 KB
#6509	`SendReceive`	net6.0	1.33μs	0.696ns	2.7ns	0.0159	1.14 KB
#6509	`SendReceive`	netcoreapp3.1	1.78μs	0.679ns	2.63ns	0.015	1.14 KB
#6509	`SendReceive`	net472	2.13μs	0.82ns	2.96ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.79μs	0.881ns	3.3ns	0.0223	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	1.6ns	6.19ns	0.0219	1.65 KB
master	`EnrichedLog`	net472	4.24μs	3.28ns	11.8ns	0.322	2.04 KB
#6509	`EnrichedLog`	net6.0	2.64μs	1.2ns	4.5ns	0.0224	1.6 KB
#6509	`EnrichedLog`	netcoreapp3.1	3.87μs	1.97ns	7.64ns	0.0212	1.65 KB
#6509	`EnrichedLog`	net472	4.31μs	1.47ns	5.3ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6509

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.156	538.86	622.97

Faster 🎉 in #6509

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.155	562.66	487.23

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	402ns	0.511ns	1.98ns	0.0081	576 B
master	`StartFinishSpan`	netcoreapp3.1	539ns	0.687ns	2.66ns	0.00777	576 B
master	`StartFinishSpan`	net472	650ns	1.43ns	5.56ns	0.0917	578 B
master	`StartFinishScope`	net6.0	563ns	0.836ns	3.24ns	0.00971	696 B
master	`StartFinishScope`	netcoreapp3.1	713ns	1.11ns	4.31ns	0.00965	696 B
master	`StartFinishScope`	net472	885ns	1.17ns	4.53ns	0.104	658 B
#6509	`StartFinishSpan`	net6.0	417ns	1.22ns	4.71ns	0.00807	576 B
#6509	`StartFinishSpan`	netcoreapp3.1	623ns	0.726ns	2.81ns	0.00762	576 B
#6509	`StartFinishSpan`	net472	655ns	0.872ns	3.38ns	0.0918	578 B
#6509	`StartFinishScope`	net6.0	487ns	0.631ns	2.44ns	0.00967	696 B
#6509	`StartFinishScope`	netcoreapp3.1	665ns	0.719ns	2.78ns	0.00938	696 B
#6509	`StartFinishScope`	net472	829ns	1.11ns	4.3ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6509

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.174	600.43	704.89

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	600ns	1.14ns	4.4ns	0.00973	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	896ns	1.28ns	4.96ns	0.00933	696 B
master	`RunOnMethodBegin`	net472	1.09μs	1.93ns	7.46ns	0.104	658 B
#6509	`RunOnMethodBegin`	net6.0	705ns	0.701ns	2.72ns	0.00956	696 B
#6509	`RunOnMethodBegin`	netcoreapp3.1	923ns	2.49ns	9.65ns	0.00921	696 B
#6509	`RunOnMethodBegin`	net472	1.1μs	1.31ns	4.72ns	0.104	658 B

## Summary of changes Randomize the order of the tests. ## Reason for change Flaky tests are much harder to fix when we discover them long after they have been written. By randomizing the order of the tests, I'm hoping to make them fail earlier. In practice, this could temporarily increase the overall flakiness, but I expect this will reduce the overall effort spent on fixing tests. ## Implementation details In `CustomTestFramework`, randomize the list of all tests in each collections, and the collections themselves. The seed is displayed in the output. When a test order causes tests to fail, this allows to deterministically reproduce that test order. ## Other details Four other issues were found thanks to this: #6535, #6532, #6511, #6509

kevingosse requested a review from a team as a code owner January 8, 2025 13:31

github-actions bot added the area:tests unit tests, integration tests label Jan 8, 2025

andrewlock approved these changes Jan 8, 2025

View reviewed changes

tonyredondo approved these changes Jan 8, 2025

View reviewed changes

Reset CI Visibility after CanDuckTypeManualTestSessionAsISession

de494e9

kevingosse force-pushed the kevin/civisibility_tests branch from 139f40c to de494e9 Compare January 8, 2025 16:05

kevingosse merged commit c8dcc63 into master Jan 9, 2025
100 of 102 checks passed

kevingosse deleted the kevin/civisibility_tests branch January 9, 2025 10:06

github-actions bot added this to the vNext-v3 milestone Jan 9, 2025

kevingosse mentioned this pull request Jan 14, 2025

Randomize the order of tests #5831

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reset CI Visibility after DuckTypingTests #6509

Reset CI Visibility after DuckTypingTests #6509

kevingosse commented Jan 8, 2025

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6509

Faster 🎉 in #6509

Raw results

Slower ⚠️ in #6509

Raw results

Reset CI Visibility after DuckTypingTests #6509

Reset CI Visibility after DuckTypingTests #6509

Conversation

kevingosse commented Jan 8, 2025

Summary of changes

Reason for change

Other details

datadog-ddstaging bot commented Jan 8, 2025 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Jan 8, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 8, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6509

Faster 🎉 in #6509

Raw results

Slower ⚠️ in #6509

Raw results

datadog-ddstaging bot commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading

andrewlock commented Jan 8, 2025 •

edited

Loading