[CI Visibility] Improve Git commands execution #6545

tonyredondo · 2025-01-13T22:32:07Z

Summary of changes

This PR changes the way CI Visiblity execute the Git commands, running the commands synchronous (CI Visibility was the only one using the async version of the ProcessHelper), also it adds support for command timeout, so we avoid hanging the process until a git command finishes.

Reason for change

I detected some deadlocks while executing git commands in Alpine Arm64.

Implementation details

Removing some async code and refactor the IntelligentTestRunnerClient class.

Test coverage

The same test should pass with no change of behaviour. The Alpine issue should be fixed with this PR.

Other details

Ticket: SDTEST-1422

datadog-ddstaging · 2025-01-13T23:11:46Z

Datadog Report

Branch report: tony/civisibility-improve-git-commands
Commit report: ea8797f
Test service: dd-trace-dotnet

✅ 0 Failed, 239431 Passed, 1981 Skipped, 18h 33m 5.79s Total Time

andrewlock · 2025-01-13T23:17:22Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (70ms)  : 63, 76
     .   : milestone, 70,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (978ms)  : 958, 999
     .   : milestone, 978,
    master - mean (975ms)  : 946, 1003
     .   : milestone, 975,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (672ms)  : 658, 687
     .   : milestone, 672,
    master - mean (677ms)  : 661, 694
     .   : milestone, 677,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (91ms)  : 89, 94
     .   : milestone, 91,
    master - mean (90ms)  : 89, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (629ms)  : 614, 644
     .   : milestone, 629,
    master - mean (630ms)  : 615, 646
     .   : milestone, 630,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (194ms)  : 189, 198
     .   : milestone, 194,
    master - mean (194ms)  : 189, 199
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (1,098ms)  : 1066, 1129
     .   : milestone, 1098,
    master - mean (1,104ms)  : 1073, 1135
     .   : milestone, 1104,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (278ms)  : 274, 282
     .   : milestone, 278,
    master - mean (280ms)  : 273, 286
     .   : milestone, 280,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (867ms)  : 842, 891
     .   : milestone, 867,
    master - mean (869ms)  : 842, 896
     .   : milestone, 869,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6545) - mean (267ms)  : 263, 271
     .   : milestone, 267,
    master - mean (267ms)  : 262, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6545) - mean (850ms)  : 821, 879
     .   : milestone, 850,
    master - mean (853ms)  : 825, 880
     .   : milestone, 853,

andrewlock

Nice work! 🤞 thanks!

tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs

tracer/src/Datadog.Trace/Util/ProcessHelpers.cs

andrewlock · 2025-01-14T16:48:19Z

Benchmarks Report for tracer 🐌

Benchmarks for #6545 compared to master:

2 benchmarks are slower, with geometric mean 1.206
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.3μs	48.2ns	449ns	0.0137	0.00686	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	57ns	369ns	0.0205	0.0102	0	5.8 KB
master	`StartStopWithChild`	net472	16.2μs	40.1ns	155ns	1.05	0.306	0.0965	6.2 KB
#6545	`StartStopWithChild`	net6.0	8.14μs	45ns	288ns	0.0167	0.00837	0	5.61 KB
#6545	`StartStopWithChild`	netcoreapp3.1	10.1μs	56.1ns	359ns	0.0152	0.00507	0	5.8 KB
#6545	`StartStopWithChild`	net472	16.5μs	48.9ns	189ns	1.05	0.315	0.0969	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	501μs	243ns	940ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	669μs	684ns	2.56μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	919ns	3.56μs	0.422	3.3 KB
#6545	`WriteAndFlushEnrichedTraces`	net6.0	480μs	310ns	1.2μs	0	2.7 KB
#6545	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	858ns	3.32μs	0	2.7 KB
#6545	`WriteAndFlushEnrichedTraces`	net472	857μs	428ns	1.66μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	164μs	1.34μs	13.3μs	0.145	14.48 KB
master	`SendRequest`	netcoreapp3.1	182μs	1.19μs	11.7μs	0.179	17.27 KB
master	`SendRequest`	net472	0.00798ns	0.00243ns	0.00908ns	0	0 b
#6545	`SendRequest`	net6.0	151μs	863ns	8.37μs	0.167	14.47 KB
#6545	`SendRequest`	netcoreapp3.1	176μs	1.12μs	11.1μs	0.166	17.27 KB
#6545	`SendRequest`	net472	0.0053ns	0.00167ns	0.00647ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	610μs	3.3μs	29.5μs	0.581	0	0	41.78 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	728μs	5.77μs	57.4μs	0.332	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	net472	874μs	3.54μs	13.2μs	8.08	2.55	0.425	53.34 KB
#6545	`WriteAndFlushEnrichedTraces`	net6.0	576μs	3.25μs	23μs	0.536	0	0	41.62 KB
#6545	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	673μs	3.32μs	13.7μs	0.345	0	0	41.82 KB
#6545	`WriteAndFlushEnrichedTraces`	net472	850μs	2.86μs	10.7μs	8.13	2.57	0.428	53.31 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.31μs	1.55ns	5.99ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.04ns	3.89ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.04μs	1.89ns	6.83ns	0.157	0.00102	987 B
#6545	`ExecuteNonQuery`	net6.0	1.35μs	1.36ns	5.09ns	0.0144	0	1.02 KB
#6545	`ExecuteNonQuery`	netcoreapp3.1	1.83μs	1.77ns	6.62ns	0.0135	0	1.02 KB
#6545	`ExecuteNonQuery`	net472	2.14μs	3.29ns	12.7ns	0.156	0.00106	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.26μs	0.915ns	3.43ns	0.0133	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.63μs	0.725ns	2.71ns	0.013	976 B
master	`CallElasticsearch`	net472	2.64μs	3.75ns	14.5ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.34μs	0.935ns	3.62ns	0.0134	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	0.983ns	3.55ns	0.0135	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	3.01ns	11.7ns	0.167	1.05 KB
#6545	`CallElasticsearch`	net6.0	1.22μs	1.39ns	5.22ns	0.0135	976 B
#6545	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.829ns	3.1ns	0.0133	976 B
#6545	`CallElasticsearch`	net472	2.64μs	2.86ns	11.1ns	0.158	995 B
#6545	`CallElasticsearchAsync`	net6.0	1.29μs	1.25ns	4.85ns	0.0136	952 B
#6545	`CallElasticsearchAsync`	netcoreapp3.1	1.74μs	1.5ns	5.61ns	0.0131	1.02 KB
#6545	`CallElasticsearchAsync`	net472	2.65μs	2.19ns	8.5ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.3μs	1.03ns	3.85ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.56μs	1.28ns	4.62ns	0.0125	952 B
master	`ExecuteAsync`	net472	1.85μs	0.853ns	3.19ns	0.145	915 B
#6545	`ExecuteAsync`	net6.0	1.43μs	0.974ns	3.65ns	0.0129	952 B
#6545	`ExecuteAsync`	netcoreapp3.1	1.67μs	0.674ns	2.43ns	0.0126	952 B
#6545	`ExecuteAsync`	net472	1.85μs	0.634ns	2.29ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.28μs	1.45ns	5.41ns	0.0323	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.25μs	0.952ns	3.69ns	0.0368	2.85 KB
master	`SendAsync`	net472	7.29μs	2.21ns	8.27ns	0.495	3.12 KB
#6545	`SendAsync`	net6.0	4.38μs	1.51ns	5.65ns	0.0311	2.31 KB
#6545	`SendAsync`	netcoreapp3.1	5.27μs	1.63ns	6.08ns	0.0375	2.85 KB
#6545	`SendAsync`	net472	7.26μs	1.47ns	5.3ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.698ns	2.7ns	0.0228	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.28μs	1.05ns	3.93ns	0.0216	1.64 KB
master	`EnrichedLog`	net472	2.73μs	2.24ns	8.37ns	0.249	1.57 KB
#6545	`EnrichedLog`	net6.0	1.56μs	0.791ns	2.96ns	0.0226	1.64 KB
#6545	`EnrichedLog`	netcoreapp3.1	2.27μs	0.673ns	2.43ns	0.0216	1.64 KB
#6545	`EnrichedLog`	net472	2.65μs	0.53ns	1.84ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	180ns	699ns	0.0585	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	132ns	510ns	0	0	4.28 KB
master	`EnrichedLog`	net472	151μs	167ns	645ns	0.68	0.227	4.46 KB
#6545	`EnrichedLog`	net6.0	118μs	117ns	452ns	0.0589	0	4.28 KB
#6545	`EnrichedLog`	netcoreapp3.1	122μs	240ns	899ns	0.0603	0	4.28 KB
#6545	`EnrichedLog`	net472	151μs	130ns	502ns	0.679	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.9μs	0.932ns	3.61ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.25μs	2.47ns	9.57ns	0.0297	2.2 KB
master	`EnrichedLog`	net472	4.99μs	1.77ns	6.86ns	0.319	2.02 KB
#6545	`EnrichedLog`	net6.0	3.15μs	9.16ns	35.5ns	0.0308	2.2 KB
#6545	`EnrichedLog`	netcoreapp3.1	3.92μs	1.66ns	6.22ns	0.0296	2.2 KB
#6545	`EnrichedLog`	net472	4.84μs	1.87ns	7.26ns	0.321	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.38μs	1.35ns	5.03ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.909ns	3.52ns	0.015	1.14 KB
master	`SendReceive`	net472	2.06μs	1.62ns	6.29ns	0.183	1.16 KB
#6545	`SendReceive`	net6.0	1.49μs	1.28ns	4.94ns	0.0158	1.14 KB
#6545	`SendReceive`	netcoreapp3.1	1.66μs	0.439ns	1.58ns	0.0149	1.14 KB
#6545	`SendReceive`	net472	2.05μs	0.705ns	2.64ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.65μs	0.845ns	3.16ns	0.0225	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.9μs	1.33ns	5.15ns	0.0215	1.65 KB
master	`EnrichedLog`	net472	4.36μs	4.27ns	16ns	0.323	2.04 KB
#6545	`EnrichedLog`	net6.0	2.69μs	0.573ns	2.06ns	0.0216	1.6 KB
#6545	`EnrichedLog`	netcoreapp3.1	3.77μs	1.49ns	5.57ns	0.0209	1.65 KB
#6545	`EnrichedLog`	net472	4.32μs	4.86ns	18.8ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6545

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.219	465.70	567.69
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.193	816.48	974.12

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.259ns	1ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	621ns	1.21ns	4.7ns	0.00773	576 B
master	`StartFinishSpan`	net472	712ns	0.333ns	1.29ns	0.0916	578 B
master	`StartFinishScope`	net6.0	466ns	0.212ns	0.766ns	0.00978	696 B
master	`StartFinishScope`	netcoreapp3.1	718ns	2.25ns	8.73ns	0.0094	696 B
master	`StartFinishScope`	net472	817ns	0.954ns	3.7ns	0.104	658 B
#6545	`StartFinishSpan`	net6.0	398ns	0.145ns	0.563ns	0.008	576 B
#6545	`StartFinishSpan`	netcoreapp3.1	628ns	0.532ns	1.84ns	0.00768	576 B
#6545	`StartFinishSpan`	net472	650ns	0.451ns	1.75ns	0.0915	578 B
#6545	`StartFinishScope`	net6.0	568ns	0.253ns	0.981ns	0.00962	696 B
#6545	`StartFinishScope`	netcoreapp3.1	745ns	0.947ns	3.67ns	0.00922	696 B
#6545	`StartFinishScope`	net472	974ns	0.589ns	2.28ns	0.105	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	609ns	0.354ns	1.37ns	0.00974	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	970ns	0.624ns	2.34ns	0.00962	696 B
master	`RunOnMethodBegin`	net472	1.12μs	0.636ns	2.46ns	0.104	658 B
#6545	`RunOnMethodBegin`	net6.0	656ns	0.369ns	1.38ns	0.0097	696 B
#6545	`RunOnMethodBegin`	netcoreapp3.1	935ns	0.556ns	2.15ns	0.00894	696 B
#6545	`RunOnMethodBegin`	net472	1.16μs	0.82ns	3.18ns	0.104	658 B

[CI Visibility] Improve Git commands execution

6c2d37a

github-actions bot added the area:ci-visibility label Jan 13, 2025

fix

5c3e9c5

tonyredondo marked this pull request as ready for review January 14, 2025 10:00

tonyredondo requested review from a team as code owners January 14, 2025 10:00

andrewlock approved these changes Jan 14, 2025

View reviewed changes

tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs Show resolved Hide resolved

tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs Show resolved Hide resolved

Force ExitCode = 1 if TimeOut == true

8a0c3a7

tonyredondo mentioned this pull request Jan 14, 2025

[CI Visibility] Add 5min timeout to Git commands #6541

Closed

tonyredondo added 2 commits January 14, 2025 13:40

changes

b8199e9

Merge branch 'master' into tony/civisibility-improve-git-commands

64763ad

kevingosse reviewed Jan 14, 2025

View reviewed changes

tracer/src/Datadog.Trace/Util/ProcessHelpers.cs Outdated Show resolved Hide resolved

kevingosse approved these changes Jan 14, 2025

View reviewed changes

Changes from review

ea8797f

tonyredondo merged commit 35b0d11 into master Jan 14, 2025
99 of 103 checks passed

tonyredondo deleted the tony/civisibility-improve-git-commands branch January 14, 2025 21:37

github-actions bot added this to the vNext-v3 milestone Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI Visibility] Improve Git commands execution #6545

[CI Visibility] Improve Git commands execution #6545

tonyredondo commented Jan 13, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025 •

edited

Loading

andrewlock left a comment

andrewlock commented Jan 14, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6545

Raw results

Raw results

[CI Visibility] Improve Git commands execution #6545

[CI Visibility] Improve Git commands execution #6545

Conversation

tonyredondo commented Jan 13, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Jan 13, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 13, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock commented Jan 14, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6545

Raw results

Raw results

tonyredondo commented Jan 13, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025 •

edited

Loading