[CI Visibility] Add more checks for the object pack files #6256

tonyredondo · 2024-11-07T17:35:10Z

Summary of changes

This PR adds more checks to the object pack files uploader.

Reason for change

Still receiving this kind of errors:

2024-11-07 16:15:11.381 +00:00 [WRN] ITR: 'git pack-objects...' returned a cross-device error, retrying using a local temporal folder.  { MachineName: ".", Process: "[4026 dotnet]", AppDomain: "[1 datacollector]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.6.0.0" }
2024-11-07 16:15:11.524 +00:00 [INF] ITR: Sending /project/tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/bin/Release/net6.0/.git_tmp/tmplEpfnp.tmp-215f1b02d59b2b9ac7f5468767913cdcbaade353.pack  { MachineName: ".", Process: "[4026 dotnet]", AppDomain: "[1 datacollector]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.6.0.0" }
2024-11-07 16:15:13.144 +00:00 [ERR] ITR: An error occurred while sending intelligent test runner data after 5 retries. System.IO.DirectoryNotFoundException: Could not find a part of the path '/project/tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/bin/Release/net6.0/.git_tmp/tmplEpfnp.tmp-215f1b02d59b2b9ac7f5468767913cdcbaade353.pack'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.File.Open(String path, FileMode mode, FileAccess access, FileShare share)
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.<>c__DisplayClass32_0.<<SendObjectsPackFileAsync>g__InternalSendObjectsPackFileAsync|0>d.MoveNext() in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 646
--- End of stack trace from previous location ---
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.WithRetries[T,TState](Func`3 sendDelegate, TState state, Int32 numOfRetries) in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 1074
 { MachineName: ".", Process: "[4026 dotnet]", AppDomain: "[1 datacollector]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.6.0.0" }
2024-11-07 16:15:13.152 +00:00 [ERR] Error detecting and reconfiguring git repository for shallow clone. System.IO.DirectoryNotFoundException: Could not find a part of the path '/project/tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/bin/Release/net6.0/.git_tmp/tmplEpfnp.tmp-215f1b02d59b2b9ac7f5468767913cdcbaade353.pack'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.File.Open(String path, FileMode mode, FileAccess access, FileShare share)
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.<>c__DisplayClass32_0.<<SendObjectsPackFileAsync>g__InternalSendObjectsPackFileAsync|0>d.MoveNext() in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 646
--- End of stack trace from previous location ---
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.WithRetries[T,TState](Func`3 sendDelegate, TState state, Int32 numOfRetries) in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 1074
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.WithRetries[T,TState](Func`3 sendDelegate, TState state, Int32 numOfRetries) in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 1090
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.SendObjectsPackFileAsync(String commitSha, String[] commitsToInclude, String[] commitsToExclude) in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 605
   at Datadog.Trace.Ci.IntelligentTestRunnerClient.UploadRepositoryChangesAsync() in /project/tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs:line 236
 { MachineName: ".", Process: "[4026 dotnet]", AppDomain: "[1 datacollector]", AssemblyLoadContext: "\"\" Datadog.Trace.ClrProfiler.Managed.Loader.ManagedProfilerAssemblyLoadContext #1", TracerVersion: "3.6.0.0" }

Other details

datadog-ddstaging · 2024-11-07T17:59:36Z

Datadog Report

Branch report: tony/civisibility-more-objectpack-file-checks
Commit report: e9f43f5
Test service: dd-trace-dotnet

✅ 0 Failed, 370940 Passed, 2098 Skipped, 14h 54m 45.43s Total Time
⌛ 2 Performance Regressions

⌛ Performance Regressions vs Default Branch (2)

CallTarget+Inlining+NGEN - Samples.FakeDbCommand.windows.net462.json.scenarios 1.18s (+63.83ms, +6%) - Details
Baseline - Samples.FakeDbCommand.windows.net462.json.scenarios 78.87ms (+8.39ms, +12%) - Details

andrewlock

So weird 😅

andrewlock · 2024-11-07T18:06:44Z

tracer/src/Datadog.Trace/Ci/IntelligentTestRunnerClient.cs

@@ -592,7 +592,7 @@ public async Task<long> SendObjectsPackFileAsync(string commitSha, string[]? com
        long totalUploadSize = 0;
        foreach (var packFile in packFilesObject.Files)
        {
-            if (!File.Exists(packFile))
+            if (!Directory.Exists(Path.GetDirectoryName(packFile)) || !File.Exists(packFile))


Does this actually do anything extra? 🤔

To be honest I don't think so, but I mean... I'm already doing a File.Exist but for some reason later is throwing DirectoryNotFoundException, so I don't know if is something weird on debian+arm64 pal implementation or what. Adding this doesn't hurt but I'm with you, I don't think it will be relevant 🤷🏻

andrewlock · 2024-11-07T18:12:32Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (79ms)  : 65, 93
     .   : milestone, 79,
    master - mean (70ms)  : 68, 71
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (1,180ms)  : 1015, 1345
     .   : milestone, 1180,
    master - mean (1,112ms)  : 1092, 1131
     .   : milestone, 1112,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (109ms)  : 107, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (772ms)  : 760, 783
     .   : milestone, 772,
    master - mean (773ms)  : 763, 784
     .   : milestone, 773,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (92ms)  : 90, 95
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (729ms)  : 714, 744
     .   : milestone, 729,
    master - mean (731ms)  : 713, 749
     .   : milestone, 731,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (190ms)  : 188, 193
     .   : milestone, 190,
    master - mean (190ms)  : 187, 193
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (1,221ms)  : 1200, 1243
     .   : milestone, 1221,
    master - mean (1,222ms)  : 1199, 1246
     .   : milestone, 1222,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (277ms)  : 271, 283
     .   : milestone, 277,
    master - mean (277ms)  : 272, 282
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (949ms)  : 930, 968
     .   : milestone, 949,
    master - mean (950ms)  : 929, 970
     .   : milestone, 950,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6256) - mean (265ms)  : 260, 270
     .   : milestone, 265,
    master - mean (265ms)  : 260, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6256) - mean (933ms)  : 914, 951
     .   : milestone, 933,
    master - mean (934ms)  : 910, 958
     .   : milestone, 934,

andrewlock · 2024-11-07T19:45:19Z

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6256) (11.134M)   : 0, 11134220
    master (11.131M)   : 0, 11130779
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (6256) (7.232M)   : 0, 7231548
    master (7.310M)   : 0, 7310157
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.599M)   : 0, 7598556

    section Manual
    master (11.073M)   : 0, 11072548

    section Manual + Automatic
    This PR (6256) (6.715M)   : 0, 6715186
    master (6.775M)   : 0, 6775349

    section DD_TRACE_ENABLED=0
    master (10.251M)   : 0, 10251396

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6256) (9.687M)   : 0, 9686964
    master (9.713M)   : 0, 9712554
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (6256) (6.401M)   : 0, 6401043
    master (6.514M)   : 0, 6513986

    section Trace stats
    master (6.710M)   : 0, 6710317

    section Manual
    master (9.670M)   : 0, 9669749

    section Manual + Automatic
    This PR (6256) (5.994M)   : 0, 5993981
    master (6.015M)   : 0, 6015065

    section DD_TRACE_ENABLED=0
    master (8.772M)   : 0, 8772254

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6256) (9.502M)   : 0, 9501964
    master (10.153M)   : 0, 10152752
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (6256) (6.147M)   : 0, 6147461
    master (6.377M)   : 0, 6376641
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.126M)   : 0, 7126124

    section Manual
    master (10.105M)   : 0, 10104968

    section Manual + Automatic
    This PR (6256) (5.922M)   : 0, 5921932
    master (5.901M)   : 0, 5901347

    section DD_TRACE_ENABLED=0
    master (9.225M)   : 0, 9225046

[CI Visibility] Add more checks for the object pack files

e9f43f5

tonyredondo requested review from a team as code owners November 7, 2024 17:35

github-actions bot added the area:ci-visibility label Nov 7, 2024

andrewlock approved these changes Nov 7, 2024

View reviewed changes

tonyredondo merged commit a509060 into master Nov 7, 2024
75 of 77 checks passed

tonyredondo deleted the tony/civisibility-more-objectpack-file-checks branch November 7, 2024 21:38

github-actions bot added this to the vNext-v3 milestone Nov 7, 2024

andrewlock added the identified-by:telemetry label Nov 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI Visibility] Add more checks for the object pack files #6256

[CI Visibility] Add more checks for the object pack files #6256

tonyredondo commented Nov 7, 2024

datadog-ddstaging bot commented Nov 7, 2024 •

edited

Loading

andrewlock left a comment

andrewlock Nov 7, 2024

tonyredondo Nov 7, 2024

andrewlock commented Nov 7, 2024

andrewlock commented Nov 7, 2024

[CI Visibility] Add more checks for the object pack files #6256

[CI Visibility] Add more checks for the object pack files #6256

Conversation

tonyredondo commented Nov 7, 2024

Summary of changes

Reason for change

Other details

datadog-ddstaging bot commented Nov 7, 2024 • edited Loading

Datadog Report

⌛ Performance Regressions vs Default Branch (2)

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock Nov 7, 2024

Choose a reason for hiding this comment

tonyredondo Nov 7, 2024

Choose a reason for hiding this comment

andrewlock commented Nov 7, 2024

Execution-Time Benchmarks Report ⏱️

andrewlock commented Nov 7, 2024

Throughput/Crank Report ⚡

datadog-ddstaging bot commented Nov 7, 2024 •

edited

Loading