Add helix-matrix.yml and windows arm 64 helix queue #22002

HaoK · 2020-05-19T21:54:58Z

Initial clone of the daily helix jobs to a new yaml for a new azdo pipeline

HaoK · 2020-05-20T01:51:56Z

This shouldn't affect anything for the ci builds, but I need to have a yml in master to play with splitting the additional helix queues into a separate pipeline

Pilchie · 2020-05-20T02:59:37Z

👀

dougbu

Have you added a pipeline using this YAML?

.azure/pipelines/helix-matrix.yml

HaoK · 2020-05-20T04:18:02Z

So as far as I can tell, I can only create a pipeline from the UI or specify a yaml file that's already in the repo, I guess I can try doing this directly in the azdo UI first with the contents of this PR

HaoK · 2020-05-20T04:21:57Z

Ah okay never mind, looks like I can add a pipeline against this branch, i'll convert this to draft then and remark it as ready for review once its got everything passing

dougbu · 2020-05-20T04:45:51Z

I'm not sure, but if that's the case why do we run on internal today?

It's the same pipeline as the official (internal) builds today.

dougbu · 2020-05-20T04:47:29Z

Forgot to ask: Shouldn't this PR remove PR / rolling build differences in ci.yml?

HaoK · 2020-05-20T04:59:52Z

Yeah once the new pipeline is working I'll update the ci yml

HaoK · 2020-05-20T08:05:51Z

@dougbu have you ever seen any issues with the submodules not updating properly? The new pipeline looks like its failing to build due to the submodules missing in sources for messagePack and google? The helix jobs are passing fine in the existing pipelines in this branch too, so I'm puzzled, is there a hidden pipelines configuration UI/settings page that has extra stuff I need to copy over to the new one?

##[warning].packages/microsoft.build.tasks.git/1.1.0-beta-20206-02/build/Microsoft.Build.Tasks.Git.targets(24,5): warning : (NETCORE_ENGINEERING_TELEMETRY=Build) Could not find file '/home/vsts/work/1/s/src/submodules/googletest/.git'. The source code won't be available via Source Link.

HaoK · 2020-05-20T08:19:27Z

Alright, for posterity, when setting up a new pipeline, there's additional stuff that needs to be configured in the 'high discover-able' buried settings page found via edit -> triggers -> YAML -> Get Sources (our non default settings are: clean = true, all build directories, checkout submodules, any nested submodules within)

.azure/pipelines/helix-matrix.yml

HaoK · 2020-05-21T05:05:07Z

Everything fails miserably on the windows 10 arm64 queue, we are building on linux arm64 though, so I'll try switching to a windows agent to see if that helps

HaoK · 2020-05-21T05:10:28Z

Weird, errors were all in restore, so might be something else:

C:\dotnetbuild\work\A8520933\w\9E7608B5\e\RunTests\RunTests.csproj : error NU1102: Unable to find package Microsoft.WindowsDesktop.App.Ref with version (= 5.0.0-preview.5.20253.1)
C:\dotnetbuild\work\A8520933\w\9E7608B5\e\RunTests\RunTests.csproj : error NU1102:   - Found 17 version(s) in nuget.org [ Nearest version: 5.0.0-preview.4.20251.1 ]
  Failed to restore C:\dotnetbuild\work\A8520933\w\9E7608B5\e\RunTests\RunTests.csproj (in 463 ms).

HaoK · 2020-05-21T10:01:19Z

Progress, looks like most of the windows arm64 tests are passing now, except for node services and libuv kestrel tests (@Tratcher ) do you have any ideas why libuv would fail to load only on windows arm64 and not the Deblian 9 arm64?

System.DllNotFoundException : Unable to load DLL 'libuv' or one of its dependencies: The specified module could not be found. (0x8007007E)

at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.NativeMethods.uv_loop_size()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.loop_size() in /_/src/Servers/Kestrel/Transport.Libuv/src/Internal/Networking/LibuvFunctions.cs:line 339
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvLoopHandle.Init(LibuvFunctions uv) in /_/src/Servers/Kestrel/Transport.Libuv/src/Internal/Networking/UvLoopHandle.cs:line 17
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.ThreadStart(Object parameter) in /_/src/Servers/Kestrel/Transport.Libuv/src/Internal/LibuvThread.cs:line 288
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnectionListener.BindAsync() in /_/src/Servers/Kestrel/Transport.Libuv/src/Internal/LibuvConnectionListener.cs:line 141
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Tests.LibuvTransportTests.CallingAcceptAfterDisposeAsyncThrows() in /_/src/Servers/Kestrel/Transport.Libuv/test/LibuvTransportTests.cs:line 152
--- End of stack trace from previous location ---

Tratcher · 2020-05-21T17:02:17Z

Progress, looks like most of the windows arm64 tests are passing now, except for node services and libuv kestrel tests (@Tratcher ) do you have any ideas why libuv would fail to load only on windows arm64 and not the Deblian 9 arm64?

No. Looking at the package it has support for both. Any thoughts @halter73?

HaoK · 2020-05-21T17:23:01Z

Doesn't look like nodejs is supported on win arm64 yet from nodejs/build#1138

It does look like there's an unofficial build support for versions 12+ from https://unofficial-builds.nodejs.org/download/release/v12.15.0/

But i'll just skip these tests on this queue for now and file an issue

halter73 · 2020-05-21T23:00:19Z

No. Looking at the package it has support for both. Any thoughts @halter73?

I'm not sure either. You're right that we include a win-arm version of libuv.dll in the Libuv NuGet package. It's at runtimes/win-arm/native/libuv.dll which seems to be the right location. I have no idea why the runtime can't find it.

HaoK · 2020-05-21T23:04:45Z

@halter73 are you ok if I just skip these tests on the win-arm64 queue for now and file an issue for these tests?

HaoK · 2020-05-21T23:06:55Z

Could it be related to how we publish/build the helix jobs? We do a single build for arch arm64 and publish to all the helix queues, does this break for libuv since we are using the same publish directory to run tests on both linux arm64 and win arm64?

halter73 · 2020-05-21T23:08:41Z

@halter73 are you ok if I just skip these tests on the win-arm64 queue for now and file an issue for these tests?

I'm OK with that. Are you going to file an issue or should I?

HaoK · 2020-05-21T23:09:49Z

Thanks you can file the issue @halter73 I'll add the skips to this PR

halter73 · 2020-05-21T23:09:51Z

We do a single build for arch arm64 and publish to all the helix queues, does this break for libuv since we are using the same publish directory to run tests on both linux arm64 and win arm64?

I don't think so. Unless it's a self-contained build, I think the publish output should include native assemblies for all RIDs defined in NuGet packages. @sebastienros would know.

halter73 · 2020-05-21T23:18:10Z

I filed the libuv issue.

HaoK · 2020-05-21T23:19:50Z

Thanks, I'll probably add a new csproj attribute to skip these, since its a pain to apply skip to tests enbulk, just updating all Facts to conditional facts is painful :/

HaoK · 2020-05-22T22:09:08Z

Okay looks like the new pipeline is green finally with the new windows arm64 queue. Marking as ready for review, the question about how often to run this pipeline (every checkin rolling or X times a day?). I also need to remove the PR trigger before merging

HaoK · 2020-05-22T22:10:21Z

eng/helix/content/runtests.cmd

@@ -30,8 +30,8 @@ echo "Installing Runtime"
 powershell.exe -NoProfile -ExecutionPolicy unrestricted -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; &([scriptblock]::Create((Invoke-WebRequest -useb 'https://dot.net/v1/dotnet-install.ps1'))) -Architecture %$arch% -Runtime dotnet -Version %$runtimeVersion% -InstallDir %DOTNET_ROOT%"

 set exit_code=0
-echo "Restore: dotnet restore RunTests\RunTests.csproj --source https://api.nuget.org/v3/index.json --ignore-failed-sources..."
-dotnet restore RunTests\RunTests.csproj --source https://api.nuget.org/v3/index.json --ignore-failed-sources
+echo "Restore: dotnet restore RunTests\RunTests.csproj --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json --ignore-failed-sources..."


@BrennanConroy I had to switch feeds for some of the packages to restore properly on the windows arm queue, not sure why?

Link to a failure otherwise I have no context to help

This is an example: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-aspnetcore-refs-pull-22002-merge-c05d1ba331d44ab196/Microsoft.AspNetCore.Blazor.DataAnnotations.Validation.Tests--net5.0/console.34513ba4.log?sv=2019-02-02&se=2020-06-10T05%3A28%3A53Z&sr=c&sp=rl&sig=O8AVhm09qCqFYpe0datZW6vd45YOGPZJ3kw8ZYjWNO0%3D

That is super strange. I'm not against this change, but I would like to know why that package is trying to be resolved :D

.azure/pipelines/helix-matrix.yml

src/Middleware/NodeServices/test/NodeServicesTest.cs

Pilchie · 2020-05-23T16:20:18Z

For frequency I suggest checking with Helix folks and see what they recommend to not overload our available arm64 machine pool.

dougbu · 2020-05-23T18:04:32Z

@HaoK I moved the pipeline into the \dotnet\aspnetcore folder in AzDO, together w/ our other public pipelines.

- all work now done in "Checkout dotnet/aspnetcore" step - reconfigured pipeline based on #22002 (comment)

dougbu · 2020-05-24T03:58:10Z

.azure/pipelines/helix-matrix.yml

+      displayName: Build shared fx
+    - script: .\restore.cmd -ci /p:BuildInteropProjects=true
+      displayName: Restore interop projects
+    - script: .\build.cmd -ci -nobl -NoRestore -test -noBuildJava -all -projects eng\helix\helix.proj /p:IsHelixDaily=true /p:IsRequiredCheck=true /p:IsHelixJob=true /p:BuildInteropProjects=true /p:RunTemplateTests=true /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log


I may have missed an earlier discussion. Why is -noBuildJava needed here?

Good question, looks like we have a difference between the quarantined tests and what was in the normal ci.yml

I copied this from the quarantined test yml which had that...

https://github.com/dotnet/aspnetcore/blob/master/.azure/pipelines/quarantined-tests.yml#L38

@BrennanConroy do we need to build java for the signalr tests?

@BrennanConroy do we need to build java for the signalr tests?

I believe so. Also, is this pipeline running anywhere? I don't see a run of this.

Should be here, https://dev.azure.com/dnceng/public/_build?definitionId=837&_a=summary

I'll add back the java flags on those pipelines

Ah new pipeline name, that's why I didn't see it.

dougbu · 2020-05-24T20:39:30Z

On another note, taking the Helix builds out of non-PR (rolling and official) builds sped rolling builds significantly. It didn't affect official builds as much because the Windows x64/x86 build took almost as long as the removed Helix builds. But, rolling builds look better than they have in ages

dougbu · 2020-05-24T20:40:37Z

/fyi @mmitche the current longest job in official builds doesn't do any testing.

Pilchie · 2020-05-25T17:31:50Z

Wait looking at this build, it looks like we aren't running any helix tests on PRs and rolling builds anymore. I think we still want to run the most common platforms (say Win10 and Ubuntu 18.04), just not the full matrix, since the chance of a PR breaking a specific platform is low, and there are constraints on machines.

HaoK · 2020-05-25T17:45:16Z

Yeah, the helix tests are run only in PR and in the helix-matrix twice a day.

I can easily tweak things to have the PR checks run on all builds including rolling, I'll open a PR for that now

Pilchie · 2020-05-25T18:06:52Z

Thanks - the idea is for those rolling builds to match the PR builds as closely as possible so that we have an idea of the baseline reliability of PR builds (without having to categorize failures as caused by the PR, or caused by flakiness)

ghost added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label May 19, 2020

HaoK requested a review from a team May 20, 2020 01:51

dougbu reviewed May 20, 2020

View reviewed changes

.azure/pipelines/helix-matrix.yml Outdated Show resolved Hide resolved

.azure/pipelines/helix-matrix.yml Outdated Show resolved Hide resolved

HaoK marked this pull request as draft May 20, 2020 04:22

HaoK force-pushed the helix/winarm branch from 6acd90a to db4285e Compare May 20, 2020 05:42

HaoK commented May 20, 2020

View reviewed changes

.azure/pipelines/helix-matrix.yml Outdated Show resolved Hide resolved

halter73 mentioned this pull request May 21, 2020

Enable Kestrel's libuv tests on the helix Windows arm64 queue #22114

Closed

HaoK force-pushed the helix/winarm branch from aa4241c to e53fbaa Compare May 22, 2020 10:42

HaoK added 4 commits May 22, 2020 12:30

Update Helix.targets

45ad30e

Update Helix.targets

fcd32d2

Update Helix.targets

d7e0fe1

Update helix-matrix.yml

a9c171f

HaoK force-pushed the helix/winarm branch from bd4a373 to a9c171f Compare May 22, 2020 19:30

HaoK marked this pull request as ready for review May 22, 2020 22:09

HaoK requested review from halter73, jkotalik and Tratcher as code owners May 22, 2020 22:09

HaoK changed the title ~~Add helix-matrix.yml~~ Add helix-matrix.yml and windows arm 64 helix queue May 22, 2020

HaoK commented May 22, 2020

View reviewed changes

dougbu reviewed May 23, 2020

View reviewed changes

.azure/pipelines/helix-matrix.yml Outdated Show resolved Hide resolved

.azure/pipelines/helix-matrix.yml Show resolved Hide resolved

src/Middleware/NodeServices/test/NodeServicesTest.cs Show resolved Hide resolved

Switch to cron triggers, add internal vars back

186dc46

dougbu approved these changes May 23, 2020

View reviewed changes

dougbu added a commit that referenced this pull request May 23, 2020

Remove extra git submodule commands

6f82d2f

- all work now done in "Checkout dotnet/aspnetcore" step - reconfigured pipeline based on #22002 (comment)

dougbu mentioned this pull request May 23, 2020

Remove extra git submodule commands #22176

Merged

HaoK merged commit a806ae6 into master May 24, 2020

HaoK deleted the helix/winarm branch May 24, 2020 00:08

dougbu added a commit that referenced this pull request May 24, 2020

Remove extra git submodule commands

0e38046

- all work now done in "Checkout dotnet/aspnetcore" step - reconfigured pipeline based on #22002 (comment)

dougbu reviewed May 24, 2020

View reviewed changes

HaoK mentioned this pull request May 25, 2020

[Helix] Always run required queues on all builds, not just PR #22200

Merged

HaoK mentioned this pull request Jul 10, 2020

Add nuget restore feed as well #23863

Merged

Add helix-matrix.yml and windows arm 64 helix queue #22002

Add helix-matrix.yml and windows arm 64 helix queue #22002

Conversation

HaoK commented May 19, 2020

HaoK commented May 20, 2020

Pilchie commented May 20, 2020

dougbu left a comment

Choose a reason for hiding this comment

HaoK commented May 20, 2020

HaoK commented May 20, 2020

dougbu commented May 20, 2020

dougbu commented May 20, 2020

HaoK commented May 20, 2020

HaoK commented May 20, 2020

HaoK commented May 20, 2020

HaoK commented May 21, 2020

HaoK commented May 21, 2020

HaoK commented May 21, 2020

Tratcher commented May 21, 2020

HaoK commented May 21, 2020

halter73 commented May 21, 2020

HaoK commented May 21, 2020

HaoK commented May 21, 2020

halter73 commented May 21, 2020

HaoK commented May 21, 2020

halter73 commented May 21, 2020 • edited Loading

halter73 commented May 21, 2020

HaoK commented May 21, 2020

HaoK commented May 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pilchie commented May 23, 2020

dougbu commented May 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougbu commented May 24, 2020

dougbu commented May 24, 2020

Pilchie commented May 25, 2020

HaoK commented May 25, 2020

Pilchie commented May 25, 2020

halter73 commented May 21, 2020 •

edited

Loading