Run libraries package testing on build agent #53905

ViktorHofer · 2021-06-08T23:23:04Z

instead of on Helix as with recent changes the entire package testing
doesn't take more than 2 minutes. Helix created a work item per package
test even though it only took seconds and the average wait time for
a client was 10-15min. This should save some machine cycles as we don't
depend on Helix clients anymore.

Also cleaning up how the generated package test projects are restored
and incorporating a fix from Eric St John to not hard code the package
feeds.

Superseding #53837

instead of on Helix as with recent changes the entire package testing doesn't take more than 2 minutes. Helix created a work item per package test even though it only took seconds and the average wait time for a client was 10-15min. Also cleaning up how the generated package test projects are restored and incorporating a fix from Eric St John to not hard code the package feeds.

ghost · 2021-06-08T23:23:08Z

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

instead of on Helix as with recent changes the entire package testing
doesn't take more than 2 minutes. Helix created a work item per package
test even though it only took seconds and the average wait time for
a client was 10-15min.

Also cleaning up how the generated package test projects are restored
and incorporating a fix from Eric St John to not hard code the package
feeds.

Superseding #53837

Author:	ViktorHofer
Assignees:	ViktorHofer
Labels:	`area-Infrastructure-libraries`
Milestone:	-

src/libraries/pkg/test/testPackages.proj

safern

This looks great! Thanks for working on this, with the current improvements I agree that it no longer makes sense to use helix for this.

Let's cleanup helix-queues-setup as well and remove instances of allConfigurations there:

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Lines 155 to 157 in 3ead4ac

    
           # AllConfigurations 
        
           - ${{ if eq(parameters.jobParameters.framework, 'allConfigurations') }}: 
        
             - Windows.10.Amd64.Server19H1.Open

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Line 131 in 3ead4ac

    
           - ${{ if notIn(parameters.jobParameters.framework, 'allConfigurations', 'net48') }}:

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Line 162 in 3ead4ac

    
           - ${{ if notIn(parameters.jobParameters.framework, 'allConfigurations', 'net48') }}:

safern · 2021-06-09T01:10:06Z

It seems like you need to adjust the build command for the all configurations leg? I don't see the tests running on the build log.

eng/pipelines/libraries/build-job.yml

ViktorHofer · 2021-06-09T19:38:59Z

Wooosh, finally it's working but I still need an approval :)

  Testing closure for System.Windows.Extensions TFM=net6.0 RID=none
  Testing closure for System.Threading.Tasks.Dataflow TFM=netcoreapp2.0 RID=none
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=netcoreapp2.0 RID=none
  Testing closure for System.Windows.Extensions TFM=net6.0 RID=win
  Testing for duplicate types for System.Windows.Extensions TFM=net6.0 RID=none
  Testing for duplicate types for System.Windows.Extensions TFM=net6.0 RID=win
  Testing System.Threading.Tasks.Dataflow runtime on TFM=net462 RIDs=none
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=net462 RID=none
  Testing System.Threading.Tasks.Dataflow TFM=netcoreapp3.0
  Testing System.Threading.Tasks.Dataflow runtime on TFM=net461 RIDs=none
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=net461 RID=none
  Testing System.Threading.Tasks.Dataflow TFM=netstandard2.0
  Testing System.Threading.Tasks.Dataflow TFM=net6.0
  Testing System.Threading.Tasks.Dataflow runtime on TFM=netstandard2.0 RIDs=none
  Testing closure for System.Threading.Tasks.Dataflow TFM=netstandard2.0 RID=none
  Testing System.Threading.Tasks.Dataflow TFM=netstandard2.1
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=netstandard2.0 RID=none
  Testing System.Threading.Tasks.Dataflow runtime on TFM=netcoreapp3.0 RIDs=none
  Testing closure for System.Threading.Tasks.Dataflow TFM=netcoreapp3.0 RID=none
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=netcoreapp3.0 RID=none
  Testing System.Threading.Tasks.Dataflow runtime on TFM=netstandard2.1 RIDs=none
  Testing closure for System.Threading.Tasks.Dataflow TFM=netstandard2.1 RID=none
  Testing for duplicate types for System.Threading.Tasks.Dataflow TFM=netstandard2.1 RID=none

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:20:12.37

safern

LGTM! Thank you for improving this! Love to see a 20 min all configurations build.

ViktorHofer · 2021-06-09T20:23:20Z

Love to see a 20 min all configurations build.

Kudos to @ericstj who contributed a great deal to achieving that goal.

ViktorHofer · 2021-06-09T20:40:46Z

Started 2h 16m 23s ago

I feel like our CI build times are out of control. Waiting more than two hours for libraries tests (normal, non-mobile) to complete isn't great...

ViktorHofer · 2021-06-09T20:54:47Z

Failures are:

[wasm] SendReceive_VaryingLengthBuffers_Success failing in CI #53957
network hiccup when restoring a package

ericstj · 2021-06-09T22:49:27Z

Very cool reduction in total time for this leg (and resource usage):

Before:

After:

ViktorHofer added the area-Infrastructure-libraries label Jun 8, 2021

ViktorHofer requested review from ericstj and safern June 8, 2021 23:23

ViktorHofer self-assigned this Jun 8, 2021

safern reviewed Jun 8, 2021

View reviewed changes

src/libraries/pkg/test/testPackages.proj Outdated Show resolved Hide resolved

safern reviewed Jun 8, 2021

View reviewed changes

ViktorHofer added 3 commits June 9, 2021 01:35

Remove allconfigurations conditions from helix ymls

0ee945a

Enable single package testing

a303aa4

Avoid global properties - cleanup

17ebcbc

ViktorHofer added 3 commits June 9, 2021 10:44

Pass down test build action for allconfig

e5e07f8

Expose useHelix argument

9ae7387

Fixesssss

107f9dc

ViktorHofer mentioned this pull request Jun 9, 2021

Use NuGet.config for restore during package testing #53837

Closed

Add missing package

4011147

safern reviewed Jun 9, 2021

View reviewed changes

eng/pipelines/libraries/build-job.yml Show resolved Hide resolved

runfoapp bot mentioned this pull request Jun 9, 2021

system.net.primitives.unit.tests.cookiecontainertest.setcookies_success #53952

Closed

safern approved these changes Jun 9, 2021

View reviewed changes

ViktorHofer merged commit eb9a5ab into dotnet:main Jun 9, 2021

ViktorHofer deleted the PackageTestingLocal branch June 9, 2021 20:55

ViktorHofer mentioned this pull request Jun 14, 2021

Package testing shouldn't hard code restore sources #53810

Closed

karelz mentioned this pull request Jun 17, 2021

[wasm] SendReceive_VaryingLengthBuffers_Success failing in CI #53957

Closed

ghost locked as resolved and limited conversation to collaborators Jul 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run libraries package testing on build agent #53905

Run libraries package testing on build agent #53905

ViktorHofer commented Jun 8, 2021 •

edited

Loading

ghost commented Jun 8, 2021

safern left a comment

safern commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

safern left a comment

ViktorHofer commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

ericstj commented Jun 9, 2021

	# AllConfigurations
	- ${{ if eq(parameters.jobParameters.framework, 'allConfigurations') }}:
	- Windows.10.Amd64.Server19H1.Open

Run libraries package testing on build agent #53905

Run libraries package testing on build agent #53905

Conversation

ViktorHofer commented Jun 8, 2021 • edited Loading

ghost commented Jun 8, 2021

safern left a comment

Choose a reason for hiding this comment

safern commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

safern left a comment

Choose a reason for hiding this comment

ViktorHofer commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

ViktorHofer commented Jun 9, 2021

ericstj commented Jun 9, 2021

ViktorHofer commented Jun 8, 2021 •

edited

Loading