CI: Add support in yml for detecting darc dependency changes in `eng/Version.Details.xml` #73435

radical · 2022-08-05T07:09:31Z

Issue

Currently, whenever a darc flow PR is opened, we cannot trigger different jobs based on which
dependency got updated. This is needed because in some cases, like for ILLinker, Emscripten
updates we want to trigger all the wasm jobs. But that might not be needed for some of the other
dependencies.

This causes failures to slip through to main, which gets discovered after the fact as other PR
failures, or rolling build failures, creating extra work for developers.

Solution

This PR identifies the changed dependencies, and emits one azure variable per dependency,
which can then be used in conditions in the yml .
The changed dependency can be checked as dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']
Included this in the evaluate_paths job itself, instead of creating another job
Also, use this for wasm jobs

ghost · 2022-08-05T07:09:47Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	radical
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

ghost · 2022-08-05T07:10:00Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	radical
Assignees:	radical
Labels:	`area-Infrastructure-mono`
Milestone:	-

radical · 2022-08-05T08:11:38Z

@akoeplinger thoughts? The changes right now include an artificial change to eng/Version.Details.xml.

In Evaluate Paths job, two variables get set, one per changed dependency:

eng/pipelines/evaluate-changed-darc-deps.sh --difftarget HEAD^1
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/45925021-bbde-4179-bee2-f8f8b3eda8da.sh
Setting pipeline variable Microsoft_NETCore_Runtime_ICU_Transport=true
Setting pipeline variable System_CommandLine=true
Finishing: Evaluate eng/Version.Details.xml for dependency changes

And I tried accessing the variable in WBT job:

  CommandLineChanged1:
    Parsing expression: <dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']>
    Evaluating: dependencies['evaluate_paths']['outputs']['DarcDependenciesChanged.System_CommandLine']
    Result: 'true'

.. conditioning on that.

ghost · 2022-08-06T00:35:08Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Issue

Currently, whenever a darc flow PR is opened, we cannot trigger different jobs based on which
dependency got updated. This is needed because in some cases, like for ILLinker, Emscripten
updates we want to trigger all the wasm jobs. But that might not be needed for some of the other
dependencies.

This causes failures to slip through to main, which gets discovered after the fact as other PR
failures, or rolling build failures, creating extra work for developers.

Solution

This PR identifies the changed dependencies, and emits one azure variable per dependency,
which can then be used in conditions in the yml .
The changed dependency can be checked as dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']
Included this in the evaluate_paths job itself, instead of creating another job

Author:	radical
Assignees:	radical
Labels:	`area-Infrastructure`
Milestone:	-

ghost · 2022-08-06T00:35:51Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Issue

Currently, whenever a darc flow PR is opened, we cannot trigger different jobs based on which
dependency got updated. This is needed because in some cases, like for ILLinker, Emscripten
updates we want to trigger all the wasm jobs. But that might not be needed for some of the other
dependencies.

This causes failures to slip through to main, which gets discovered after the fact as other PR
failures, or rolling build failures, creating extra work for developers.

Solution

This PR identifies the changed dependencies, and emits one azure variable per dependency,
which can then be used in conditions in the yml .
The changed dependency can be checked as dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']
Included this in the evaluate_paths job itself, instead of creating another job

Author:	radical
Assignees:	radical
Labels:	`area-Infrastructure`, `area-Infrastructure-mono`
Milestone:	-

ghost · 2022-08-06T00:51:22Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Issue

Currently, whenever a darc flow PR is opened, we cannot trigger different jobs based on which
dependency got updated. This is needed because in some cases, like for ILLinker, Emscripten
updates we want to trigger all the wasm jobs. But that might not be needed for some of the other
dependencies.

This causes failures to slip through to main, which gets discovered after the fact as other PR
failures, or rolling build failures, creating extra work for developers.

Solution

This PR identifies the changed dependencies, and emits one azure variable per dependency,
which can then be used in conditions in the yml .
The changed dependency can be checked as dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']
Included this in the evaluate_paths job itself, instead of creating another job

Author:	radical
Assignees:	radical
Labels:	`arch-wasm`, `area-Infrastructure`, `area-Infrastructure-mono`
Milestone:	-

akoeplinger

LGTM but I'd prefer to get someone from the infrastructure team to take a look as well

akoeplinger · 2022-08-08T19:02:06Z

eng/pipelines/common/variables.yml

@@ -34,4 +34,11 @@ variables:
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    value: Debug

+- name: wasmDarcDepsChanged
+  value: $[ or(
+              eq(dependencies.evaluate_paths.outputs['DarcDependenciesChanged.Microsoft_NET_Workload_Emscripten_Manifest-7_0_100'], true),


can we somehow do a string contains check here so we don't need to include the 7_0_100 version number? we'll almost certainly forget to update this here.

yeah, I'll try

eng/pipelines/common/variables.yml

radical · 2022-08-09T00:51:19Z

/azp run runtime, runtime-staging

azure-pipelines · 2022-08-09T00:51:41Z

Azure Pipelines successfully started running 2 pipeline(s).

radical · 2022-08-09T19:15:15Z

Build windows arm64 Release NativeAOT is #69832 .
Libraries Build windows x64 Debug is #73646 .
Libraries Test Run checked coreclr Linux arm64 Debug is #73647
Libraries Test Run release coreclr OSX x64 Debug failed due to network issue in downloading tests
mono llvmfullaot Pri0 Runtime Tests Run Linux x64 release is #69832 .

And all of these are unrelated to this PR.

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Aug 5, 2022

ghost assigned radical Aug 5, 2022

radical added area-Infrastructure-mono and removed area-Infrastructure-libraries labels Aug 5, 2022

radical changed the title ~~Add support in yml for detecting darc dependency changes in eng/Version.Details.xml~~ [WIP] Add support in yml for detecting darc dependency changes in eng/Version.Details.xml Aug 5, 2022

radical added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it labels Aug 5, 2022

This was referenced Aug 5, 2022

Infra improvements for Helix #68176

Closed

GC/API/GC/GetGCMemoryInfo/GetGCMemoryInfo.sh test failing intermittently on CoreCLR Linux ARM32 #73247

Closed

radical force-pushed the darc-deps branch from 7cfe5a1 to 5ce4a9d Compare August 6, 2022 00:27

radical changed the title ~~[WIP] Add support in yml for detecting darc dependency changes in eng/Version.Details.xml~~ CI: Add support in yml for detecting darc dependency changes in eng/Version.Details.xml Aug 6, 2022

CI: Detect dependency changes in eng/Version.Details.xml, and allow

c800698

.. conditioning on that.

radical force-pushed the darc-deps branch from 5ce4a9d to c800698 Compare August 6, 2022 00:33

radical marked this pull request as ready for review August 6, 2022 00:34

radical requested review from lewing and akoeplinger August 6, 2022 00:34

radical added area-Infrastructure and removed NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it area-Infrastructure-mono labels Aug 6, 2022

radical added the area-Infrastructure-mono label Aug 6, 2022

radical requested a review from steveisok August 6, 2022 00:36

[wasm] CI: trigger wasm jobs when certain darc dependencies change

8099d08

radical added the arch-wasm WebAssembly architecture label Aug 6, 2022

akoeplinger approved these changes Aug 8, 2022

View reviewed changes

radical requested a review from safern August 8, 2022 19:06

radical added 2 commits August 8, 2022 15:17

Address review feedback

5dfca77

Merge remote-tracking branch 'origin/main' into darc-deps

2873a92

radical added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 8, 2022

Use dependency outputs in a job, instead of outside

039102e

radical removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 9, 2022

runfoapp bot mentioned this pull request Aug 9, 2022

Methodical_others test JIT/Methodical/Coverage/copy_prop_byref_to_native_int crashing #69832

Open

radical mentioned this pull request Aug 9, 2022

xunit.extensibility.execution 2.4.2-pre.22 - primary signature's signing certificate is not trusted by the trust provider. #73646

Closed

radical merged commit 4211dc4 into dotnet:main Aug 9, 2022

radical deleted the darc-deps branch August 9, 2022 19:16

ghost locked as resolved and limited conversation to collaborators Sep 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Add support in yml for detecting darc dependency changes in `eng/Version.Details.xml` #73435

CI: Add support in yml for detecting darc dependency changes in `eng/Version.Details.xml` #73435

radical commented Aug 5, 2022 •

edited

Loading

ghost commented Aug 5, 2022

ghost commented Aug 5, 2022

radical commented Aug 5, 2022

ghost commented Aug 6, 2022

Issue

Solution

ghost commented Aug 6, 2022

Issue

Solution

ghost commented Aug 6, 2022

Issue

Solution

akoeplinger left a comment

akoeplinger Aug 8, 2022

radical Aug 8, 2022

radical commented Aug 9, 2022

azure-pipelines bot commented Aug 9, 2022

radical commented Aug 9, 2022

CI: Add support in yml for detecting darc dependency changes in eng/Version.Details.xml #73435

CI: Add support in yml for detecting darc dependency changes in eng/Version.Details.xml #73435

Conversation

radical commented Aug 5, 2022 • edited Loading

Issue

Solution

ghost commented Aug 5, 2022

ghost commented Aug 5, 2022

radical commented Aug 5, 2022

ghost commented Aug 6, 2022

Issue

Solution

ghost commented Aug 6, 2022

Issue

Solution

ghost commented Aug 6, 2022

Issue

Solution

akoeplinger left a comment

Choose a reason for hiding this comment

akoeplinger Aug 8, 2022

Choose a reason for hiding this comment

radical Aug 8, 2022

Choose a reason for hiding this comment

radical commented Aug 9, 2022

azure-pipelines bot commented Aug 9, 2022

radical commented Aug 9, 2022

CI: Add support in yml for detecting darc dependency changes in `eng/Version.Details.xml` #73435

CI: Add support in yml for detecting darc dependency changes in `eng/Version.Details.xml` #73435

radical commented Aug 5, 2022 •

edited

Loading