Create workload set insertion process #302

MiYanni · 2025-01-17T23:54:54Z

Summary

The primary goals of this PR are to:

Update the official.yml build to include a process for:
- Downloading the workloads associated with the workload set created by this branch (via Arcade via Version.Details.xml/Versions.props)
- Create the VS drops of the acquired workloads (published to https://vsdrop.microsoft.com)
- Create the appropriate VS insertion PRs to the desired VS branches (both against the 'current' VS branch and 'prior' branches)
Update the public.yml to actually build and be used in PR builds to this repo

If I've done my job well, the result should be a lot of dense YAML pipeline logic and PowerShell scripts. There is a significant amount of 'magic' involved in making this function. I'll walk through the details below.

The Process

To preface, I attempted to reuse as much existing circuitry as possible. This means Azure Pipeline tasks that already exist and Arcade build 'enchantments' (🧙). In doing so, there were a lot of hurdles and I used PowerShell to try to do the least amount of work possible to overcome those hurdles. Let's review the steps that the official.yml build goes through.

The Steps

The official.yml is configured to only run a single Build stage. I've created a template called workload-build.yml that does the entire build process.

Why is workload-build.yml a separate template?

Azure Pipelines does not allow you to hide pipeline parameters. In workload-build.yml, there is a static set of values I needed to be defined as a parameter (not a variable) as those values are required at compile time (variables are not evaluated at compile time). Therefore, the only way to have static, 'hidden' parameter is to encase all the logic in a template, which is why workload-build.yml exists. The parameter in question is workloadDropTypes declared at the top of workload-build.yml.

`workload-build.yml`

The jobs define in this template are within the jobs.yml context from Arcade. This allows the creation of the Publish Assets job automatically, which pushes the build assets to DARC.

When doing an insertion, we first run the download-workloads.ps1 script.

What does download-workloads.ps1 do?

This script downloads the workload NuGet packages via DARC. It requires a GitHub PAT and an AzDO PAT to run the DARC command. This script can be ran locally without those explicit PATs as outlined in the README.md. This script also configured to take a workloadListJson, which specifies which workloads to acquire. It does this via a regex sent to the DARC command. The regex also accounts for components versus pre.components, which is designated when running the pipeline using the usePreComponentsForVSInsertion pipeline parameter.

Next, we the "regular build" via the build.ps1 provided by Arcade. This will build the workload-versions.sln. I will not go over the process of the building the workload set JSON via the Microsoft.NET.Workloads.csproj, as that process has not changed in this PR. The new projects added are Microsoft.NET.Workloads.Vsman.csproj and Microsoft.NET.Workloads.Vsman.vsmanproj.

Why are there 2 projects, a .csproj and a .vsmanproj?

The .csproj only contains one target, BuildVsmanProj. It only runs this target. The .csproj is a bootstrapper to build the .vsmanproj via the create-workload-drops.ps1 script. Before running that script, the BuildVsmanProj target does the local-build version of using the download-workloads.ps1 script (when necessary). After that, it runs create-workload-drops.ps1 which ends up building the .vsmanproj multiple times, once per workload drop acquired from the download-workloads.ps1 script.

What does create-workload-drops.ps1 do?

This is the main script for creating the workload insertion metadata. It:
1. Extracts the workload drop zips.
2. On a loop per drop, uses a regex to parse metadata about the drop out of the drop name.
3. Computes a unique hash for the files in the drop.
4. Reads the version information for the drop out of the .metadata file.
5. Builds the Microsoft.NET.Workloads.Vsman.vsmanproj for the drop, which creates the vsman file for that drop.
6. Sets the Azure Pipelines variables for the drop metadata, required for subsequent processes in the official build pipeline.
7. Updates the strings that represent the component list for doing the VS insertion later in the official build pipeline.
8. After the loop completes, cleans up intermediate build files from the workload drop folders.
9. Sets the Azure Pipeline variables for the component lists, one for the primary (components & packs) and one for the secondary (packs only).
Note: Per loop iteration in the script, pipeline variables are dynamically created based on the workload drop's shortname, drop type, and purpose. For example, for the emsdk components drop, variables named emsdk_components_name, emsdk_components_dir, and emsdk_components_url would be created.

After the regular build if the publishToFeed pipeline parameter is set, the pipeline will publish the workload set to the designated AzDO feed via the feedForPublishing pipeline parameter. Currently, this is only related to the workload set package and does nothing with the workload drops.

Now comes the fun part (😈). When the pipeline is doing an insertion, it will utilize compile-time foreach loops over the parameters workloadDropNames and workloadDropTypes. The types-loop is nested inside of the names-loop. This means for each workload drop, it will run all the following tasks for each workload drop type. For reference, the workload drop types are components, packs, and precomponents. This list of types must be statically defined as a parameter to be used with these compile-time foreach loops.

Within this looping mechanism, there are two tasks. The first task is a small, embedded PowerShell script that simply checks to see if the drop exists for the combination of drop name and drop type. This has to be checked as the drop type components and precomponents won't be downloaded at the same time (they are mutually exclusive). Also, there could be a workload designated in the Version.Details that doesn't have a published workload drop yet, so it wouldn't have been downloaded. This script just validates that the drop folder exists on disk.

The second task in the loop is the 1ES.MicroBuildVstsDrop task. This task uploads the workload drop to the VS drop blob storage for use in creating a VS insertion PR. This task only allows you to upload one drop at a time, which is why this task needs to be looped over in the pipeline. This only runs if the previous PowerShell script set the pipeline variable PublishWorkloadDrop to true.

Outside of those nested loops, the last part of workload-build.yml has 2 loops, one per each set of insertion branches designated via the pipeline parameters primaryVsInsertionBranches and secondaryVsInsertionBranches respectively. These are the branches in the VS repo that the primary (component & packs) and secondary (packs only) will be used for VS insertion PRs. In each loop, the workload-insertion.yml template is called.

`workload-insertion.yml`

This template has two tasks. The first is a small, embedded PowerShell script that sets the InsertionTopicBranch pipeline variable. This variable is calculated based on the value passed in as the topicBranch parameter to this template, which is passed in as the vsTopicBranch pipeline parameter. By default, this value is |default| which resolves to a month-based branch name. Someone could also supply |temp| which creates a branch name using the build number and run attempt for the pipeline (so it always changes). Lastly, if someone supplies any other value, that value is used as the topic branch name.

The second task is MicroBuildInsertVsPayload which is used to create the VS insertion PR. It required creating our own service connection named MicroBuildInsertionVS in the DncEng org to have the proper permissions for this task. This task primarily relies on the component JSON values (supplied via componentJsonValues) to create the VS insertion PR changes. Other than that, every other parameter to the task is fairly self explanatory.

…ipt for publishing workload sets. Just downloading the workload assets at the moment.

…e existing one to test the script. # Conflicts: # eng/pipelines/official.yml

# Conflicts: # eng/pipelines/official.yml

…vertFrom-SecureString when passing to DARC.

…ile. # Conflicts: # global.json # workload-versions.sln

…uild properly. But, that's running into issues too.

…on set in the global.json. # Conflicts: # global.json

# Conflicts: # global.json

…he VSMAN file.

…ontain upstreams.

…cally.

# Conflicts: # eng/common/tools.ps1

# Conflicts: # global.json # Conflicts: # NuGet.config # eng/Version.Details.xml # eng/Versions.props # eng/common/templates-official/job/job.yml # global.json

… VS drop creation. # Conflicts: # eng/pipelines/official.yml

…. This will be renamed in a future commit. Made the official.yml finally able to create the VS drops, but needs tested.

eng/pipelines/templates/jobs/workload-build.yml

eng/check-drop-publish.ps1

README.md

MiYanni · 2025-01-18T00:26:46Z

eng/Publishing.props

-  <ItemGroup>
-    <SdkAssetsToPublish Include="$(ArtifactsShippingPackagesDir)*.msi" />
-    <SdkAssetsToPublish Include="$(ArtifactsNonShippingPackagesDir)*.wixpack.zip" />
-  </ItemGroup>
-


This doesn't do anything. SdkAssetsToPublish is not an item group in this repo/Arcade.

MiYanni · 2025-01-18T00:28:02Z

eng/Signing.props

-  <ItemGroup>
-    <ItemsToSign Include="$(ArtifactsPackagesDir)**\*.msi" />
-    <ItemsToSign Include="$(ArtifactsPackagesDir)**\*.nupkg" />
-  </ItemGroup>
-


These are automatically added via Arcade as part of the publish process. If they remain here, Arcade will actually fail to sign because it'll try to double sign these files.

Forgind

I didn't approve because I don't feel comfortable really reviewing ps1 files, but the parts I looked at looked good 🙂

build.cmd

eng/pipelines/official.yml

…oadIfExists parameter.

Forgind

I tried to review this as thoroughly as I could. Hopefully my questions at least make sense 🙂

Directory.Build.targets

eng/check-drop-publish.ps1

Forgind · 2025-01-27T23:37:50Z

eng/pipelines/official.yml

+  displayName: Primary VS insertion branches
+  type: object
+  default:
+  - rel/d17.12


The preview version is 17.12, and the non-preview version is 17.13?

I have no idea. The primary branches insert components and packs. The secondary only insert packs. If that is related to preview versions... I don't know.

@marcpopMSFT What are the VS branches that are appropriate here?

Shouldn't we be using the insertion client JSON configuration to find the relevant branches for insertions?

I think 17.12 and 17.13 are currently primary. I think 18.0 will be secondary in the workloads 9 branches eventually but that branch doesn't exist yet. 17.14 is still in preview on the VS side so when I insert those workloads in, they go into main and I usually copy from the N-1 branch so we could theoretically include main in addition to 12 and 13 until .14 snaps in april

Forgind · 2025-01-27T23:38:49Z

eng/pipelines/official.yml

+############### ARCADE ###############
+# Both this (used in Arcade for the MicroBuildSigningPlugin) and DotNetSignType (used in Arcade in Sign.proj) are necessary to set the sign type.
+# https://github.com/dotnet/arcade/blob/ccae251ef033746eb0213329953f5e3c1687693b/Documentation/ArcadeSdk.md#common-steps-in-azure-devops-pipeline
+- name: _SignType


Is this only listed under secondaryVsInsertionBranches? Does that mean the primary one isn't signed?

I don't know what you mean by "listed under". It is the pipeline variable used in the build to set the sign type for the build. You can see the usage when we call build.ps1 inside of workload-build.yml.

If I understand it correctly, the indentation here means that this applies to secondaryVsInsertionBranches but not primary...if it applies to the whole pipeline (i.e., both), then I was just wrong

global.json

src/Microsoft.NET.Workloads.Vsman/Microsoft.NET.Workloads.Vsman.vsmanproj

src/Microsoft.NET.Workloads/Microsoft.NET.Workloads.csproj

MiYanni · 2025-01-28T01:20:17Z

eng/VstsDropFolder/plugin.ps1

-# $SkipUploadIfExists = Get-VstsInput -Name 'SkipUploadIfExists'
-# if ($SkipUploadIfExists) {
-if ($true) {
-    # Returns a JSON string. If the drop exists, an array with a single object is returned. If the drop does not exist, an empty array is returned.
-    $dropJson = $server.Client.List($DropName) | ConvertFrom-Json
-    if ($dropJson.Count -ne 0) {
-        Write-Host "Drop '$DropName' has already been published. Skipping VS drop publish..."
-        Write-Telemetry "DropSkipped" "True"
-        Write-TelemetryMetricFinishSeconds "Upload Drop"
-        return
-    }
-}


Note: Tested logic

…script.

… for component insertion.

# Conflicts: # eng/Version.Details.xml # eng/Versions.props # eng/pipelines/official.yml

MiYanni · 2025-01-29T21:31:51Z

eng/create-workload-drops.ps1

-    # $vsComponentValue = "$assemblyName.vsman{$workloadVersion}=$dropUrl,"
-    $vsComponentValue = "$assemblyName.vsman=$dropUrl,"
+    $vsComponentValue = "$assemblyName.vsman{$workloadVersion}=$dropUrl,"


Note: Tested logic for VS Drop access

joeloff · 2025-01-29T21:49:27Z

README.md

+6. Request access to the [.NET Daily Internal Build Access](https://coreidentity.microsoft.com/manage/Entitlement/entitlement/netdailyinte-q2ql) entitlement
+  - This allows the local AzDO machine auth to gather internal assets from AzDO.
+  - ***Message Matt Mitchell*** (@mmitche) for approval after requesting access to the entitlement.


should we publicly list alias and names? Seems like these docs should be internal only. They do not apply to any external user

I would normally just use the public GH handle but I do know he has his full name in his profile currently. That could change in the future.

joeloff · 2025-01-29T21:50:58Z

eng/Versions.props

@@ -56,7 +56,7 @@
  </PropertyGroup>
  <PropertyGroup>
    <WixPackageVersion>3.14.0-8606.20240208.1</WixPackageVersion>
-    <SwixPackageVersion>1.1.87-gba258badda</SwixPackageVersion>
+    <SwixPackageVersion>1.1.392</SwixPackageVersion>


Is CG going to hit us when the version changes? As I recall, this typically comes from the MSENG feed.

If there's a known CVE, I would expect so. Not sure anything else we're supposed to do ahead of time.

joeloff · 2025-01-29T21:53:57Z

eng/Versions.props

@@ -56,7 +56,7 @@
  </PropertyGroup>
  <PropertyGroup>
    <WixPackageVersion>3.14.0-8606.20240208.1</WixPackageVersion>


Should we just use the Arcade version?

yeah, better to use the central one than have to remember to update it every time there's a wix security fix.

joeloff · 2025-01-29T21:54:44Z

eng/create-workload-drops.ps1

+$dropInfoRegex = '^Workload\.VSDrop\.(?<full>(?<short>\w*)\..*?(?<type>(pre\.)?components$|packs$))'
+$primaryVSComponentJsonValues = ''
+$secondaryVSComponentJsonValues = ''
+$hashAlgorithm = [System.Security.Cryptography.SHA256]::Create()


Should this be non crypto hash like xxhash instead?

You would need approval to use SHA256 and SHA1 would get flagged.

joeloff · 2025-01-29T21:59:28Z

eng/Signing.props

-    <ItemsToSign Include="$(ArtifactsPackagesDir)**\*.msi" />
-    <ItemsToSign Include="$(ArtifactsPackagesDir)**\*.nupkg" />
-  </ItemGroup>
-
  <ItemGroup>


This is for signing the workload set MSI, correct?

marcpopMSFT · 2025-01-30T20:57:38Z

README.md

+  - When it asks for a subscription to select, just press Enter. The default subscription selection does not affect DARC.
+4. [Install DARC](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#setting-up-your-darc-client)
+5. [Add GitHub auth for DARC](https://github.com/dotnet/arcade/blob/main/Documentation/Darc.md#step-3-set-additional-pats-for-azure-devops-and-github-operations)


Do you need al of the darc auth details here or just link to arcade (or the details are included when you run darc authenticate)?

marcpopMSFT · 2025-01-30T21:00:49Z

eng/create-workload-drops.ps1

@@ -0,0 +1,78 @@
+param ([Parameter(Mandatory=$true)] [string] $workloadPath, [Parameter(Mandatory=$true)] [string] $msBuildToolsPath)


should there be a comment on what the parameters are for and their format?

marcpopMSFT · 2025-01-30T21:01:20Z

eng/download-workloads.ps1

@@ -0,0 +1,73 @@
+param ([Parameter(Mandatory=$true)] [string] $workloadPath, [SecureString] $gitHubPat, [SecureString] $azDOPat, [string] $workloadListJson = '', [bool] $usePreComponents = $false)


same as above. Comment the parameters and their common format

marcpopMSFT · 2025-01-31T00:13:41Z

eng/pipelines/templates/jobs/workload-build.yml

+          /p:DotNetPublishUsingPipelines=true
+          /p:OfficialBuildId=$(Build.BuildNumber)
+          /p:StabilizePackageVersion=${{ parameters.stabilizePackageVersion }}
+        displayName: 🟣 Build solution


this is the spot we'd add an additional build if we wanted to do the other bands from the same branch. We'd set CreateVSInsertion to false in the later builds and it might just work even though it's in the same agent. But per offline conversation, we need to determine if we're inserting workload sets into VS as that complicates things.

marcpopMSFT · 2025-01-31T00:19:05Z

eng/pipelines/official.yml

+  type: object
+  default:
+  - emsdk
+  - mono


Did you ever end up getting a drop of maui, android, or ios that worked for this testing?

MiYanni added 30 commits January 17, 2025 15:24

Added basic pipeline to do workload set publishing. Added initial scr…

4be4d43

…ipt for publishing workload sets. Just downloading the workload assets at the moment.

Realized I don't want to make a new AzDO pipeline yet. So, reusing th…

cf016b5

…e existing one to test the script. # Conflicts: # eng/pipelines/official.yml

Fix calling the darc command over the version details.

756cfa3

Fixed tools path. Clean up commented script.

46cfed9

Added logic to move the workloads into a drops folder.

465efcd

Clean up commented out scripting. Prep prior to drop creation work.

d7a3a59

# Conflicts: # eng/pipelines/official.yml

Passing in PAT information into the gather-drop script.

be61d02

Forgot to comment out parameters node.

e90c923

Add asset-filter to gather-drop.

e3729ca

Convert the script parameters to secure strings.

464304c

Forgot the additional arguments for ConvertTo-SecureString. Added Con…

2ad818a

…vertFrom-SecureString when passing to DARC.

Consolidate ConvertFrom-SecureString. Fixed --output-dir.

1d8bdb9

Extract the workload packages into the drops directory.

67ce7f3

In-progress determining how to use the Vsman SDK to build the vsman f…

4f7cf15

…ile. # Conflicts: # global.json # workload-versions.sln

Trying to update the SwixBuild version so that the Vsman stuff will b…

78d6343

…uild properly. But, that's running into issues too.

Got the build to work from build.cmd via the xcopy-msbuild tool versi…

24040fe

…on set in the global.json. # Conflicts: # global.json

Progress on making a multi-build vsman target for each drop folder.

6784044

# Conflicts: # global.json

Made each workload use the drop folder as their output for creating t…

de67bea

…he VSMAN file.

Trying parameter use for dynamic variable access.

f4a3333

Temporarily comment out other feeds because those feeds potentially c…

2a9a9fd

…ontain upstreams.

In-progress having the drop acquisition process being able to work lo…

33526f9

…cally.

Continued in-progress of enabling local drop download processing.

0718e95

Got the local build process for VSMAN files working.

7fe5688

# Conflicts: # eng/common/tools.ps1

Merge branch 'release/8.0.4xx' into feature/workload-publish

de46dfa

# Conflicts: # global.json # Conflicts: # NuGet.config # eng/Version.Details.xml # eng/Versions.props # eng/common/templates-official/job/job.yml # global.json

In-progress of getting the workload info spit out to the pipeline for…

11d28fc

… VS drop creation. # Conflicts: # eng/pipelines/official.yml

Moved all the logic for doing the build to the download-workloads.ps1…

5c32157

…. This will be renamed in a future commit. Made the official.yml finally able to create the VS drops, but needs tested.

Using the 1ES task for creating a drop.

1d2d21d

Missed a slash in file path. Updated VSDrop task display name.

463689d

Using plain pats for testing the built.

d722f71

Trying a different image pool for CI.

e1d92ae

MiYanni commented Jan 18, 2025

View reviewed changes

eng/pipelines/templates/jobs/workload-build.yml Outdated Show resolved Hide resolved

MiYanni commented Jan 18, 2025

View reviewed changes

eng/check-drop-publish.ps1 Outdated Show resolved Hide resolved

Updated based on PR creation review.

0fc7204

MiYanni commented Jan 18, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

MiYanni commented Jan 18, 2025

View reviewed changes

Forgind reviewed Jan 23, 2025

View reviewed changes

build.cmd Show resolved Hide resolved

eng/pipelines/official.yml Show resolved Hide resolved

Removed unused testing plugin.ps1 replacements. Using the new skipUpl…

6a4cd67

…oadIfExists parameter.

Forgind approved these changes Jan 27, 2025

View reviewed changes

Trying string 'True' for the moment on skipUploadIfExists.

3932dcc

MiYanni commented Jan 28, 2025

View reviewed changes

MiYanni added 7 commits January 28, 2025 10:58

Debugging skipUploadIfExists issues.

704b647

Added more debugging output.

6f880bb

Update filtering to not use ForEach. This should let return exit the …

3387e00

…script.

Missed commenting out a curly brace.

d58d5ff

See if the new MicroBuild task works now.

aba7d75

Removed debugging commented code and logic. Enabled version injection…

a00e952

… for component insertion.

Merge branch 'main' into feature/workload-publish-main2

573de13

# Conflicts: # eng/Version.Details.xml # eng/Versions.props # eng/pipelines/official.yml

MiYanni commented Jan 29, 2025

View reviewed changes

joeloff reviewed Jan 29, 2025

View reviewed changes

marcpopMSFT reviewed Jan 30, 2025

View reviewed changes

marcpopMSFT reviewed Jan 31, 2025

View reviewed changes

MiYanni requested a review from a team January 31, 2025 00:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create workload set insertion process #302

Create workload set insertion process #302

MiYanni commented Jan 17, 2025 •

edited

Loading

MiYanni Jan 18, 2025

MiYanni Jan 18, 2025

Forgind left a comment

Forgind left a comment

Forgind Jan 27, 2025

MiYanni Jan 28, 2025

MiYanni Jan 29, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 31, 2025

Forgind Jan 27, 2025

MiYanni Jan 28, 2025

Forgind Jan 30, 2025

MiYanni Jan 28, 2025

MiYanni Jan 29, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 30, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 30, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 30, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 30, 2025

joeloff Jan 29, 2025

marcpopMSFT Jan 30, 2025

marcpopMSFT Jan 30, 2025

marcpopMSFT Jan 30, 2025

marcpopMSFT Jan 31, 2025

marcpopMSFT Jan 31, 2025

		@@ -0,0 +1,78 @@
		param ([Parameter(Mandatory=$true)] [string] $workloadPath, [Parameter(Mandatory=$true)] [string] $msBuildToolsPath)

		@@ -0,0 +1,73 @@
		param ([Parameter(Mandatory=$true)] [string] $workloadPath, [SecureString] $gitHubPat, [SecureString] $azDOPat, [string] $workloadListJson = '', [bool] $usePreComponents = $false)

Create workload set insertion process #302

Are you sure you want to change the base?

Create workload set insertion process #302

Conversation

MiYanni commented Jan 17, 2025 • edited Loading

Summary

The Process

The Steps

workload-build.yml

workload-insertion.yml

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Forgind left a comment

Choose a reason for hiding this comment

Forgind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MiYanni commented Jan 17, 2025 •

edited

Loading

`workload-build.yml`

`workload-insertion.yml`