Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Silent workload updates break builds #40222

Open
svaldetero opened this issue Apr 9, 2024 · 9 comments
Open

Silent workload updates break builds #40222

svaldetero opened this issue Apr 9, 2024 · 9 comments

Comments

@svaldetero
Copy link

Description

Another update Tuesday and another broken Azure Pipelines CI build. One build works and then an hour late the build breaks.

I have a step to use a dotnet version 8.0.x. It downloads 8.0.203. Specifically: https://download.visualstudio.microsoft.com/download/pr/87f31249-1415-4edb-87d5-7f16e63414e9/a1ad58c05a131c618ad458a1564784e4/dotnet-sdk-8.0.203-osx-x64.tar.gz.

Next step is to:
dotnet workload install maui-ios maui-android --source https://api.nuget.org/v3/index.json.

The first time it ran, it installed successfully:

Installing workload manifest microsoft.net.sdk.android version 34.0.85...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8043...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8043...
Installing workload manifest microsoft.net.sdk.maui version 8.0.7...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12...
Installing pack Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7...
Writing workload pack installation record for Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7...
Installing pack Microsoft.Maui.Sdk version 8.0.7...

The second time it ran (an hour later), it failed:

Installing workload manifest microsoft.net.sdk.android version 34.0.95...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8043...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8043...
Installing workload manifest microsoft.net.sdk.maui version 8.0.7...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043...
Installing workload manifest microsoft.net.workload.mono.toolchain.current version 8.0.4...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12...
Workload installation failed. Rolling back installed packs...
Installation rollback failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Workload installation failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]

Looking at the nugets:
https://www.nuget.org/packages/Microsoft.NET.Workload.Mono.ToolChain.Current.Manifest-8.0.100/8.0.4 was published between the builds.

HOWEVER, the nuget package that it claims is required:
https://www.nuget.org/packages/Microsoft.NET.Workload.Emscripten.Current.Manifest-8.0.100/8.0.4
DOES NOT EXIST.

I understand these may be separate teams, however these nugets should have been deployed at the same time.

Steps to Reproduce

No response

Link to public reproduction project repository

No response

Version with bug

8.0.7 SR2

Is this a regression from previous behavior?

No, this is something new

Last version that worked well

8.0.7 SR2

Affected platforms

iOS, Android

Affected platform versions

No response

Did you find any workaround?

I changed my workload restore to include :
--from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Relevant log output

No response

@svaldetero
Copy link
Author

Update: After submitting this, the dependent nuget package showed up (~3-4 hours later).

The issue still remains, we need a way to pin a version of the workload (without using the undocumented and hard to find rollback file). Because pinning a dotnet version doesn't stop the problem.

@svaldetero
Copy link
Author

Same type of issue, but different package: dotnet/maui#19934

@Redth
Copy link
Member

Redth commented Apr 9, 2024

@svaldetero please try again. It seems there was a delay in publishing of some packages for this update, which should now be available.

@Redth
Copy link
Member

Redth commented Apr 9, 2024

Related: #23820

@svaldetero
Copy link
Author

@Redth Yes it works now after I removed the rollback option.

As a developer, how do I prevent this in the future? How can I pin a workload version without having to find a hidden rollback file? In that issue you linked, it doesn't happen every time, but it happens frequently enough. Today it just happened to be ~3 hours before all the nugets were available. That suddenly blocks and backlogs ALL of my builds for close to half the day. Realistically, I can't just wait for the delay to catch up because I don't know how long it will be.

@kgouraw
Copy link

kgouraw commented Apr 9, 2024

I experienced the same with "dotnet workload install maui" in my GitHub workflow today. I changed it to "dotnet workload install maui-mobile" and it works for me for now.

@Ghostbird
Copy link

Ghostbird commented Apr 10, 2024

For us, the way we managed to "pin" the MAUI version for Android CI builds is by building in docker, which we need to do anyway for CI. For an example see our repository. Note: We do not use the docker for builds on the developer's own machine, but updating MAUI there takes only a minute when it breaks.
Alas, our iOS builds keep automatically breaking, but that happens from Apple's side too. Their updates often break things too, so in that sense MAUI is not doing any worse.

@MatthewRamsey
Copy link

Problem:
We were experiencing similar issues this week which were the result of an issue with the Android sdk version bump. Specifically, Microsoft.Android.Sdk.Windows.Msi.x64 (34.0.95) which caused malformed content in in our Android http responses.

Solution:
As others have mentioned, we found a workaround by pinning a rollback file for the specific version of the workloads we need. The rollback file below reverted the Android dependencies to 34.0.79 which solved our problem.

dotnet workload install maui --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

dotnet workload install android ios --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Suggestion:
It would be great to have a page with a list of available rollback files. We're content to pin a version while new releases come out so we can control stability of our applications.

@PureWeen PureWeen transferred this issue from dotnet/maui Apr 16, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added Area-Workloads untriaged Request triage from a team member labels Apr 16, 2024
@nagilson nagilson added needs team triage Requires a full team discussion and removed untriaged Request triage from a team member labels Apr 30, 2024
@marcpopMSFT
Copy link
Member

Without functionality in nuget to batch these, the best we have is pushing all of the packs first and then the manifest. Looking forward, we'll have workload versions coming soon that will make this easier as we can ensure all of the packs and manifests are published and then publish a single workload set version package.

@marcpopMSFT marcpopMSFT removed the needs team triage Requires a full team discussion label May 7, 2024
@marcpopMSFT marcpopMSFT added this to the 9.0.1xx milestone May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants