Improve publishing performance (by a lot) #5096
Conversation
Improve the publishing performance by doing a number of optimizations - Pass all manifests to the publish task at once and read them in bulk. The publishing machines have two cores, so repos with lots of manifests (e.g. core-sdk and aspnetcore) were seeing less benefit from parallelism in the publish task - Increase the max upload parallelism to 16 - Change the overall publish process to use async/await much more heavily. - Remove publishes of symbol packages to azdo feeds for public non-stable builds. It's possible this could be extended to stable/internal builds with some additional work. These still go to blob storage. - Remove publishes of installers/checksums to the static azdo blob storage (dotnet-core) if publishing is already sending these the installer/checksum locations (dotnetcli, etc.) - Remove publishes to dotnet-core for packages. After these changes, the future critical path publishing time (runtime, aspnetcore, sdk, and core-sdk) goes from ~130 mins to ~22 mins. About 40-50% of that appears to be prep time for publishing (downloading artifacts, etc.) - Runtime (99th) - 35 mins to 11 mins - aspnetcore - 50m to 4m - Sdk - 5.75m to 2m - core-sdk - 33m to 4.5m
|
For the dotnet-core publishing removal here, I'm working my way through places that get arcade's eng - latest to ensure they are referencing the right feeds. |
|
/fyi @jaredpar @Pilchie @dsplaisted |
Does this significantly impact developer workflow for developers wanting to get symbols for a repo that is a dependency to them. |
I don't think so. The symbols are still published, just to blob storage instead of a package feed. I'm not sure anyone has used those anyway. I can check and see what the number of downloads of those packages is. |
|
@Pilchie I checked the download numbers on the symbol feed. It looks like it's only ever been used as a source for the gather-drop tool. |
|
Thanks! |
src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/PublishArtifactsInManifest.proj
Show resolved
Hide resolved
src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs
Outdated
Show resolved
Hide resolved
Improve the publishing performance by doing a number of optimizations
After these changes, the future critical path publishing time (runtime, aspnetcore, sdk, and core-sdk) goes from ~130 mins to ~22 mins. About 40-50% of that appears to be prep time for publishing (downloading artifacts, etc.)