Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add publishing to the dotnetbuilds storage account alongside dotnetcli #7877

Merged
merged 6 commits into from
Oct 4, 2021
Merged

Add publishing to the dotnetbuilds storage account alongside dotnetcli #7877

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bf41bf0
Add publishing to the dotnetbuilds storage account alongside dotnetcli
alexperovich Sep 9, 2021
6ba4be3
PR feedback
alexperovich Sep 10, 2021
1dfe763
More PR feedback
alexperovich Sep 13, 2021
7fa8f1b
Merge branch 'main' into publishToDotNetBuilds
alexperovich Sep 14, 2021
d073ef7
Refactor Channel Configs based on feedback
alexperovich Sep 24, 2021
060c3d2
Put the base64 back in the right place
alexperovich Sep 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 19 additions & 16 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/PublishArtifactsInManifest.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,22 @@
<StreamingPublishingMaxClients Condition="'$(UseStreamingPublishing)' == 'true' and '$(StreamingPublishingMaxClients)' == ''">16</StreamingPublishingMaxClients>
</PropertyGroup>

<ItemGroup>
<FeedKey Include="https://dotnetclichecksums.blob.core.windows.net/dotnet/index.json" Key="$(ChecksumsFeedKey)"/>
<FeedKey Include="https://dotnetcli.blob.core.windows.net/dotnet/index.json" Key="$(InstallersFeedKey)"/>
<FeedKey Include="https://dotnetclichecksumsmsrc.blob.core.windows.net/dotnet/index.json" Key="$(InternalChecksumsFeedKey)"/>
<FeedKey Include="https://dotnetclimsrc.blob.core.windows.net/dotnet/index.json" Key="$(InternalInstallersFeedKey)"/>
<FeedKey Include="https://pkgs.dev.azure.com/dnceng" Key="$(AzureDevOpsFeedsKey)"/>
<FeedSasUri Include="https://dotnetbuilds.blob.core.windows.net/public" Base64Uri="$(DotNetBuildsPublicUriBase64)"/>
<FeedSasUri Include="https://dotnetbuilds.blob.core.windows.net/public-checksums" Base64Uri="$(DotNetBuildsPublicChecksumsUriBase64)"/>
<FeedSasUri Include="https://dotnetbuilds.blob.core.windows.net/internal" Base64Uri="$(DotNetBuildsInternalUriBase64)"/>
<FeedSasUri Include="https://dotnetbuilds.blob.core.windows.net/internal-checksums" Base64Uri="$(DotNetBuildsInternalChecksumsUriBase64)"/>
<FeedOverride Include="https://dotnetclichecksums.blob.core.windows.net/dotnet/index.json" Replacement="$(ChecksumsFeedOverride)" Condition="'$(ChecksumsFeedOverride)' != ''"/>
michellemcdaniel marked this conversation as resolved.
Show resolved Hide resolved
<FeedOverride Include="https://dotnetcli.blob.core.windows.net/dotnet/index.json" Replacement="$(InstallersFeedOverride)" Condition="'$(InstallersFeedOverride)' != ''"/>
<FeedOverride Include="transport-packages" Replacement="$(TransportFeedOverride)" Condition="'$(TransportFeedOverride))' != ''"/>
<FeedOverride Include="shipping-packages" Replacement="$(ShippingFeedOverride)" Condition="'$(ShippingFeedOverride)' != ''"/>
</ItemGroup>

<Error
Condition="'@(ManifestFiles)' == ''"
Text="No manifest file was found in the provided path: $(ManifestsBasePath)" />
Expand All @@ -128,12 +144,6 @@
MaestroApiEndpoint="$(MaestroApiEndpoint)"
BuildAssetRegistryToken="$(BuildAssetRegistryToken)"
PublishInstallersAndChecksums="$(PublishInstallersAndChecksums)"
ChecksumsFeedKey="$(ChecksumsFeedKey)"
InstallersFeedKey="$(InstallersFeedKey)"
InternalChecksumsFeedKey="$(InternalChecksumsFeedKey)"
InternalInstallersFeedKey="$(InternalInstallersFeedKey)"
AzureDevOpsFeedsKey="$(AzureDevOpsFeedsKey)"
AzureStorageTargetFeedKey="$(AzureStorageTargetFeedPAT)"
AssetManifestPaths="@(ManifestFiles)"
BlobAssetsBasePath="$(BlobBasePath)"
PackageAssetsBasePath="$(PackageBasePath)"
Expand All @@ -151,12 +161,6 @@
PublishSpecialClrFiles="$(PublishSpecialClrFiles)"
BuildQuality="$(BuildQuality)"
AllowFeedOverrides="$(AllowFeedOverrides)"
InstallersFeedOverride="$(InstallersFeedOverride)"
ChecksumsFeedOverride="$(ChecksumsFeedOverride)"
TransportFeedOverride="$(TransportFeedOverride)"
ShippingFeedOverride="$(ShippingFeedOverride)"
SymbolsFeedOverride="$(SymbolsFeedOverride)"
PublicSymbolsFeedOverride="$(PublicSymbolsFeedOverride)"
AzdoApiToken="$(AzdoApiToken)"
ArtifactsBasePath="$(ArtifactsBasePath)"
BuildId="$(BuildId)"
Expand All @@ -165,10 +169,9 @@
UseStreamingPublishing="$(UseStreamingPublishing)"
StreamingPublishingMaxClients="$(StreamingPublishingMaxClients)"
NonStreamingPublishingMaxClients="$(NonStreamingPublishingMaxClients)"
michellemcdaniel marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the feed overrides are present, we should write a test that ensures that an asset is only pushed to a target endpoint once. In the new configuration here, there are multiple target feed configs for an installer blob, for instance. That might cause a bad interaction when the target is overridden to be a specific feed.

DotNetBuildsPublicUriBase64="$(DotNetBuildsPublicUriBase64)"
DotNetBuildsPublicChecksumsUriBase64="$(DotNetBuildsPublicChecksumsUriBase64)"
DotNetBuildsInternalUriBase64="$(DotNetBuildsInternalUriBase64)"
DotNetBuildsInternalChecksumsUriBase64="$(DotNetBuildsInternalChecksumsUriBase64)"
FeedKeys="@(FeedKey)"
FeedSasUris="@(FeedSasUri)"
FeedOverrides="@(FeedOverride)"
/>
</Target>

Expand Down
14 changes: 8 additions & 6 deletions src/Microsoft.DotNet.Build.Tasks.Feed.Tests/GeneralTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

using System;
using System.IO;
using System.Net;
using System.Net.Http;
Expand All @@ -24,12 +25,13 @@ public void ChannelConfigsHaveAllConfigs()
foreach (var channelConfig in PublishingConstants.ChannelInfos)
{
channelConfig.Id.Should().BeGreaterThan(0);
channelConfig.ShippingFeed.Should().NotBeNullOrEmpty();
channelConfig.ShippingFeed.Should().NotBeNullOrEmpty();
channelConfig.TransportFeed.Should().NotBeNullOrEmpty();
channelConfig.SymbolsFeed.Should().NotBeNullOrEmpty();
channelConfig.ChecksumsFeed.Should().NotBeNullOrEmpty();
channelConfig.InstallersFeed.Should().NotBeNullOrEmpty();
channelConfig.TargetFeeds.Should().NotBeEmpty();
foreach (TargetFeedContentType type in Enum.GetValues(typeof(TargetFeedContentType)))
{
if (type == TargetFeedContentType.None)
continue;
channelConfig.TargetFeeds.Should().Contain(f => f.ContentTypes.Contains(type));
}
}
}

Expand Down
27 changes: 0 additions & 27 deletions src/Microsoft.DotNet.Build.Tasks.Feed.Tests/PublishArtifactsInManifestTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,33 +28,6 @@ public class PublishArtifactsInManifestTests
private const string RandomToken = "abcd";
private const string BlobFeedUrl = "https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json";

// This test should be refactored: https://github.com/dotnet/arcade/issues/6715
[Fact]
public void ConstructV2PublishingTask()
{
var manifestFullPath = TestInputs.GetFullPath(Path.Combine("Manifests", "SampleV2.xml"));

var buildEngine = new MockBuildEngine();
var task = new PublishArtifactsInManifest()
{
BuildEngine = buildEngine,
TargetChannels = GeneralTestingChannelId
};

// Dependency Injection setup
var collection = new ServiceCollection()
.AddSingleton<IFileSystem, FileSystem>()
.AddSingleton<IBuildModelFactory, BuildModelFactory>();
task.ConfigureServices(collection);
using var provider = collection.BuildServiceProvider();

// Act and Assert
task.InvokeExecute(provider);

var which = task.WhichPublishingTask(manifestFullPath);
which.Should().BeOfType<PublishArtifactsInManifestV2>();
}

// This test should be refactored: https://github.com/dotnet/arcade/issues/6715
[Fact]
public void ConstructV3PublishingTask()
Expand Down
Loading