Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rewrite the script using sparse checkout #3392

Closed
wants to merge 2 commits into from
Closed

rewrite the script using sparse checkout #3392

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0d3b502
rewrite the script using sparse checkout
ArcturusZhang May 12, 2023
7eacd4f
update the way we copy the file to avoid format issue
ArcturusZhang May 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 83 additions & 23 deletions eng/DownloadSharedSource.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,89 @@
$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'

function DownloadAll([string[]]$files, [string]$baseUrl, [string]$downloadPath)
{
foreach ($file in $files)
{
Write-Host "Downloading" $file
$text = (Invoke-WebRequest -Uri "$baseUrl/$file").Content
$text.Trim() | Out-File (Join-Path $downloadPath $file)
$sparseCheckoutFile = ".git/info/sparse-checkout"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you copied this from another script but I think you can simplify this to something like:

$repoRoot = "$PSScriptRoot/.."
$clonedPath = "$repoRoot/../core-shared-source"
$azCoreSharedPath = "sdk/core/Azure.Core/src/Shared/"
$armCoreSharedPath = "sdk/resourcemanager/Azure.ResourceManager/src/Shared/"

Write-Host "git clone --no-checkout --filter=tree:0 https://github.com/Azure/azure-sdk-for-net.git $clonedPath"
git clone --no-checkout --filter=tree:0 https://github.com/Azure/azure-sdk-for-net.git $clonedPath
Push-Location $clonedPath
Write-Host "git sparse-checkout init"
git sparse-checkout init
Write-Host "git sparse-checkout set --no-cone $azCoreSharedPath $armCoreSharedPath"
git sparse-checkout set --no-cone $azCoreSharedPath $armCoreSharedPath
Write-Host "git checkout"
git checkout
Pop-Location

$files = @('AsyncLockWithValue.cs', 'ClientDiagnostics.cs', 'DiagnosticScope.cs', 'DiagnosticScopeFactory.cs', 'ContentTypeUtilities.cs', 'HttpMessageSanitizer.cs',
    'OperationInternalBase.cs', 'OperationInternal.cs', 'OperationInternalOfT.cs', 'TaskExtensions.cs', 'Argument.cs', 'Multipart/MultipartFormDataContent.cs',
    'Multipart/MultipartContent.cs', 'AzureKeyCredentialPolicy.cs', 'AppContextSwitchHelper.cs',
    'OperationPoller.cs', 'FixedDelayWithNoJitterStrategy.cs', 'SequentialDelayStrategy.cs',
    'ForwardsClientCallsAttribute.cs', 'AsyncLockWithValue.cs', 'VoidValue.cs')
foreach ($file in $files)
{
   $clonedFilePath = Join-Path $clonedPath $azCoreSharedPath $file
   $assetsFilePath = Join-Path $repoRoot "src/assets/Azure.Core.Shared"
   Copy-Item $clonedFilePath $assetsFilePath 
}

$files = 'SharedExtensions.cs', 'ManagedServiceIdentityTypeV3Converter.cs'
foreach ($file in $files)
{
   $clonedFilePath = Join-Path $clonedPath $armCoreSharedPath $file
   $assetsFilePath = Join-Path $repoRoot "src/assets/Management.Shared"
   Copy-Item $clonedFilePath $assetsFilePath 
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure we want to download outside of the repo root? Also, we should delete the folder if it exists and maybe clean up when we're done:

$clonedPath = "$repoRoot/artifacts/core-shared-source"
if(Test-Path $clonedPath) {
    Remove-Item $clonedPath -Recurse -Force
}

...

Write-Host "Removing temporary source folder '$clonedPath' ..."
Remove-Item $clonedPath -Recurse -Force

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with putting it under a gitignored directory as well.

$gitRemoteUrl = "https://github.com/Azure/azure-sdk-for-net.git"
$gitRemoteBranchName = "main"
$azCoreSharedPath = "sdk/core/Azure.Core/src/Shared/"
$armCoreSharedPath = "sdk/resourcemanager/Azure.ResourceManager/src/Shared/"

function InitializeSparseGitClone([string]$repo) {
git clone --no-checkout --filter=tree:0 $repo .
if ($LASTEXITCODE) { exit $LASTEXITCODE }
git sparse-checkout init
if ($LASTEXITCODE) { exit $LASTEXITCODE }
Remove-Item $sparseCheckoutFile -Force
}

function AddSparseCheckoutPath([string]$subDirectory) {
if (!(Test-Path $sparseCheckoutFile) -or !((Get-Content $sparseCheckoutFile).Contains($subDirectory))) {
Write-Output $subDirectory >> $sparseCheckoutFile
}
}

function GetSDKCloneDir() {
$root = git rev-parse --show-toplevel

$sparseSpecCloneDir = "$root/sdk"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a place out of this repo?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes lets put this somewhere not inside the existing repo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be run in 2 pipelines where the current repo is one of several in the agent work folder and possibly on dev machines. I wouldn't write outside of the repo on dev machines unless we take the location as a parameter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend we put it in a gitignored folder within the repo. I think under /artifacts is appropriate.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd recommend we put it in a gitignored folder within the repo. I think under /artifacts is appropriate.

This sound reasonable.
@pshao25 @weshaggard I do not want to put it outside this repo because I do not want to introduce the possibility that the script in this repo accidentally changes something outside this repo because those might be irreversible.

New-Item $sparseSpecCloneDir -Type Directory -Force | Out-Null
$createResult = Resolve-Path $sparseSpecCloneDir
return $createResult
}

$sdkCloneDir = GetSDKCloneDir
Write-Host "Setting up sparse clone for $gitRemoteUrl at $sdkCloneDir"

Push-Location $sdkCloneDir.Path
try {
if (!(Test-Path ".git")) {
Write-Host "Initializing sparse clone for $gitRemoteUrl"
InitializeSparseGitClone $gitRemoteUrl
AddSparseCheckoutPath $azCoreSharedPath
AddSparseCheckoutPath $armCoreSharedPath
$commitHash = git rev-parse $gitRemoteBranchName
git checkout $commitHash
}
if ($LASTEXITCODE) { exit $LASTEXITCODE }
}
finally {
Pop-Location
}

function CopyFilesToDestination([string]$sdkCloneRoot, [string]$dirName, [string]$dest, [string[]]$files) {
$source = Join-Path $sdkCloneRoot $dirName
foreach ($file in $files) {
$sourceFile = Join-Path $source $file
$destFile = Join-Path $dest $file
# to avoid the format issues (the file is required to end by a newline)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand this issue. If the newlines don't match then we should fix this in the .NET repo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed this in the .NET repo:
Azure/azure-sdk-for-net#36250

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no standard encoding or trailing new-line rule in the .NET repo but this repo's build enforces trailing new lines. We should either relax that rule here or start enforcing the rule for shared code in azure-sdk-for-net

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then let me change it back so that those unwanted changes will disappear as well.

# here we read all the content in the source file, and then output it into the destination file
Write-Host "Copying $file"
$text = Get-Content -Path $sourceFile
$text | Out-File -FilePath $destFile
}
}

$downloadPath = Resolve-Path (Join-Path $PSScriptRoot '..' 'src' 'assets' 'Azure.Core.Shared')
Get-ChildItem $downloadPath -Filter *.cs | Remove-Item;
$files = @('AsyncLockWithValue.cs', 'ClientDiagnostics.cs', 'DiagnosticScope.cs', 'DiagnosticScopeFactory.cs', 'ContentTypeUtilities.cs', 'HttpMessageSanitizer.cs',
'OperationInternalBase.cs', 'OperationInternal.cs', 'OperationInternalOfT.cs', 'TaskExtensions.cs', 'Argument.cs', 'Multipart/MultipartFormDataContent.cs',
'Multipart/MultipartContent.cs', 'AzureKeyCredentialPolicy.cs', 'AppContextSwitchHelper.cs',
'OperationPoller.cs', 'FixedDelayWithNoJitterStrategy.cs', 'SequentialDelayStrategy.cs',
'ForwardsClientCallsAttribute.cs', 'AsyncLockWithValue.cs', 'VoidValue.cs')
$baseUrl = 'https://raw.githubusercontent.com/Azure/azure-sdk-for-net/main/sdk/core/Azure.Core/src/Shared/'
DownloadAll $files $baseUrl $downloadPath

#Download management Shared
$files = 'SharedExtensions.cs', 'ManagedServiceIdentityTypeV3Converter.cs'
$downloadPath = Resolve-Path (Join-Path $PSScriptRoot '..' 'src' 'assets' 'Management.Shared')
Get-ChildItem $downloadPath -Filter *.cs | Remove-Item;
$baseUrl = 'https://raw.githubusercontent.com/Azure/azure-sdk-for-net/main/sdk/resourcemanager/Azure.ResourceManager/src/Shared/'
DownloadAll $files $baseUrl $downloadPath
try {
Write-Host "Downloading Azure.Core.Shared"
$downloadPath = Resolve-Path (Join-Path $PSScriptRoot '..' 'src' 'assets' 'Azure.Core.Shared')
Write-Host "Removing the existing files in " $downloadPath
Get-ChildItem $downloadPath -Filter *.cs | Remove-Item;
$files = @('AsyncLockWithValue.cs', 'ClientDiagnostics.cs', 'DiagnosticScope.cs', 'DiagnosticScopeFactory.cs', 'ContentTypeUtilities.cs', 'HttpMessageSanitizer.cs',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I this every file in the directory? Ideally we would sync the entire directory and not have to maintain this list of files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This is a subset we need. We only included things we need.

'OperationInternalBase.cs', 'OperationInternal.cs', 'OperationInternalOfT.cs', 'TaskExtensions.cs', 'Argument.cs', 'Multipart/MultipartFormDataContent.cs',
'Multipart/MultipartContent.cs', 'AzureKeyCredentialPolicy.cs', 'AppContextSwitchHelper.cs',
'OperationPoller.cs', 'FixedDelayWithNoJitterStrategy.cs', 'SequentialDelayStrategy.cs',
'ForwardsClientCallsAttribute.cs', 'AsyncLockWithValue.cs', 'VoidValue.cs')

CopyFilesToDestination $sdkCloneDir $azCoreSharedPath $downloadPath $files

Write-Host "Downloading management Shared"
$downloadPath = Resolve-Path (Join-Path $PSScriptRoot '..' 'src' 'assets' 'Management.Shared')
Write-Host "Removing the existing files in " $downloadPath
Get-ChildItem $downloadPath -Filter *.cs | Remove-Item;
$files = 'SharedExtensions.cs', 'ManagedServiceIdentityTypeV3Converter.cs'

CopyFilesToDestination $sdkCloneDir $armCoreSharedPath $downloadPath $files
}
finally {
Write-Host "Removing the sparse checkout files in" $sdkCloneDir
Remove-Item $sdkCloneDir -Force -Recurse
}
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/AppContextSwitchHelper.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not know why there is this difference (I did not even see what this difference is), but if we keep this script, and we do not need to care about this change

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you set core.autocrlf to false so that the eol difference between your file and azure.core file shows up?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's please revert all of these. I'm not sure what the change is, but I don't think we intended to make it.

// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/Argument.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#if AZURE_NULLABLE
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/AsyncLockWithValue.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable enable
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/AzureKeyCredentialPolicy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using Azure.Core.Pipeline;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/ClientDiagnostics.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/ContentTypeUtilities.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable disable
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/DiagnosticScope.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable enable
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/DiagnosticScopeFactory.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/FixedDelayWithNoJitterStrategy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/HttpMessageSanitizer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/Multipart/MultipartFormDataContent.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System.Collections.Generic;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/OperationInternalOfT.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable enable
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/SequentialDelayStrategy.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System;
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Azure.Core.Shared/TaskExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable disable
Expand Down
2 changes: 1 addition & 1 deletion src/assets/Management.Shared/SharedExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

#nullable enable
Expand Down