Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add full clone fallback to sparse checkout #3661

Merged
merged 3 commits into from
Jul 25, 2022
Merged

Add full clone fallback to sparse checkout #3661

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
59adaf9
Add full clone fallback to sparse checkout
benbp Jul 15, 2022
75abd0b
Improve clone handling and overrides for sparse checkout
benbp Jul 22, 2022
507c213
Use SkipSparseCheckout variable name
benbp Jul 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 68 additions & 52 deletions eng/common/pipelines/templates/steps/sparse-checkout.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ steps:
script: |
function Clone([Hashtable]$repository)
benbp marked this conversation as resolved.
Show resolved Hide resolved
{
Write-Warning "Sparse checkout failed, falling back to full clone"
if (Test-Path .git) {
Write-Warning "Deleting existing git repository"
Write-Host "Remove-Item -Force -Recurse ./*"
Expand All @@ -34,6 +33,9 @@ steps:

Write-Host "git clone https://github.com/$($repository.Name) ."
git clone https://github.com/$($repository.Name) .
benbp marked this conversation as resolved.
Show resolved Hide resolved
if ($LASTEXITCODE) {
exit $LASTEXITCODE
}
Write-Host "git -c advice.detachedHead=false checkout $($repository.Commitish)"
# This will use the default branch if repo.Commitish is empty
git -c advice.detachedHead=false checkout $($repository.Commitish)
Expand All @@ -44,65 +46,56 @@ steps:

function SparseCheckout([Array]$paths, [Hashtable]$repository)
{
$dir = $repository.WorkingDirectory
if (!$dir) {
$dir = "./$($repository.Name)"
}
New-Item $dir -ItemType Directory -Force
Push-Location $dir

if (Test-Path .git/info/sparse-checkout) {
$hasInitialized = $true
Write-Host "Repository $($repository.Name) has already been initialized. Skipping this step."
} else {
Write-Host "Repository $($repository.Name) is being initialized."

Write-Host "git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) ."
git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) .
if ($LASTEXITCODE) {
return (Clone $repository && Pop-Location)
}
if (Test-Path .git/info/sparse-checkout) {
$hasInitialized = $true
Write-Host "Repository $($repository.Name) has already been initialized. Skipping this step."
} else {
Write-Host "Repository $($repository.Name) is being initialized."

Write-Host "git sparse-checkout init"
git sparse-checkout init
if ($LASTEXITCODE) {
return (Clone $repository && Pop-Location)
}
Write-Host "git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) ."
git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) .
if ($LASTEXITCODE) {
throw
}

# Set non-cone mode otherwise path filters will not work in git >= 2.37.0
# See https://github.blog/2022-06-27-highlights-from-git-2-37/#tidbits
Write-Host "git sparse-checkout set --no-cone '/*' '!/*/' '/eng'"
git sparse-checkout set --no-cone '/*' '!/*/' '/eng'
if ($LASTEXITCODE) {
return (Clone $repository && Pop-Location)
}
Write-Host "git sparse-checkout init"
git sparse-checkout init
if ($LASTEXITCODE) {
throw
}

# Prevent wildcard expansion in Invoke-Expression (e.g. for checkout path '/*')
$quotedPaths = $paths | ForEach-Object { "'$_'" }
$gitsparsecmd = "git sparse-checkout add $quotedPaths"
Write-Host $gitsparsecmd
Invoke-Expression -Command $gitsparsecmd
# Set non-cone mode otherwise path filters will not work in git >= 2.37.0
# See https://github.blog/2022-06-27-highlights-from-git-2-37/#tidbits
Write-Host "git sparse-checkout set --no-cone '/*' '!/*/' '/eng'"
git sparse-checkout set --no-cone '/*' '!/*/' '/eng'
if ($LASTEXITCODE) {
return (Clone $repository && Pop-Location)
throw
}
}

# Prevent wildcard expansion in Invoke-Expression (e.g. for checkout path '/*')
$quotedPaths = $paths | ForEach-Object { "'$_'" }
$gitsparsecmd = "git sparse-checkout add $quotedPaths"
Write-Host $gitsparsecmd
Invoke-Expression -Command $gitsparsecmd
if ($LASTEXITCODE) {
throw
}

Write-Host "Set sparse checkout paths to:"
Get-Content .git/info/sparse-checkout
Write-Host "Set sparse checkout paths to:"
Get-Content .git/info/sparse-checkout

# sparse-checkout commands after initial checkout will auto-checkout again
if (!$hasInitialized) {
Write-Host "git -c advice.detachedHead=false checkout $($repository.Commitish)"
# This will use the default branch if repo.Commitish is empty
git -c advice.detachedHead=false checkout $($repository.Commitish)
if ($LASTEXITCODE) {
return (Clone $repository && Pop-Location)
}
} else {
Write-Host "Skipping checkout as repo has already been initialized"
# sparse-checkout commands after initial checkout will auto-checkout again
if (!$hasInitialized) {
Write-Host "git -c advice.detachedHead=false checkout $($repository.Commitish)"
# This will use the default branch if repo.Commitish is empty
git -c advice.detachedHead=false checkout $($repository.Commitish)
if ($LASTEXITCODE) {
throw
}

Pop-Location
} else {
Write-Host "Skipping checkout as repo has already been initialized"
}
}

# Paths may be sourced as a yaml object literal OR a dynamically generated variable json string.
Expand All @@ -111,7 +104,30 @@ steps:
# Replace windows backslash paths, as Azure Pipelines default directories are sometimes formatted like 'D:\a\1\s'
$repositories = '${{ convertToJson(parameters.Repositories) }}' -replace '\\', '/' | ConvertFrom-Json -AsHashtable
foreach ($repo in $Repositories) {
SparseCheckout $paths $repo
$dir = $repo.WorkingDirectory
if (!$dir) {
$dir = "./$($repo.Name)"
}
New-Item $dir -ItemType Directory -Force
Push-Location $dir

try {
# Enable global override if there are sparse checkout issues
if ('$(DisableSparseCheckout)' -ne 'true') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use the usual Skip naming here instead of disable? So SkipSparseCheckout?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps both Disable and Skip aren't descriptive enough, as they only specify not doing something, as opposed to replacing sparse with a full clone.

Some other ideas:

  • OverrideSparseCheckout
  • UseFullClone
  • CheckoutMode == sparse|full|clone

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Override doesn't give any better details of what is actually happening. Maybe CheckoutMode but right now I think there is only 2 options I'm not sure what would be different between full and clone in your options there.

I don't have a strong feeling but I think Disable or Skip (slightly favoring skip for consistency) would still be the most descriptive of what we are trying to do which is to not run the sparse part of the checkout.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah re: full vs. clone, I figured we would pick one or the other to mean "full clone." Let's go with skip for now.

try {
SparseCheckout $paths $repo
} catch {
# Fallback to full clone if sparse checkout is not working properly
Write-Warning "Sparse checkout failed, falling back to full clone"
Clone $repo
}
} else {
Write-Warning "Sparse checkout disabled, performing full clone"
Clone $repo
}
} finally {
Pop-Location
}
}
pwsh: true
workingDirectory: $(System.DefaultWorkingDirectory)