Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate python docs packages using docker #21657

Merged
merged 1 commit into from
Nov 9, 2021
Merged

Validate python docs packages using docker #21657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions eng/pipelines/docindex.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ jobs:
DocRepoLocation: $(Pipeline.Workspace)/docs
DocRepoOwner: MicrosoftDocs
DocRepoName: azure-docs-sdk-python
DocValidationImageId: azuresdkimages.azurecr.io/pyrefautocr:latest
steps:
# Docs CI uses Python 3.6.8 but that is not available on this image
- task: UsePythonVersion@0
Expand All @@ -21,6 +22,12 @@ jobs:
- pwsh: pip install --upgrade pip wheel setuptools
displayName: Update python tools for package verification

# Pull and build the docker image.
- template: /eng/common/pipelines/templates/steps/docker-pull-image.yml
parameters:
ContainerRegistryClientId: $(azuresdkimages-cr-clientid)
ContainerRegistryClientSecret: $(azuresdkimages-cr-clientsecret)
ImageId: "$(DocValidationImageId)"
# Sync docs repo onboarding files/folders
- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
Expand All @@ -38,7 +45,7 @@ jobs:
inputs:
pwsh: true
filePath: eng/common/scripts/Update-DocsMsPackages.ps1
arguments: -DocRepoLocation $(DocRepoLocation)
arguments: -DocRepoLocation $(DocRepoLocation) -ImageId '$(DocValidationImageId)'
displayName: Update Docs Onboarding
condition: and(succeeded(), or(eq(variables['Build.Reason'], 'Schedule'), eq(variables['Force.MainUpdate'], 'true')))
# Push changes to docs repo
Expand Down Expand Up @@ -70,7 +77,7 @@ jobs:
inputs:
pwsh: true
filePath: eng/common/scripts/Update-DocsMsPackages.ps1
arguments: -DocRepoLocation $(DocRepoLocation) -PackageSourceOverride "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/"
arguments: -DocRepoLocation $(DocRepoLocation) -PackageSourceOverride "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/" -ImageId '$(DocValidationImageId)'
displayName: Update Docs Onboarding for Daily branch
- template: /eng/common/pipelines/templates/steps/git-push-changes.yml
parameters:
Expand Down
39 changes: 32 additions & 7 deletions eng/scripts/Language-Settings.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,18 +162,43 @@ function Get-python-GithubIoDocIndex()
}

function ValidatePackage($packageName, $packageVersion, $workingDirectory) {
$packageExpression = "$packageName$packageVersion"
Write-Host "Validating $packageExpression"

$installTargetFolder = Join-Path $workingDirectory $packageName
New-Item -ItemType Directory -Force -Path $installTargetFolder | Out-Null

# Add more validation by replicating as much of the docs CI process as
# possible
# https://github.com/Azure/azure-sdk-for-python/issues/20109
if (!$ImageId) {
Write-Host "Validating using pip command directly on $packageName."
FallbackValidation -packageName "$packageName" -packageVersion "$packageVersion" -workingDirectory $workingDirectory
}
else {
Write-Host "Validating using $ImageId on $packageName."
DockerValidation -packageName "$packageName" -packageVersion "$packageVersion"
}
}
function DockerValidation($packageName, $packageVersion) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider moving this to a common location if we start duplicating most of the logic but I'm find with it being hear until we get the other languages in place as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Azure/azure-sdk-tools#2096

$packageExpression = "$packageName==$packageVersion"
docker run -e TARGET_PACKAGE=$packageExpression -t $ImageId
# The docker exit codes: https://docs.docker.com/engine/reference/run/#exit-status
# If the docker failed because of docker itself instead of the application,
# we should skip the validation and keep the packages.
if ($LASTEXITCODE -eq 125 -Or $LASTEXITCODE -eq 126 -Or $LASTEXITCODE -eq 127) {
Write-Host $commandLine
LogWarning "The `docker` command does not work with exit code $LASTEXITCODE. Fall back to npm install $packageName directly."
FallbackValidation -packageName "$packageName" -packageVersion "$packageVersion"
}
elseif ($LASTEXITCODE -ne 0) {
Write-Host $commandLine
LogWarning "Package $($Package.name) ref docs validation failed."
return $false
}
return $true
}

function FallbackValidation($packageName, $packageVersion, $workingDirectory) {
$installTargetFolder = Join-Path $workingDirectory $packageName
New-Item -ItemType Directory -Force -Path $installTargetFolder | Out-Null
$packageExpression = "$packageName$packageVersion"
try {
$pipInstallOutput = ""
$extraIndexUrl = " --extra-index-url=$PackageSourceOverride"
if ($PackageSourceOverride) {
Write-Host "pip install $packageExpression --no-cache-dir --target $installTargetFolder --extra-index-url=$PackageSourceOverride"
$pipInstallOutput = pip `
Expand Down