Runs az and az-devops during warmup #8294
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@jessehouwing , user is deleted after image generation. running "az ..." will put files to user profile that will be deleted. to properly work warmed up files should be copied to DEFAULT profile , for example |
|
Hmmm. I wonder where python stores this mess in that case. I have no clue what to copy exactly. |
|
@ilia-shipitsin it looks like a number of files are compiled in the %temp% folder and cached there for a limited time. I'm not versed enough in Python to change that behavior or to keep the intermediate files stored somewhere unfortunately:
|
|
maybe it allows to specify some other location (not in user profile), it might be nice (well, if you know where files are kept, we can recursively copy them to C:\Users\DEFAULT ) |
|
Much of the precompilation is stored in the actual cli extension directory which is already redirected to the common program files directory. I've added the copying of the .Azure directory in the profile in which a few other files are generated in first run. What's the easiest way to check whether this grandots the desired perf improvements. I could setup a scales agent, but I suppose your team has benchmarks for this stuff. |
|
on the image generation we do not have benchmarks. "az cli" is widely used by many ADO tasks, maybe they benchmark we appreciate your efforts, if you can help with benchmarking, it will be great |
|
I have a pipeline that exhibits the issue, but have no way to spin up a comparative hosted runner based on the image. I'm thinking 2 scalesets in azure could work. |
|
I came to an idea to ask for help in https://github.com/Azure/azure-cli taking into account numbers you measured warm up definitely makes sense and I wonder if there are best practices from Azure CLI team |
|
@ilia-shipitsin looking at the debug logs, my changes to the warmup script should ensure the new user has the correct command index, which takes 40s to rebuild on average. |
|
|
|
Using this pipeline I've simulated the presence of these files in both these directories: By copying them folders from a previous run. I've experimented whether calling When both the
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
vmImage: windows-latest
variables:
"system.debug": true
stages:
- stage: 'Cache'
displayName: 'Cache'
jobs:
- job: 'Cache'
displayName: 'Cache'
steps:
- script: |
c:
cd %USERPROFILE%
md .azure
cd .azure
echo [core] > config
echo first_run = false >> config
echo collect_telemetry = false >> config
echo error_recommendation = off >> config
echo output = json >> config
echo only_show_errors = true >> config
echo no_color = true >> config
echo disable_progress_bar = true >> config
echo [auto-upgrade] >> config
echo enable = False >> config
echo [logging] >> config
echo enable_log_file = no >> config
echo [output] >> config
echo show_survey_link = no >> config
az --help
az devops --help
az pipelines --help
az boards --help
- task: PublishPipelineArtifact@1
inputs:
targetPath: 'C:\Users\VssAdministrator\.azure'
artifact: '.azure'
publishLocation: 'pipeline'
- task: PublishPipelineArtifact@1
inputs:
targetPath: 'C:\Program Files\Common Files\AzureCliExtensionDirectory'
artifact: 'AzureCliExtensionDirectory'
publishLocation: 'pipeline'
- job: 'Run'
displayName: 'Run'
dependsOn: 'Cache'
steps:
- script:
rd /s /q "C:\Program Files\Common Files\AzureCliExtensionDirectory"
- task: DownloadPipelineArtifact@2
inputs:
buildType: 'current'
artifactName: '.azure'
targetPath: 'C:\Users\VssAdministrator\.azure'
- task: DownloadPipelineArtifact@2
inputs:
buildType: 'current'
artifactName: 'AzureCliExtensionDirectory'
targetPath: 'C:\Program Files\Common Files\AzureCliExtensionDirectory'
- pwsh: |
az pipelines runs show --id $env:BUILD_BUILDID --query "definition.id" --verbose --debug --organization $env:SYSTEM_COLLECTIONURI --project $env:SYSTEM_TEAMPROJECT
env:
AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
this reduces the call to ~30s: Previously: |
jessehouwing
changed the title
jessehouwing
force-pushed
the
patch-1
branch
from
1c90bbd to
b1de316
Compare
|
@jessehouwing few questions from my side (while I'm testing changes) first, executing external commands from Powershell (for instance, "az") requires checking $LASTEXITCODE after each execution (external commands behaviour do not obey ErrorActionPreference), we have a helper in macos, maybe it worth to have similar helper in windows. otherwise we do not know what is result of invoking "az" second, if we can warmup system wide, maybe we should keep it in Install-AzureCli.ps1, not a Warmup-User |
|
or maybe we can use something like ... |
|
Makes a lot of sense. I'll update that tomorrow. |
Now we wait. |
|
latest code successfully builds an image! @ilia-shipitsin ship.it ;)! |
|
well, I tend not to copy helper (at least this time). let's see how it will go. btw, is there a way to suppress that warning ?
|
|
well, I think we should not change |
One way would be calling setting Another way would be to write the config file manually: Let me see if I can find a way to pass the setting only temporarily. Worst case we could add |
|
I think manually adding lines to "config" may break things. we never know whether option is supported or not.
|
One way would be calling setting Another way would be to write the config file manually: Let me see if I can find a way to pass the setting only temporaily. Worst case we could add
Fixed by adding |
Commented out with justification for now. |
|
@jessehouwing , I did couple of iterations today. Still on it (yet resolving my own issues). I noticed you've found additional 10 sec delay. if you have spare time, you can try the following approach to collect WPP telemetry. It might give some insights on what was going on during measured command run: |
My knowledge of Python is not at the level you want me hacking away at that. From what I'm reading into the issue, the AzureDevOps CLI was built as an extension to az-cli, but for a long time could have been a stand-alone thing. az-cli does a number of sanity checks on first launch to iterate through accounts and such and az-devops catches that and substitutes its own auth mechanism if the environment variable is set. I'm not sure az-devops can currently convince az-cli not to do its discovery and thus the solution is a bit hacky. I don't have the telemetry on how many times az-devops runs on the hosted runner, but if this is another 10 seconds at 100.000+ invocations per day, it's a lot of compute saved, trees saved and water preserved. Ideally ac-cli would have check with the extension whether account iteration is needed before proceeding. |
jessehouwing
force-pushed
the
patch-1
branch
from
3238aaa to
10eaf15
Compare
|
Found that |
|
yep, I'll play with login. "az" appears to be a little bit trickier than I expected. I expected that since I invoke it durin image genegation with "az devops --help", it should have created all required stuff. |
|
I ran "az devops --help --debug" and compared output between first and second invocation. first invocation: it's interesting. somehow it invalidates either index or cloud profile. Most probably "index", because it decided to invalidate it. (image was generated with "az devops --help" run. config & cache was stored outside of ephemeral user profile) maybe it builds index based on user ID ? actually used is changed. |
|
I'd be interested to see which files are different before/after |
|
According to the az-cli team the normal The advantage is that it automatically provisions an access token to access self. |
|
Lemme see if I can setup a command a place you can run with a token from me. |
ilia-shipitsin
added a commit
to ilia-shipitsin/runner-images
that referenced
this pull request
Sep 28, 2023
as Jesse Houwing has found "az devops" warms up during first run which takes ~40 sec. warm up is done by running "az devops <subsystem> --help" with config redirected to persistent "C:\" location, also keyring installation requires "fake" login original PR: actions#8294 Co-authored-by: Jesse Houwing <[email protected]>
4 tasks
Co-authored-by: Stijn Devogel <[email protected]>
ilia-shipitsin
added a commit
to ilia-shipitsin/runner-images
that referenced
this pull request
Oct 2, 2023
Improve Azure CLI first run timing. Follow up: actions#8294 Co-authored-by: Jesse Houwing <[email protected]>
4 tasks
|
Closing then as a part of #8427 |
4 tasks
mikhailkoliada
pushed a commit
that referenced
this pull request
Oct 5, 2023
* [windows] warmup Azure CLI Improve Azure CLI first run timing. Follow up: #8294 Co-authored-by: Jesse Houwing <[email protected]> * expose AZURE_EXTENSION_DIR to image generation * suppress az warmup output * refresh PATH before warmup --------- Co-authored-by: Jesse Houwing <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Speed up performance of
azandaz devops. In my own experiments savings are sometimes more than 60 seconds.This is done by changing the default settings of
azbased on the recommended settings fromaz initfor automation workflows:az initfor automation workflows.It invokes
azandaz devopsduring the image build to ensure Python pre-caches its objects.It moves the
.azureand.azure-devopsand.azure-devops/cachefolders into the CommonFiles directory similarly to how the runner already moved installed extensions to the CommonFiles directory.First call performance, especially of
az devopscommands will still be slower than subsequent calls due to the fact thataz devopsfetches and caches information about the Azure DevOps Organization that it connects to. These calls still add about a 15s overhead. Users that want to speed upaz devopsfurther could consider caching the$env:AZURE_DEVOPS_CACHE_DIR.Links to sources used to make this pull request:
az initautomation recommended settings. - I left out the one that only outputs errors as it is hard to see what the workflow/pipeline's doing. Ideally that setting would tigger on the diagnostics flag of the workflow.az-cliconfiguration directory environment variable is defined hereaz devopsconfiguration environment variable is defined hereaz devopscache environment variable is defined in multiple places, it sometimes keys off of the configuration environment variable but sometimes it's hardcoded against the home directory if the environment variable isn't defined. Filed: Bug: the azure-devops cache folder location is computed differently in different places Azure/azure-devops-cli-extension#1366Technically changing any default configuration settings can be breaking. My personal view on these settings:
az config set auto-upgrade.enable=false- should not break anyone. The hosted image is updated regularly enough that it should not be needed to auto-upgrade az-cli during a workflow/pipeline run.az config set core.error_recommendation=off- should not break anyone. Worst case it makes debugging scripts a little harder, though running the same script locally should still provide recommendations.az config set core.disable_progress_bar=true- progress bars don't work in the hosted runner anyway.az config set core.no_color=true- will make the logs less pretty, but also fixes issues where certain ascii colors aren't coming through correctly. Should not really break anyone.az config set core.survey_message=false- This setting suppresses the survey links that the CLI sometimes outputs, they have no place in a pipeline or workflow anyway.I had already removed the settings from the suggestions made by
az initand left them on default:az config set logging.enable_log_file=no- I personally don't rely on the log file, I suppose someone could. This might be the most breaking change in the list.. There is anecdotal evidence in other github issues that suggests file logging slows down az-cli. I'd prefer to make it opt in. The less the agent does by default the better.az config set only_show_errors=true- technically not a breaking change and making the log less chatty. I removed this cause I feel it makes it harder to see what the cli is and isn't doing. Ideally this flag would be set based on the diagnostic settings of the pipeline itself.az config set collect_telemetry=false- though there is a lot of anecdotal evidence that telemetry slows downazor some of its extensions, I imagine the cli team would like to get telemetry from the hosted runners.People who want to save a little more time can opt to add these settings to their pipeline/workflow.
Related issue:
Fixes: #8296?
Check list