Improve performance of az on ubuntu hosted runners

[jessehouwing]

Description

Similar to: #8294

The logs show that first invocation on Ubuntu also does a command index rebuild and installs the python keyring package.

This pull request follows the same improvements that were just added to the Windows Hosted Runner.

Expected performance improvement about 10s for calling az for the first time.

Related issue:

• Closes: az behaves similarly on Ubuntu as it did on windows #8442

Check list

• Related issue / work item is attached
• Tests are written (if applicable)
• Documentation is updated (if applicable)
• Changes are tested and related VM images are successfully generated

[jessehouwing]

@ilia-shipitsin FYI.

Attached logs show that Ubuntu has the same issue as windows, the performance impact is a lot lower though, but still, this should improve things further:

dev.azure.com_jessehouwing_6484ebc3-af16-4af9-aa66-6b3398db7214__apis_build_builds_4830_logs_6.txt

Changes are the same as on windows.

[mikhailkoliada]

@jessehouwing forgot to mention, please add the source $HELPER_SCRIPTS/etc-environment.sh line before the actual code, it will let you import the setEtcEnvironmentVariable function

[jessehouwing]

@mikhailkoliada ✅

[mikhailkoliada]

/azp run ubuntu2004,ubuntu2204

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[mikhailkoliada]

@jessehouwing it looks like az --version does not work as it is being bitten by the permission denied error on the /opt/az/config/ dir. I'd say we can call sudo chmod 777 -R /opt/az because later we do sudo chmod 777 -R /opt in anyway.

[jessehouwing]

@jessehouwing it looks like az --version does not work as it is being bitten by the permission denied error on the /opt/az/config/ dir. I'd say we can call sudo chmod 777 -R /opt/az because later we do sudo chmod 777 -R /opt in anyway.

Hmz Funny... that worked on my machine...

Added the creation and setting permissions on that folder. Would it make sense to create and set permissions on that folder earlier in the process of creating the image, instead of setting the permissions trice?

Added to az-devops too.

[mikhailkoliada]

/azp run ubuntu2004,ubuntu2204

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[jessehouwing]

That didn't do the trick. 😥

Would we need to add this too?

https://github.com/actions/runner-images/blob/main/images/linux/scripts/installers/Install-Toolset.ps1#L53

But then for /opt/az and /opt/az-devops?

[jessehouwing]

Running a build to verify changes...

[jessehouwing]

Build succeeded.
@mikhailkoliada can you /azp run?

[jessehouwing]

/azp run ubuntu2004,ubuntu2204

[azure-pipelines]

Commenter does not have sufficient privileges for PR 8441 in repo actions/runner-images

[jessehouwing]

I wonder if I can, I suppose it won't work.

[jord1e]

@jessehouwing is something blocking you on this that I can assist in?

[jessehouwing]

@jessehouwing is something blocking you on this that I can assist in?

No clue what this is waiting for. As far as I can tell someone needs to run /azp run, test and merge.

[jord1e]

@mikhailkoliada could you trigger the pipeline/merge?
Thank you :)

[jessehouwing]

@vpolikarpov-akvelon Fixed a couple and replied to remaining comments.

[jessehouwing]

As requested:

• [Windows] Uses the correct env var on windows #8800
• [Windows] Cleans telemetry and logs from az #8801

[jessehouwing]

@mikhailkoliada 👍

[jessehouwing]

This PR seems to have changed the convention away from the setEtcEnvironmentVariable call back to | tee -a /etc/environment. I'd be happy to change these calls either way.

[jessehouwing]

@mikhailkoliada your thoughts? I'd like to get this closed with the pace at which you folks are updating files, I need to rebase this PR every other day at the moment ;)>

[jessehouwing]

These lines aren't needed, but were introduced by the refactoring of the build scripts.

The environment variable is already set in install-azure-cli.ch and it makes more sense there, as it's a config setting of az-cli and not of az-devops.

[jessehouwing]

@mikhailkoliada your thoughts? I'd like to get this closed with the pace at which you folks are updating files, I need to rebase this PR every other day at the moment ;)>

[jessehouwing]

Ping @mikhailkoliada @jord1e @vpolikarpov-akvelon I really want this thing done. What can we do to move it off the list? I've rebased since the restructure, it looks like some styling things changed, hence my open questions above.

I'd like to resolve those ASAP and then close this, it is taking longer than I'd anticipated.

[mikhailkoliada]

After observations I do not think we are ready to change this aspect.

1. I am not sure that customers won't be broken if we change the configuration directory and make it the global one
2. We do not observe hight demand in az optimisation from customers in general
3. Even if (2) were true I do not think it should be fixed from az upstream side and not the runner-images side

[jord1e]

My initial az logins take 22+ seconds first run, which is insane.

1. I am not sure that customers won't be broken if we change the configuration directory and make it the global one
   How would it break customers' setups? All configuration is handled via https://learn.microsoft.com/en-us/cli/azure/devops?view=azure-cli-latest#az-devops-configure for az devops, is it not? I get the concern but all configuration should be done via the CLI anyways.
2. We do not observe hight demand in az optimisation from customers in general
   Because they don't know about these things in general as it would only be observed in CI environments where there is no command & update index caching
3. Even if (2) were true I do not think it should be fixed from az upstream side and not the runner-images side
   How would az upstream be able to create an index and do an update check before the executable was every called?

And it is implemented on the windows runners already, why would implementing it on Ubuntu change anything summed up in these statements?
https://github.com/actions/runner-images/pull/8427/files

[jessehouwing]

After observations I do not think we are ready to change this aspect.

1. I am not sure that customers won't be broken if we change the configuration directory and make it the global one

This isn't as much of a change, as much as it is formalizing the default configuration. If not instructed through the environment variable, az will create a global configuration on first invocation and use that. If people do not want to use the default configuration, they should be overriding the environment variables to begin with, and that would still be possible.

2. We do not observe high demand in az optimisation from customers in general

I do see high demand for faster runners and faster workflows. Many organizations spend a large amount of time trying to speed up, parallelize and optimize their workflows. That there is no demand specifically for az might be because people aren't aware a 15s gain is available. GitHub even added large runners to reduce the duration of workflows.

3. Even if (2) were true I do not think it should be fixed from az upstream side and not the runner-images side

While I agree the az and az devops team could do more here, they're not building the tool to be primarily used on a hosted runner. az seems to me first and foremost a developer/administrator tool. The default setup works pretty well in those circumstances. The ephemeral hosted runner is a pretty specific environment and the installer isn't optimizing for that at the moment.

I think it would be possible on the ubuntu image to forego setting the environment variables, still have a reasonable speed improvement. The reason I set the environment variables on the ubuntu runner, is to make the setup aligned to the windows setup, where (due to the removal of the provisioning account) it was deemed preferred to set these environment variables.

Note that the current scripts already set a variable for the Azure DevOps extension installation folder.