[xaprepare] Download and install dotnet ourselves, with caching #7026
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I believe the test failures here are irrelevant |
jonathanpeppers
approved these changes
May 24, 2022
There was a problem hiding this comment.
This looks good if it improves local dev (& CI times, too).
We'll have an issue if we ever need to use our internal fork, see:
If we have to consume an internal .NET build with a security fix, we use --internal --azure-feed --feed-credential. We've not had to use this yet, maybe we can just wait until then?
|
@jonathanpeppers if we need to do that, it will be easy to modify our code for this |
grendello
force-pushed
the
dotnet-download-caching
branch
from
d50c678 to
1ae83e9
Compare
pjcollins
reviewed
May 24, 2022
dellis1972
reviewed
May 24, 2022
| args.AddRange (new string [] { "-Runtime", "dotnet" }); | ||
| } | ||
| if (onlyGetUrls) { | ||
| args.Add ("--dry-run"); |
There was a problem hiding this comment.
don't we already add --dry-run on line 163? Do we need this as well or do we need the arg on line 163?
There was a problem hiding this comment.
Good eye, it's a mistake, will fix
pjcollins
reviewed
May 24, 2022
| } | ||
|
|
||
| if (!ReportAndCheckCached (message)) { | ||
| return false; |
There was a problem hiding this comment.
We shouldn't return here if the GetDownloadSizeWithStatus check is non-fatal.
There was a problem hiding this comment.
We return false from ReportAndCheckCached only when the script isn't found in ~/android-archives/. The idea was to let the code continue if we have the script, assuming that the network failures here would be because of actually having no internet access, but I think it does make more sense to make it non-critical in all cases. Will update
Context: dotnet/install-scripts#15 Context: https://dot.net/v1/dotnet-install.sh Context: https://dot.net/v1/dotnet-install.ps1 We've been installing dotnet versions using the Unix shell and Windows powershell scripts, which work fine. However, they do not cache the downloaded archive and therefore we end up re-downloading the same archive over and over again on CI servers as well as locally. Additionally, if one finds themselves without internet connection, there's no way to install the required version of dotnet. The installation scripts don't provide a way to cache the payloads and they appear to be in maintenance mode, so there's rather no chance to add caching support to them. Instead of relying on the scripts, we can do the downloading and installation ourselves, which lets us take the first step towards fully "offline" builds (and smaller downloads on CI servers). With this commit, we will cache both the scripts and the payloads and we will perform the installation ourselves. Scripts are still used to obtain the download URL(s) for dotnet and as a fallback should our attempt to download dotnet archives fail.
grendello
force-pushed
the
dotnet-download-caching
branch
from
7f6bbba to
9518175
Compare
The reason is that if the size check fails, the subsequent Download will throw an exception, terminating xaprepare. We don't want that because dotnet install scripts can return multiple URLs and not all of them will be available, so we need to try them all.
|
For the commit message, "what does this do?" What this change does is run: Which returns strings such as: We then split each line on When we want to actually install/provision |
|
@jonpryor yes, some of those URLs are 404, we keep on trying until we find one that works or we run out of URLs. |
[xaprepare] cache .NET install artifacts (#7026)
Context: https://github.com/dotnet/install-scripts/issues/15
Context: https://dot.net/v1/dotnet-install.sh
Context: https://dot.net/v1/dotnet-install.ps1
We've been installing dotnet versions using the [`dotnet-install`][0]
scripts for Unix & Windows. However, they do not cache the
downloaded archive, and therefore we end up re-downloading the same
archive over and over again.
Additionally, if one finds themselves without an internet connection,
there's no way to easily install the required version of dotnet.
The installation scripts don't provide a way to cache the payloads
and they appear to be in maintenance mode (dotnet/install-scripts#15),
so there doesn't appear to be a chance to add caching support to them.
Fortunately, we can "ask" the scripts what they're downloading:
% curl -o dotnet-install.sh 'https://dot.net/v1/dotnet-install.sh'
% ./dotnet-install.sh --version 7.0.100-preview.5.22273.1 --verbose --dry-run \
| grep 'dotnet-install: URL'
This returns a list of URLs, which may or may not exist:
dotnet-install: URL #0 - primary: https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100-preview.5.22273.1/dotnet-sdk-7.0.100-preview.5.22273.1-osx-x64.tar.gz
dotnet-install: URL #1 - legacy: https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100-preview.5.22273.1/dotnet-dev-osx-x64.7.0.100-preview.5.22273.1.tar.gz
dotnet-install: URL #2 - primary: https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.5.22273.1/dotnet-sdk-7.0.100-preview.5.22273.1-osx-x64.tar.gz
dotnet-install: URL #3 - legacy: https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.5.22273.1/dotnet-dev-osx-x64.7.0.100-preview.5.22273.1.tar.gz
We now parse this output, extract the URLs, then download and cache
the URL contents into `$(AndroidToolchainCacheDirectory)`.
When we need to install .NET, we just extract the cached archive
into the appropriate directory.
If no `dotnet-install: URL…` messages are generated, then we run
the `dotnet-install` script as we previously did.
This process lets us take a first step towards fully "offline" builds,
along with smaller downloads on CI servers.
[0]: https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-install-script |
simonrozsival
pushed a commit
to simonrozsival/xamarin-android
that referenced
this pull request
May 31, 2022
Context: dotnet/install-scripts#15 Context: https://dot.net/v1/dotnet-install.sh Context: https://dot.net/v1/dotnet-install.ps1 We've been installing dotnet versions using the [`dotnet-install`][0] scripts for Unix & Windows. However, they do not cache the downloaded archive, and therefore we end up re-downloading the same archive over and over again. Additionally, if one finds themselves without an internet connection, there's no way to easily install the required version of dotnet. The installation scripts don't provide a way to cache the payloads and they appear to be in maintenance mode (dotnet/install-scripts#15), so there doesn't appear to be a chance to add caching support to them. Fortunately, we can "ask" the scripts what they're downloading: % curl -o dotnet-install.sh 'https://dot.net/v1/dotnet-install.sh' % ./dotnet-install.sh --version 7.0.100-preview.5.22273.1 --verbose --dry-run \ | grep 'dotnet-install: URL' This returns a list of URLs, which may or may not exist: dotnet-install: URL #0 - primary: https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100-preview.5.22273.1/dotnet-sdk-7.0.100-preview.5.22273.1-osx-x64.tar.gz dotnet-install: URL #1 - legacy: https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100-preview.5.22273.1/dotnet-dev-osx-x64.7.0.100-preview.5.22273.1.tar.gz dotnet-install: URL #2 - primary: https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.5.22273.1/dotnet-sdk-7.0.100-preview.5.22273.1-osx-x64.tar.gz dotnet-install: URL #3 - legacy: https://dotnetbuilds.azureedge.net/public/Sdk/7.0.100-preview.5.22273.1/dotnet-dev-osx-x64.7.0.100-preview.5.22273.1.tar.gz We now parse this output, extract the URLs, then download and cache the URL contents into `$(AndroidToolchainCacheDirectory)`. When we need to install .NET, we just extract the cached archive into the appropriate directory. If no `dotnet-install: URL…` messages are generated, then we run the `dotnet-install` script as we previously did. This process lets us take a first step towards fully "offline" builds, along with smaller downloads on CI servers. [0]: https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-install-script
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context: dotnet/install-scripts#15
Context: https://dot.net/v1/dotnet-install.sh
Context: https://dot.net/v1/dotnet-install.ps1
We've been installing dotnet versions using the Unix shell and Windows
powershell scripts, which work fine. However, they do not cache the
downloaded archive and therefore we end up re-downloading the same
archive over and over again on CI servers as well as locally.
Additionally, if one finds themselves without internet connection,
there's no way to install the required version of dotnet.
The installation scripts don't provide a way to cache the payloads and
they appear to be in maintenance mode, so there's rather no chance to
add caching support to them.
Instead of relying on the scripts, we can do the downloading and
installation ourselves, which lets us take the first step towards fully
"offline" builds (and smaller downloads on CI servers).
With this commit, we will cache both the scripts and the payloads and we
will perform the installation ourselves. Scripts are still used to
obtain the download URL(s) for dotnet and as a fallback should our
attempt to download dotnet archives fail.