Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Version in tool cache is often behind go-versions manifest's latest #73

Closed
bdd opened this issue Sep 3, 2020 · 3 comments
Closed

Version in tool cache is often behind go-versions manifest's latest #73

bdd opened this issue Sep 3, 2020 · 3 comments
Labels
feature request New feature or request to improve the current logic

Comments

@bdd
Copy link

bdd commented Sep 3, 2020 •
edited
Loading

Goal: Set minimum required Go version but pick the latest available

For the use case of specifying a lower bound Go version for a project but defaulting to the latest release I'm using this action config fragment:

      - uses: actions/setup-go@v2
        with:
          go-version: '>=1.5 <2'

Down the rabbit hole 🐇

Check tool-cache

setup-go first checks if a matching version Go is in the tool cache. https://github.com/actions/setup-go/blob/main/src/installer.ts#L41

With ACTIONS_STEP_DEBUG=true, the relevant part from a build run:

Setup go stable version spec >=1.5 <2
##[debug]isExplicit: 
##[debug]explicit? false
##[debug]isExplicit: 1.11.13
##[debug]explicit? true
##[debug]isExplicit: 1.12.17
##[debug]explicit? true
##[debug]isExplicit: 1.13.15
##[debug]explicit? true
##[debug]isExplicit: 1.14.7
##[debug]explicit? true
##[debug]isExplicit: 1.15.0
##[debug]explicit? true
##[debug]evaluating 5 versions
##[debug]matched: 1.15.0
##[debug]checking cache: /opt/hostedtoolcache/go/1.15.0/x64
##[debug]Found tool in cache go 1.15.0 x64
Found in cache @ /opt/hostedtoolcache/go/1.15.0/x64

(Although the debug looks like it's iterating from oldest version in cache, it actually isn't the behavior. These are _isExplicitVersion(...) calls from findAllVersions() in @actions/toolkit/.../tool-cache.ts
https://github.com/actions/toolkit/blob/master/packages/tool-cache/src/tool-cache.ts#L514)

GitHub local Go Versions Manifest

Only if there isn't a matching version in the cache, it proceeds to checking @actions/go-versions:versions-manifest.json. A file that seems to be periodically generated, approved by humans in a PR, and gets committed again by a bot.

If we check versions-manifest.json, it has Go 1.15.1 in it. On the other hand the cache only has—semver cleaned up—1.15.0.

Can we prime the cache?

To debug this, I tried priming the cache by hardcoding the version and seeing if installGoVersion() from installer.ts would actually put this into the cache.

It says so:

##[debug]Caching tool go 1.15.1 x64
##[debug]source dir: /home/runner/work/_temp/9b74f02e-5739-4ef8-8416-f3c2b222710b
##[debug]destination /opt/hostedtoolcache/go/1.15.1/x64
##[debug]finished caching tool

...but when I revert back to >=1.5 <2 version spec and run the action again, it continues to match 1.5.0 from the tool-cache.

Questions

How does /opt/hostedtoolcache get populated?

Seems like only PyPy, Python, Ruby, boost, go, and node are the tools in this directory.
Makes me think this is centrally curated, and not a globally shared cache.

What cache gets updated by installGoVersion()?

When it calls tc.cacheDir on the extracted Go install, what is the effective cache domain for this? Is it global, per project, per user, per run?

Is checking the cache with a version spec intentional?

If we don't have an explicit version, setup-go checks the cache with the supplied semver version spec. Is this intentional? Because of this, there's no way to provide a version spec but hope to expect to get the latest version in go-versions manifest file, or directly from Google.
bdd added a commit to bdd/setup-go that referenced this issue Sep 14, 2020
@bdd
Add version-resolver input
0a5d868 
Tool cache can be stale in GitHub and self-hosted runners. If supplied
`go-version` is a semver range--instead of an explicit version--it will
be tried through the version inventory in cache, first. Even though
the GitHub local copy or origin distribution may have newer versions
matching the passed range, as long as the cache can satisfy the range,
the latest version from cache will be used.

Introduce an optional input, `version-resolver`. If defined, it tries to
resolve the version spec against either GitHub local copy or Go's
canonical source of version manifest.
If the value of `version-resolver` is:
  - "manifest": the manifest file under @actions/go-version get used.
  - "dist": the manifest at https://golang.org/dl/?mode=json&include=all
  gets used.  It can take values

Example:
--------
- Latest Go Version: 1.15.2
- tool-cache has go1.15 (1.15.0), latest.

build.yml:
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
[...]
```

Although the intention of the user might be to get latest
version between "oldest Go 1.15.0, newest--not inclusive--Go 2.0", the
cache will match this range with Go 1.15.0.

build.yml (v2):
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
      version-resolver: 'manifest'
[...]
```

With supplied version resolver, the semver range will be checked against
the local GitHub version manifest from @actions/go-versions[1], and
match 1.15.2. Cache will be queried with the resolved version, instead
of the range. When the cache gets updated globally, next runs will use
the tool from the cache, instead of downloading locally or in case of
resolver 'dist', directly from Google.

[1]: https://raw.githubusercontent.com/actions/go-versions/main/versions-manifest.json

Closes actions#73
bdd added a commit to bdd/setup-go that referenced this issue Sep 14, 2020
@bdd
Add version-resolver input
9333ae6 
Tool cache can be stale in GitHub and self-hosted runners. If supplied
`go-version` is a semver range--instead of an explicit version--it will
be tried through the version inventory in cache, first. Even though
the GitHub local copy or origin distribution may have newer versions
matching the passed range, as long as the cache can satisfy the range,
the latest version from cache will be used.

Introduce an optional input, `version-resolver`. If defined, it tries to
resolve the version spec against either GitHub local copy or Go's
canonical source of version manifest.
If the value of `version-resolver` is:
  - "manifest": the manifest file under @actions/go-version get used.
  - "dist": the manifest at https://golang.org/dl/?mode=json&include=all
  gets used.  It can take values

Example:
--------
- Latest Go Version: 1.15.2
- tool-cache has go1.15 (1.15.0), latest.

build.yml:
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
[...]
```

Although the intention of the user might be to get latest
version between "oldest Go 1.15.0, newest--not inclusive--Go 2.0", the
cache will match this range with Go 1.15.0.

build.yml (v2):
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
      version-resolver: 'manifest'
[...]
```

With supplied version resolver, the semver range will be checked against
the local GitHub version manifest from @actions/go-versions[1], and
match 1.15.2. Cache will be queried with the resolved version, instead
of the range. When the cache gets updated globally, next runs will use
the tool from the cache, instead of downloading locally or in case of
resolver 'dist', directly from Google.

[1]: https://raw.githubusercontent.com/actions/go-versions/main/versions-manifest.json

Closes actions#73
@bdd bdd mentioned this issue Sep 14, 2020
Closed
bdd added a commit to bdd/setup-go that referenced this issue Dec 16, 2020
@bdd
Add version-resolver input
b926061 
Tool cache can be stale in GitHub and self-hosted runners. If supplied
`go-version` is a semver range--instead of an explicit version--it will
be tried through the version inventory in cache, first. Even though
the GitHub local copy or origin distribution may have newer versions
matching the passed range, as long as the cache can satisfy the range,
the latest version from cache will be used.

Introduce an optional input, `version-resolver`. If defined, it tries to
resolve the version spec against either GitHub local copy or Go's
canonical source of version manifest.
If the value of `version-resolver` is:
  - "manifest": the manifest file under @actions/go-version get used.
  - "dist": the manifest at https://golang.org/dl/?mode=json&include=all
  gets used.  It can take values

Example:
--------
- Latest Go Version: 1.15.2
- tool-cache has go1.15 (1.15.0), latest.

build.yml:
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
[...]
```

Although the intention of the user might be to get latest
version between "oldest Go 1.15.0, newest--not inclusive--Go 2.0", the
cache will match this range with Go 1.15.0.

build.yml (v2):
```
[...]
  - uses: actions/setup-go@v2
    with:
      go-version: '>=1.5 <2'
      version-resolver: 'manifest'
[...]
```

With supplied version resolver, the semver range will be checked against
the local GitHub version manifest from @actions/go-versions[1], and
match 1.15.2. Cache will be queried with the resolved version, instead
of the range. When the cache gets updated globally, next runs will use
the tool from the cache, instead of downloading locally or in case of
resolver 'dist', directly from Google.

[1]: https://raw.githubusercontent.com/actions/go-versions/main/versions-manifest.json

Closes actions#73
sonnysideup added a commit to dominodatalab/distributed-compute-operator that referenced this issue Mar 13, 2021
@sonnysideup
set explicit go version in ci
15d5852 
we won't be able to use latest version until
actions/setup-go#73 is fixed
sonnysideup added a commit to dominodatalab/distributed-compute-operator that referenced this issue Mar 13, 2021
@sonnysideup
Adds initial smoke test for RayClusterReconciler (#15)
aa2077a 
* adds initial smoke test for RayClusterReconciler

* attemps to match latest bug fix go version in ci

* set explicit go version in ci

we won't be able to use latest version until
actions/setup-go#73 is fixed

* finishes smoke test for full raycluster create

* disables codecov patch testing

this isn't very helpful right now but it could be with the proper
configuration after we stabilize the project

* adds missing /fixes docs

* reenables project coverage checks

but this time explicitly
@marzocchi marzocchi mentioned this issue May 1, 2021
Merged
@Sergey-Murtazin Sergey-Murtazin added the feature request New feature or request to improve the current logic label Oct 25, 2021
This was referenced Dec 19, 2021
Closed
Closed
@Sergey-Murtazin
Copy link
Contributor

Sergey-Murtazin commented Dec 20, 2021 •
edited
Loading

Hi, @bdd!
Thanks for your request!

Answers to your questions:
How does /opt/hostedtoolcache get populated?
A cache of the virtual environment machine is set up by the virtual-environments team. And it isn't a "globally shared cache".

What cache gets updated by installGoVersion()?
All compilers are cached on the virtual machine image used to run your WF. For example, this is a list of all cached tools on the ubuntu-latest VM:
https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md

Is checking the cache with a version spec intentional?
Yes, you are right. The "checking the cache with a version spec is intentional". It's made to reduce the go compiler installation time if the specified version matched the cached one. Or when a user doesn't specify any version and the default one is used.
If you use a fully specified version, e.g. 1.17.3, this version will be installed if there is such version in "go-versions manifest file", or on Google server.
We are considering whether to add check latest version functionality.

@Sergey-Murtazin Sergey-Murtazin mentioned this issue Dec 20, 2021
Closed
@brcrista
Copy link
Contributor

brcrista commented Jan 4, 2022

We are considering whether to add check latest version functionality.

For more context, setup-node does this today: https://github.com/actions/setup-node/blob/d08cf222111d5c1d21b3cd4b958937f818d10d9a/docs/advanced-usage.md#check-latest-version

This was referenced Jan 20, 2022
Closed
Open
@dmitry-shibanov dmitry-shibanov mentioned this issue Feb 2, 2022
Merged
2 tasks
@dmitry-shibanov
Copy link
Contributor

Hello @bdd. We released a new version and updated tag v2 with adding check-latest input. For now I'm going to close the issue. If you have any concerns feel free to ping us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request to improve the current logic
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add version-resolver input bdd/setup-go
4 participants
@bdd @dmitry-shibanov @brcrista @Sergey-Murtazin