cmd/go: support fetching git repos with sha256 hashes #68359

johnnybubonic · 2024-07-09T23:42:45Z

Go version

go version go1.22.4 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/[REDACTED USER]/.cache/go-build'
GOENV='/home/[REDACTED USER]/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/opt/dev/go/pkg/mod'
GONOPROXY='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GONOSUMDB='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GOOS='linux'
GOPATH='/opt/dev/go'
GOPRIVATE='[REDACTED REPO]/[REDACTED NAME],[REDACTED REPO]/[REDACTED ORG]/*,github.com/[REDACTED ORG2]/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1827441876=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Let me preface this that this is likely (also?) an issue with the Go mod proxy, and my apologies if there's already an open issue for this but I looked and didn't see one (though one can imagine searching for "git" "sha256" in the Issues is going to return a lot of unrelated issues regardless for this project).

It is, however, unfortunately painfully apparent that SHA256 commit hashes do not seem to be supported:

(NOTE: Please see this comment; the module shown below has been recreated with SHA1 hashes and a new repository/module has been created to test with.)

$ go mod download r00t2.io/[email protected]
go: r00t2.io/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/dev/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: Server does not support this repository's object format
	fatal: the remote end hung up unexpectedly
	Gitea: Failed to execute git command

See also here, which provides a bit more useful info.

The same issue is present with 1.23 RC1 as well, of course:

$ go1.23rc1 mod download r00t2.io/[email protected]
go: r00t2.io/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/dev/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: Server does not support this repository's object format
	fatal: the remote end hung up unexpectedly
	Gitea: Failed to execute git command

The repository itself is fine (I have so-called "vanity" Go module roots/import paths):

$ git clone [email protected]:r00t2/go_wireproto && echo '##' ; rm -rf go_wireproto ; git clone https://git.r00t2.io/r00t2/go_wireproto && echo '##' ; cd go_wireproto && git tag -l
Cloning into 'go_wireproto'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 74 (delta 7), reused 0 (delta 0), pack-reused 53 (from 1)
Receiving objects: 100% (74/74), 61.25 KiB | 1.80 MiB/s, done.
Resolving deltas: 100% (31/31), done.
##
Cloning into 'go_wireproto'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 74 (delta 7), reused 0 (delta 0), pack-reused 53 (from 1)
Receiving objects: 100% (74/74), 61.25 KiB | 1.53 MiB/s, done.
Resolving deltas: 100% (31/31), done.
##
v1.0.0

The redirect is correct:

$ curl -H "user-agent: Go-http-client/v1" -sL "https://r00t2.io/wireproto?go-get=1" | grep -F 'content='
		<meta name="go-import" content="r00t2.io/wireproto git https://git.r00t2.io/r00t2/go_wireproto">

And it works fine with older repositories still using SHA1 commit hashes.

What did you see happen?

(See above)

What did you expect to see?

SHA256 commit hashes supported (as they've been in upstream git since ~ early 2021-ish?).

The text was updated successfully, but these errors were encountered:

gabyhelp · 2024-07-09T23:45:41Z

Related Issues and Documentation

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

seankhliao · 2024-07-09T23:56:16Z

The error form the .info endpoint:

not found: r00t2.io/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: mismatched algorithms: client sha1; server sha256

is the same as if you used GOPROXY=direct:

$ go get -v -x r00t2.io/[email protected]
# get https://r00t2.io/?go-get=1
# get https://r00t2.io/wireproto?go-get=1
# get https://r00t2.io/?go-get=1: 200 OK (0.775s)
# get https://r00t2.io/wireproto?go-get=1: 200 OK (0.882s)
get "r00t2.io/wireproto": found meta tag vcs.metaImport{Prefix:"r00t2.io/wireproto", VCS:"git", RepoRoot:"https://git.r00t2.io/r00t2/go_wireproto"} at //r00t2.io/wireproto?go-get=1
mkdir -p /home/user/.data/go/pkg/mod/cache/vcs # git3 https://git.r00t2.io/r00t2/go_wireproto
# lock /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225.lock
# /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225 for git3 https://git.r00t2.io/r00t2/go_wireproto
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git tag -l
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git tag -l
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git ls-remote -q origin
0.519s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git ls-remote -q origin
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 15fd8df0bff4b9b0f2c455422e57ef2bc6f8add5c95e045f9b51021874f22440 --
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 15fd8df0bff4b9b0f2c455422e57ef2bc6f8add5c95e045f9b51021874f22440 --
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v1.0.0:refs/tags/v1.0.0
0.513s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v1.0.0:refs/tags/v1.0.0
cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
0.504s # cd /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
go: r00t2.io/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/user/.data/go/pkg/mod/cache/vcs/3e171304ac1e55b26a8f8d1dae1cb29af5fc6089d97d912ba5cb27873649d225: exit status 128:
	fatal: mismatched algorithms: client sha1; server sha256

go fetches the a version using git init + git remote add + fetches,
and the local init will use sha1, resulting in a mismatch with the remote using sha256

https://go.googlesource.com/go/+/e89e880eacb1aceaa14733f44c38e6748d5ffa9c/src/cmd/go/internal/modfetch/codehost/git.go#83

johnnybubonic · 2024-07-10T03:52:31Z

I needed this library accessible for other projects, so I have deleted the above repository and re-recreated it using SHA1 commit hashes.

In its place, I have turned up a testing repository/module that can be used (and is quite a bit smaller) to aid in testing SHA256 commit hashes with go mod:

https://git.r00t2.io/r00t2/go_sha256repo

The import path is r00t2.io/sha256repo; I have set up an identical import URL for it.

rsc · 2024-07-25T18:53:44Z

It sounds like we can look at the hash length from ls-remote and then add --objectformat=sha256 to the 'git init' that we do.

gopherbot · 2024-12-16T13:53:59Z

Change https://go.dev/cl/636475 mentions this issue: cmd/go: add support for git sha256 hashes

dfinkel · 2024-12-16T13:55:58Z

I put together a quick CL to fix this: https://golang.org/cl/636475

I've marked it WIP because of the 1.24 freeze. (it does have (passing) tests and does work for my a sha256 git repo that I setup in Gitlab a couple weeks ago)

If there's an appetite for a freeze exception, I'll mark it as ready for review.

dfinkel · 2025-02-18T13:46:33Z

Since we're out of the go 1.24 freeze, I've rebased https://golang.org/cl/636475 and marked it as ready for review.

dfinkel · 2025-02-18T16:49:26Z

I should probably call out that the approach I took in cl/636475 has the unfortunate side-effect of requiring an extra touch on initial clone for anyone pulling private git repos over SSH and using security-key-resident SSH-keys that have touch-confirmation-enabled.

I think this is a necessary evil until Git supports mixing sha1 and sha256 hashes in the same repo.

seankhliao changed the title ~~cmd./go: go mod (nor pkg.go.dev) doesn't support SHA256 git repos~~ cmd/go: support fetching git repos with sha256 hashes Jul 9, 2024

seankhliao added FeatureRequest Issues asking for a new feature that does not need a proposal. GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 9, 2024

matloob assigned samthanawalla Jul 30, 2024

gabyhelp mentioned this issue Oct 8, 2024

SHA256 not supported #69812

Closed

samthanawalla assigned dfinkel and unassigned samthanawalla Feb 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: support fetching git repos with sha256 hashes #68359

cmd/go: support fetching git repos with sha256 hashes #68359

johnnybubonic commented Jul 9, 2024 •

edited

Loading

gabyhelp commented Jul 9, 2024

seankhliao commented Jul 9, 2024

johnnybubonic commented Jul 10, 2024

rsc commented Jul 25, 2024

gopherbot commented Dec 16, 2024

dfinkel commented Dec 16, 2024

dfinkel commented Feb 18, 2025

dfinkel commented Feb 18, 2025

cmd/go: support fetching git repos with sha256 hashes #68359

cmd/go: support fetching git repos with sha256 hashes #68359

Comments

johnnybubonic commented Jul 9, 2024 • edited Loading

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

gabyhelp commented Jul 9, 2024

seankhliao commented Jul 9, 2024

johnnybubonic commented Jul 10, 2024

rsc commented Jul 25, 2024

gopherbot commented Dec 16, 2024

dfinkel commented Dec 16, 2024

dfinkel commented Feb 18, 2025

dfinkel commented Feb 18, 2025

johnnybubonic commented Jul 9, 2024 •

edited

Loading

Output of `go env` in your module/workspace: