go gets confused outside of gopath with dgit

[driusan]

When using go modules in go 1.11 if you're outside of $GOPATH and you try to go get a dependency, dgit gets confused by the fact that it's being used in a bare repository and it can't find the .git directory.

[sirnewton01]

I can't seem to reproduce this problem with these steps.

• installed go 1.11.5
• unset GOPATH
• go get github.com/shurcooL/vfsgen

Note that I didn't have the vfsgen package in my ~/go/src/github.com/shurcooL/vfsgen and dgit cloned it there just fine.

[sirnewton01]

Nm, I didn't realize that by not setting GOPATH Go still has an implied GOPATH of ~/go. To work outside this with Go 1.11 you have to work inside a directory that you've run "go mod init" and then the modules magic starts happening.

First problem. When running "go get" outside of GOPATH in a module, it creates a special hashed directory to represent the dependency in your ~/go/pkg/mod/cache/vcs. That directory is initialized using "git init --bare" and works fine with dgit.

However, it then attempts to add a remote to it with "git remotes add origin <url_of_dependency>" and that command is failing because dgit doesn't think that it is inside a git directory.

fatal: Not a git repository (or any parent)

[sirnewton01]

First, there are problems with handling of bare repos. I have a PR on the way for this. However, there's a bunch of different git commands that "go get" is doing on the special bare repos that aren't working:

• git init --bare (fixes coming shortly)
• git remote add (on a bare repo, fixes to work with a bare repo coming)
• git ls-remote (bug with the quiet flag that squelched all of the output, fix coming soon)
• git -c log.showsignature=false log -n1 --format=format:%H %ct %D (inline config not yet supported, -n flag not available, -format not available)
• git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ (-f not yet supported, multiple groups not supported)

Then it fails for now. I'm not sure what else go get will expect to be able to do after these are working.

[sirnewton01]

Splitting some of this work out into separate issues.

[driusan]

After the fetch, the next thing to fail seems to be referencing commits by a shorter prefix than the whole sha1

[sirnewton01]

I didn't notice commit prefixes in my case. I was getting ref specs coming into fetch that look like this with the full commit ID: ebfeedcc82378e1d1b9751bbc33446cf0fc49702:refs/dummy

I didn't even realize that you could fetch individual commits using git fetch before. :)

[driusan]

You can't, it's the git log -n1 --formatpart that uses about half the commit name, and then gives an error of "unrecognized revision" for me.

[driusan]

I updated the title because it's not just go get.. I've been trying to work outside of $GOPATH since upgrading to 1.12, and just about everything tries to fetch the remote dependencies if you don't pass -mod vendor.. go build, go install, go get, even go fmt..

[driusan]

The next problem if I merge #201 is that there's a checksum mismatch for golang.org/x/crypto if you try and do anything in module mode with dgit. I think we need to support core.autocrlf=input and core.eol=lf in the archive subcommand to fix it, based on what I see when running go with the -x parameter

[sirnewton01]

I updated the title because it's not just go get.. I've been trying to work outside of $GOPATH since upgrading to 1.12, and just about everything tries to fetch the remote dependencies if you don't pass -mod vendor.. go build, go install, go get, even go fmt..

I remember when Go used to be simple...

At this stage, I wonder why "go get" doesn't just include a git library to do its work rather than invoking more and more complicated command-line options.

[sirnewton01]

The next problem if I merge #201 is that there's a checksum mismatch for golang.org/x/crypto if you try and do anything in module mode with dgit. I think we need to support core.autocrlf=input and core.eol=lf in the archive subcommand to fix it, based on what I see when running go with the -x parameter

I'd rather not get into line ending conversion. It's a world of pain. It's really too bad that git was invented when it was. The world is slowly gravitating towards just LF for everything. Even MS is getting there. Then we can just store files in git as-is.

[driusan]

I looked a little more into the config options, and I think I was mistaken.. autocrlf=input means "convert on commit", and it's overwritten by a .gitattributes in the repo which treats everything as binary.. the reason for the mismatch needs more investigation

[driusan]

I think after #206 and #212 are merged this issue will be resolved.

[sirnewton01]

I pulled the latest from master branch and my PR for #206. I'm using Go 1.11.

It's trying to do this fetch command:
git fetch -f --depth=1 origin 6a9ea43bcacdf716a5c1b38efff722c07adf0069:refs/dummy

Which, I believe will fetch just that commit and make a ref "refs/dummy" that points to it. The command completes successfully, but I don't see the commit object, its tree or anything in the .git directory. Nor do I see the refs/dummy ref.

In my case, this fetch doesn't really work and then go get bails because it can't find it in the git log.

[driusan]

maybe a difference between 1.11 and 1.12? For me it with 1.12 it does a git fetch -f url refs/heads/*:refs/heads/* and then does a git cat-file blob sha1:go.mod

Are you sure you're looking in the right repo locally? Or maybe since it's using "origin" you're encountering the same problem as #211 and it's silently eating the error?

[sirnewton01]

The sequence of commands seems to be quite different in my case. Maybe there are different variations based on the state of the bare repo in ~/go/pkg/mod/vcs/<....> or perhaps there is a different with go 1.2.

I'll check once #211 is merged and see if it was the cause of the problem.

[sirnewton01]

I've pulled in all of the latest changes and analyzed what's going on with the special fetch command for a specific commit ID into a refs/dummy ref.

The problem appears to be in the call to conn.GetRefs() (fetchpack.go:111) where it is unable to get the remote ref for the commit ID. I'm not certain how the protocol is meant to work in this case. A commit ID is not a ref. Perhaps dgit is meant to link the commit ID to a ref on the remote system and then use that to fetch?

[driusan]

What are the exact commands go commands that you're running? I want to try them with go 1.12 and go 1.11 tonight to see what the difference is..

The protocol is documented (rather poorly) in the official git repo in Documentation/technical/pack-*.txt.. I usually just read it online here: https://github.com/git/git/tree/master/Documentation/technical

Effectively, both v1 and v2 protocols boil down to "get a list of remote refs and their commit id (implicitly on connect with v1, explicitly negotiated with v2), use some heuristic to send a list of commit ids that I have (we just send the tip of all our refs, the official git client walks up the commit graph a little before giving up in case something isn't a fast forward), then send a list of commit ids that I want, then receive a packfile".

For your situation, if you can determine that it's a SHA1 and not a ref, you can probably just append it to refs at that point.. I don't think it matters what name the ref has, because it looks like we just use ref.Value on line 129 to send the want line.

[sirnewton01]

Thanks, I did some reading on the protocol and with your guidance I have something working locally to handle the fetching of commits directly instead of through a remote ref. In my case go get is working now from within an initted go module. Here's what I'm running as a test with Go 1.11.

• cd [empty directory]
• go mod init
• go get github.com/shurcooL/vfsgen

In this case go get is running dgit to do these things:

• cd ~/go/pkg/mod/cache/vcs/<some_guid>
• git init --bare (Initialize a bare repo with no work dir)
• git remote add origin https://github.com/shurcooL/vfsgen (Add the remote for the repo as origin)
• git ls-remote -q origin (Find all of the remote heads and tags)
• git -c log.showsignature=false log -n1 "--format=format:%H %ct %D" 6a9ea43bcacdf716a5c1b38efff722c07adf0069 (Find whether we have the top commit from the master branch - we don't have it yet)
• git tag -l (See what tags are available in the local repo - why?)
• git fetch -f --depth=1 origin 6a9ea43bcacdf716a5c1b38efff722c07adf0069:refs/dummy (Fetch only this one commit and the related objects and point to it with a 'dummy' reference, neither a head, remote or tag)
• git -c log.showsignature=false log -n1 "--format=format:%H %ct %D" 6a9ea43bcacdf716a5c1b38efff722c07adf0069 (Ah, now the master commit is in the local repo)
• git cat-file blob 6a9ea43bcacdf716a5c1b38efff722c07adf0069:go.mod (Read the go.mod metadata file from the commit)

[sirnewton01]

The flow is different when you have a repo that has the module metadata in it and uses semver tags:

• cd ~/go/pkg/mod/cache/vcs/<some_guid>
• git init --bare
• git remote add origin https://github.com/golang/text
• git ls-remote -q origin
• git tag -l
• git -c log.showsignature=false log -n1 "--format=format:%H %ct %D" 1cbadb444a806fd9430d14ad08967ed91da4fa0a
• git fetch -f --depth=1 origin refs/tags/v0.2.0-unicode10.0.0:refs/tags/v0.2.0-unicode10.0.0 (I think it picks this tag based on the largest version number using "semver" semantics)
• git -c log.showsignature=false log -n1 "--format=format:%H %ct %D" refs/tags/v0.2.0-unicode10.0.0
• git cat-file blob 1cbadb444a806fd9430d14ad08967ed91da4fa0a:go.mod (it does this same command a couple of times for some reason, probably because it fails. The object is not a blob and the colon afterwards doesn't work)
• git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 1cbadb444a806fd9430d14ad08967ed91da4fa0a (it's interesting how it uses the commit ID here, but assumes with the above command that it is a treeish)
• git tag -l

[sirnewton01]

With #214 and the latest changes I am able to get simple "go get" to work with Go 1.11. There could be some minor problems (see comments above) but they don't seem to surface in the simple cases at least.

[sirnewton01]

@driusan if the latest changes are working for you in Go 1.12 then perhaps we can close this?

[driusan]

I didn't test it with the latest, but it was working before you made the fixes for 1.11 and it's passing the new Travis test, so I'm confident enough without manually testing it..