cmd/go: go get fails on non-ASCII github packages

[mcandre]

$ go get -u github.com/mcandre/toys/go/испытание
package github.com/mcandre/toys/go/испытание: invalid github.com/ import path "github.com/mcandre/toys/go/испытание"

In vcs.go:

var vcsPaths = []*vcsPath{
        // Github                                                                                                                               
        {
                prefix: "github.com/",
                re:     `^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`,
                vcs:    "git",
                repo:   "https://{root}",
                check:  noVCSSuffix,
        },

Could the regex be made more flexible?

[dmitshur]

I'll add some additional information about this bug below.

This affects golang.org/x/tools/go/vcs package in addition to cmd/go.

I've made a smaller reproduce repo (@rsc is welcome to fork it to his collection if desired; consider my repo temporary):

https://github.com/dmitshur-test/go-get-issue-unicode

Note that this issue affects only import paths that are statically known (GitHub, Bitbucket, etc.). It's not an issue for vanity import paths.

E.g., here is an alternative vanity import path that works without issues:

dmitri.shuralyov.com/test/go-get-issue-unicode/испытание

You can go get, go install, go test, go doc it without issues:

$ go get -u dmitri.shuralyov.com/test/go-get-issue-unicode/испытание
$ go install dmitri.shuralyov.com/test/go-get-issue-unicode/испытание
$ go test dmitri.shuralyov.com/test/go-get-issue-unicode/испытание
ok  	dmitri.shuralyov.com/test/go-get-issue-unicode/испытание	0.014s
$ go doc dmitri.shuralyov.com/test/go-get-issue-unicode/испытание
package испытание // import "dmitri.shuralyov.com/test/go-get-issue-unicode/испытание"

Package испытание demonstrates Unicode capabilities in Go source code.

type Эксперимент struct{ ... }
    func Испытание() Эксперимент

You can also call vcs.RepoRootForImportPath("dmitri.shuralyov.com/test/go-get-issue-unicode/испытание", false) successfully on the vanity import path:

$ goexec 'vcs.RepoRootForImportPath("dmitri.shuralyov.com/test/go-get-issue-unicode/испытание", false)'
(*vcs.RepoRoot)(&vcs.RepoRoot{
	VCS: (*vcs.Cmd)(&vcs.Cmd{
		Name:        (string)("Git"),
		Cmd:         (string)("git"),
		CreateCmd:   (string)("clone {repo} {dir}"),
		DownloadCmd: (string)("pull --ff-only"),
		TagCmd: ([]vcs.TagCmd)([]vcs.TagCmd{
			(vcs.TagCmd)(vcs.TagCmd{
				Cmd:     (string)("show-ref"),
				Pattern: (string)("(?:tags|origin)/(\\S+)$"),
			}),
		}),
		TagLookupCmd: ([]vcs.TagCmd)([]vcs.TagCmd{
			(vcs.TagCmd)(vcs.TagCmd{
				Cmd:     (string)("show-ref tags/{tag} origin/{tag}"),
				Pattern: (string)("((?:tags|origin)/\\S+)$"),
			}),
		}),
		TagSyncCmd:     (string)("checkout {tag}"),
		TagSyncDefault: (string)("checkout master"),
		LogCmd:         (string)(""),
		Scheme: ([]string)([]string{
			(string)("git"),
			(string)("https"),
			(string)("http"),
			(string)("git+ssh"),
		}),
		PingCmd: (string)("ls-remote {scheme}://{repo}"),
	}),
	Repo: (string)("https://github.com/dmitshur-test/go-get-issue-unicode"),
	Root: (string)("dmitri.shuralyov.com/test/go-get-issue-unicode"),
})
(interface{})(nil)

However, both go get and vcs.RepoRootForImportPath() fail on the "github.com/dmitshur-test/go-get-issue-unicode/испытание" import path, because the regexps in vcs.go only allow ASCII letters, not unicode ones.

[mattn]

It's not easy to expand character range of unicode. go's identifier doesn't accept some unicode classes. For example, IDEOGRAPHIC SPACE is.

[bradfitz]

@mattn, that's not a blocker for this bug. We can match on [^/]+.

[mattn]

@bradfitz please try this.

https://github.com/mattn/misc/blob/master/foo.go

the first import github.com/mattn/misc/世界 is okay to import. But second one github.com/mattn/misc/世　界 occur invalid import path. And I guess [^/]+ should not accept .., :, ;, &, % or something characters which can make another problems for each OSs.

[mattn]

If you think go get doesn't have to matter the letters, and the error should be occured expectedly, i don't have strong opinion. So please fixes this.

[gopherbot]

CL https://golang.org/cl/41750 mentions this issue.

[dmitshur]

I think I can apply a similar solution to fix this as I did for golang/gddo#468:

gosrc.IsValidRemotePath reports false for such import paths because
validPathElement regexp only allows ASCII letters A-Za-z, not Unicode
ones.

This change fixes that by using a predefined character class, the
Unicode character property class \p{L} that describes the Unicode
characters that are letters.

References

• https://stackoverflow.com/questions/3617797/regex-to-match-only-letters
• https://stackoverflow.com/questions/6005459/is-there-a-way-to-match-any-unicode-non-alphabetic-character
• http://www.regular-expressions.info/unicode.html#prop

See commit golang/gddo@cdd60fa for full details.

The fix for cmd/go then becomes:

diff --git a/src/cmd/go/internal/get/vcs.go b/src/cmd/go/internal/get/vcs.go
index 7439cc8649..c72d52bc1b 100644
--- a/src/cmd/go/internal/get/vcs.go
+++ b/src/cmd/go/internal/get/vcs.go
@@ -851,7 +851,7 @@ var vcsPaths = []*vcsPath{
 	// Github
 	{
 		prefix: "github.com/",
-		re:     `^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`,
+		re:     `^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[\p{L}0-9_.\-]+)*$`,
 		vcs:    "git",
 		repo:   "https://{root}",
 		check:  noVCSSuffix,

I tested it, and it works on the original bug report. It doesn't seem neccessary to allow Unicode for GitHub username and repository name, because from what I can tell, GitHub doesn't allow those:

So a Unicode character can only come up within a directory name of a GitHub repository.

I'd rather make the smallest possible change that'll resolve this issue (unless I'm advised otherwise).

If that sounds reasonable, I can send a CL that'll resolve this.

[josharian]

Possibly relevant: #20115

[bradfitz]

@shurcooL, SGTM.

[dmitshur]

#20115 provides further support for the choice of not allowing Unicode for GitHub usernames and repository names needlessly. That way, only the directory names will be susceptible, but not the actual repository, which helps.

@bradfitz Ok, I'll send a CL. I'll work on adding/updating a test case for it.

Would it be possible/good idea for @rsc to create a copy of (or fork) my https://github.com/shurcooL-test/go-get-issue-unicode repository, so I could rely on it to be there and use it in my tests? I ask because most go get issue repos are already there. @rsc can rename the repository to go-get-issue-18660.

[dmitshur]

Although, I'm not sure if this really needs a live go get test, I think some offline test cases for vcsPath path regexp matching could be sufficient.

[bradfitz]

Self-contained non-network test only please. :)

[gopherbot]

CL https://golang.org/cl/41822 mentions this issue.

[gopherbot]

CL https://golang.org/cl/42017 mentions this issue.