go/types: UsesCgo exposes mangled names

[heschi]

When UsesCgo is enabled, cgo symbols appear in the package as their mangled names, e.g. _CFunc_GoString. This breaks autocomplete in gopls for C symbols. Also, it exposes things that are implementation details, like the _cgo_ and __cgofn symbols.

We can try to fix this in gopls, but probably it'd be better in go/types. I'm not sure offhand how it'll work, given that each package needs its own version of "C". Maybe the package path could be rewritten to something specific?

[mdempsky]

In CL 33677, I added support for resolving C.foo as _Cprefix_foo, but otherwise left the cgo-generated symbols in place in package scope. This is adequate for mere type-checking and analysis, but evidently suboptimal for tools like gopls, which want an accurate description of identifiers declared by the package (and imported packages, like the "C" pseudo-package).

[mdempsky]

So two limitations that will need to be addressed through further cmd/cgo support, and probably can't be resolved this late in the release cycle:

1. cmd/cgo only emits Go wrapper code for C identifiers that are actually used. E.g., if we have a package that includes <string.h> and uses C.strlen, but not C.strcmp, then go/types isn't going to know the latter exists. So a completion attempt for C.str<tab> won't be able to list it.

   Getting a complete description of all C identifiers that are accessible is possible, but would be tricky and might be prohibitively slow in practice. E.g., instead of analyzing the Go code to see what identifiers are used, cgo could analyze the C preprocessor output from the preamble and just analyze every possible C identifier.

2. Conversely, cmd/cgo emits a single Go wrapper file for the entire package, whereas each file is only allowed to access C symbols that are declared in its import "C" preamble. I.e., even if one file includes <string.h> and uses C.strlen, that doesn't necessarily mean it's valid to use in every other file that imports C. But gopls will probably suggest it anyway.

[mdempsky]

@heschik provided https://golang.org/cl/234057 as a test case.

[gopherbot]

Change https://golang.org/cl/234525 mentions this issue: go/types: type-check _cgo_gotypes.go as its own Package

[mdempsky]

@heschik I've uploaded CL 234525. It passes your test case. I'd appreciate if you could test it out in the context of gopls.

My expectation is that it should mean the _Cfoo_bar identifiers are no longer exposed within the cgo-using package, and that auto-completion for names from package C should work (aside from the limitations pointed out above).

I'll give the caveat that I think this should be fine, but it's somewhat more invasive than I feel really comfortable with at this point in the release cycle. I still think it's worth testing out now though, because if this is a desirable direction to go, I think we should tweak the UsesCgo API for this release (before it's cemented in place under Go 1 compat).

[gopherbot]

Change https://golang.org/cl/234526 mentions this issue: go/types: replace Config.UsesCgo with Checker.CgoFiles

[gopherbot]

Change https://golang.org/cl/237417 mentions this issue: go/types: rename UsesCgo to go115UsesCgo

[gopherbot]

Change https://golang.org/cl/237423 mentions this issue: go/packages: move TypecheckCgo to packagesinternal

[gopherbot]

Change https://golang.org/cl/237422 mentions this issue: internal/typesinternal: remove fallback check for UsesCgo

[mdempsky]

@heschik FYI, last night I rebased golang.org/cl/234525 and cleaned it up somewhat. It changes the "C" package so that it looks more like a normal Go package. It still fixes your test case (CL 234057).

There's still one issue I'm aware of: if you have a global C variable or macro, the Go type will likely be wrong. In particular, for something like:

// int x;
// #define xpp (x++)
import "C"

cgo makes C.x and C.xpp appear as C.int-typed values, and expressions accessing them will be type checked appropriately; but the objects declared by the pseudo C package will appear to have types *C.int and func() C.int, respectively. I have thoughts on how to fix these, but I might punt on them if this doesn't visibly affect gopls.

I'd be interested to know if you can spot any other issues.

[heschi]

Apologies for the rather extreme delay.

It looks good and is a definite improvement. The one thing I noticed is that while the mangled names are no longer visible at the top level, they've moved to the C package. It would be nice to hide them from there too if possible.

The incorrect types will confuse autocomplete somewhat. If the user has a function that takes a C.int, and they trigger autocompletion on that parameter, it will propose *C.x instead of C.x, which will then fail to compile. I don't think it's the end of the world.

[mdempsky]

It looks good and is a definite improvement.

Thanks for the test report.

The one thing I noticed is that while the mangled names are no longer visible at the top level, they've moved to the C package. It would be nice to hide them from there too if possible.

Interesting. You mean like the _Cfoo_bar identifiers are still visible somehow to gopls users?

I thought this should hide the mangled names as non-exported (https://go-review.googlesource.com/c/go/+/234525/5/src/go/types/object.go):

func (obj *object) Exported() bool {
	if obj.pkg != nil && obj.pkg.cgo {
		return !strings.HasPrefix(obj.name, "_C")
	}
	return token.IsExported(obj.name)
}

I can probably just suppress declaring the mangled identifiers though.

[heschi]

Yeah. I see _GoString_, __cgo_x, __cgofn__cgo_..., etc. as autocomplete suggestions on the C package.

[heschi]

@mdempsky is it too late to get this merged?

[odeke-em]

Howdy, and thank you Heschik and Matthew!

@mdempsky is it too late to get this merged?

Yes, it currently is. However I shall punt this to Go1.17 and make it early-in-cycle.

[ianlancetaylor]

Should we try to merge this for 1.17, or push again? Thanks.

[einthusan]

Bump. What is the status of this? Did this get merged into 1.17 .. ? I am using go 1.17.1 and this problem is still there. Any suggestions to workaround would be appreciated. Thanks in advance.

[ianlancetaylor]

@mdempsky @heschi This is in the Go 1.18 milestone. Is it likely to happen for 1.18? Thanks.

[heschi]

At this point it doesn't seem like a high priority. Bumping to backlog.

cc @findleyr

[mccolljr]

I ran into this problem this morning using golang.org/x/tools/gopls v0.15.2, and finally stumbled upon this issue. Is this dead in the water due to the issues mentioned previously, or is there some sort of plan for a change that will allow supporting completion after C. in gopls at some point?

[findleyr]

@mccolljr, sorry, catching up with this discussion. Doing some research, I think #35721 covers better support for working with cgo, particularly completion after C.. However, @mdempsky's https://go.dev/cl/234525 may be a better solution, as it results in fewer hacks required in gopls.

Matthew, any interest in reviving that CL? If not, perhaps someone else can pick it up--do you know of any reason why your approach wouldn't work?