VS Code setup of language server fails to load packages.

[sambatyon]

What version of rules_go are you using?

v0.31.0

What version of gazelle are you using?

v0.24.0

What version of Bazel are you using?

❯ bazel version
Build label: 5.0.0-homebrew

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

macOS Monterrey 12.3.1 (Apple M1 Max)

Any other potentially useful information about your toolchain?

Setting up the toolchain with:

go_register_toolchains(version = "1.18")

System's go:

❯ go version
go version go1.18 darwin/arm64
❯ gopls version
golang.org/x/tools/gopls v0.8.2  

Running locally

What did you do?

Followed the instructions in Editor Setup. Current .vscode/settings.json contents are:

{
  "go.goroot": "${workspaceFolder}/bazel-testproject/external/go_sdk",
  "go.toolsEnvVars": {
    "GOPACKAGESDRIVER": "${workspaceFolder}/tools/gopackagesdriver.sh"
  },
  "go.enableCodeLens": {
    "references": false,
    "runtest": false
  },
  "gopls": {
    "build.directoryFilters": [
      "-bazel-bin",
      "-bazel-out",
      "-bazel-testlogs",
      "-bazel-testproject",
    ],
    "formatting.gofumpt": true,
    "formatting.local": "testproject",
    "ui.completion.usePlaceholders": true,
    "ui.semanticTokens": true,
    "ui.codelenses": {
      "gc_details": false,
      "regenerate_cgo": false,
      "generate": false,
      "test": false,
      "tidy": false,
      "upgrade_dependency": false,
      "vendor": false
    },
  },
  "go.useLanguageServer": true,
  "go.buildOnSave": "off",
  "go.lintOnSave": "off",
  "go.vetOnSave": "off"
}

golang code is under a directory called golang/testproject, there there is a build file:

load("@io_bazel_rules_go//go:def.bzl", "go_library")

go_library(
  name = "example",
  srcs = [
    "example.go",
    "helper.go",
  ],
  importpath = "testproject/example",
)

Any import in my go files shows a diagnostics error, which hovering shows:

could not import fmt (cannot find package "fmt" in any of 
	/Users/user/testproject/bazel-testproject/external/go_sdk/src/fmt (from $GOROOT)
	/Users/user/go/src/fmt (from $GOPATH))
compiler[BrokenImport](https://pkg.go.dev/golang.org/x/tools/internal/typesinternal?utm_source%3Dgopls#BrokenImport)

gopls log within VS Code shows:

[Info  - 12:02:44 PM] 2022/04/10 12:02:43 go env for /Users/user/testproject
(root /Users/user/testproject)
(go version go version go1.18 darwin/arm64)
(valid build configuration = true)
(build flags: [])
GOMOD=/dev/null
GOMODCACHE=/Users/user/go/pkg/mod
GONOSUMDB=
GOPATH=/Users/user/go
GOSUMDB=sum.golang.org
GOWORK=
GOPRIVATE=
GOPROXY=https://proxy.golang.org,direct
GOROOT=/Users/user/testproject/bazel-testproject/external/go_sdk
GOINSECURE=
GONOPROXY=
GOCACHE=/Users/user/Library/Caches/go-build
GOFLAGS=
GO111MODULE=


[Info  - 12:02:46 PM] 2022/04/10 12:02:46 go/packages.Load
	snapshot=0
	directory=/Users/user/testproject
	query=[./... builtin]
	packages=0

[Error - 12:02:46 PM] 2022/04/10 12:02:46 initial workspace load failed: no packages returned: packages.Load error

[Error - 12:02:46 PM] 2022/04/10 12:02:46 errors loading workspace: no packages returned: packages.Load error
	snapshot=0
	directory=file:///Users/user/testproject

[Info  - 12:02:55 PM] 2022/04/10 12:02:55 go/packages.Load
	snapshot=1
	directory=/Users/user/testproject
	query=[file=/Users/user/testproject/golang/example/example.go]
	packages=1

[Info  - 12:02:55 PM] 2022/04/10 12:02:55 go/packages.Load
	snapshot=1
	package="//golang/example:example"
	files=[/Users/user/testproject/golang/example/example.go /Users/user/testproject/golang/example/helper.go]

[Info  - 12:02:55 PM] 2022/04/10 12:02:55 go/packages.Load
	snapshot=1
	package="//golang/example:example"
	files=[/Users/user/testproject/golang/example/example.go /Users/user/testproject/golang/example/helper.go]

[Info  - 12:02:55 PM] 2022/04/10 12:02:55 go/packages.Load
	snapshot=1
	directory=/Users/user/testproject
	query=[file=/Users/user/testproject/golang/example/example.go]
	packages=1

[Error - 12:02:56 PM] 2022/04/10 12:02:56 errors loading workspace: no packages returned: packages.Load error
	snapshot=1
	directory=file:///Users/user/testproject

[Info  - 12:03:25 PM] 2022/04/10 12:03:25 background imports cache refresh starting

[Info  - 12:03:25 PM] 2022/04/10 12:03:25 background refresh finished after 19.435209ms

[Info  - 12:07:44 PM] 2022/04/10 12:07:44 background imports cache refresh starting

[Info  - 12:07:44 PM] 2022/04/10 12:07:44 background refresh finished after 1.085458ms

[Error - 1:47:09 PM] 2022/04/10 13:47:09 errors loading workspace: no packages returned: packages.Load error
	snapshot=2
	directory=file:///Users/user/testproject

Checking the output of gopackagesdriver.sh yields:

❯ echo {} | ./tools/gopackagesdriver.sh file=golang/example/helper.go
INFO: Analyzed target @io_bazel_rules_go//go/tools/gopackagesdriver:gopackagesdriver (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @io_bazel_rules_go//go/tools/gopackagesdriver:gopackagesdriver up-to-date:
  bazel-bin/external/io_bazel_rules_go/go/tools/gopackagesdriver/gopackagesdriver_/gopackagesdriver
INFO: Elapsed time: 0,142s, Critical Path: 0,00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/io_bazel_rules_go/go/tools/gopackagesdriver/gopackagesdriver_/gopacka
INFO: Build completed successfully, 1 total action
Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps kind("go_library|go_test", same_pkg_direct_rdeps("golang/example/helper.go"))]
Running: [bazel build --tool_tag=gopackagesdriver --ui_actions_shown=0 --show_result=0 --build_event_json_file=/var/folders/_q/26xl2x2d1lv2dk8cmsc_kqkh0000gn/T/gopackagesdriver_bep_2017423708 --build_event_json_file_path_conversion=no --experimental_convenience_symlinks=ignore --ui_event_filters=-info,-stderr --noshow_progress --aspects=@io_bazel_rules_go//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs --keep_going //golang/example:example]
ERROR: /private/var/tmp/_bazel_user/3f5cdde5c8ec1d0625488cfc1f9bd206/external/io_bazel_rules_go/BUILD.bazel:42:7: GoStdlibList external/io_bazel_rules_go/stdlib_/stdlib.pkg.json failed: (Exit 1): builder failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/go_sdk/builder stdliblist -sdk external/go_sdk -installsuffix darwin_arm64 -out ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/go_sdk/src/crypto/elliptic/p256_asm.go:24:12: pattern p256_asm_table.bin: cannot embed irregular file p256_asm_table.bin
stdliblist: error running subcommand external/go_sdk/bin/go: exit status 1
{"NotHandled":false,"Sizes":{"WordSize":8,"MaxAlign":8},"Roots":["//golang/example:example"],"Packages":[{"ID":"//golang/example:example","Name":"example","PkgPath":"dragonbook","GoFiles":["/Users/user/testproject/golang/example/example.go","/Users/user/testproject/golang/example/helper.go"],"CompiledGoFiles":["/Users/user/testproject/golang/example/example.go","/Users/user/testproject/golang/example/helper.go"],"ExportFile":"/private/var/tmp/_bazel_user/3f5cdde5c8ec1d0625488cfc1f9bd206/execroot/dragonbook/bazel-out/darwin_arm64-fastbuild/bin/golang/example/example.x"}]}

Checking from other people with similar problems I attempted clearing the go cache:

❯ go clean -cache

As well as resetting ownership of the cache:

sudo chown -R $USER ~/Library/Caches/go-build

Building and running the binaries succeed without problems.

What did you expect to see?

Features related to the standard library do not work:

• Diagnostics
• Completion
• Hover
• Signature Help

However modules internal to the project can be used without problems.

What did you see instead?

[sambatyon]

As I imagined, this seems to be related to go not wanting to see a symlink. The issue seems to have been introduced with cl/380475/ there is cl/383995/ which attempts to fix the issue, but I am unsure if this was tested or if it plays nicely with bazel.

I checked the contents of external/go_sdk/src/crypto/elliptic/ and all the files there seem to be real files, but it is unclear to me where p256_asm_table.bin is being embeded from. So probably this is indeed a bug in rules_go?

[pierreis]

Getting the same issue, which forces me to revert to go 1.17.6 for now. It is indeed quite annoying.

[iamricard]

👋 we think we figured out the smallest patch to work around this issue, but we're not sure about all the implications. like @sambatyon explains above, the issue was introduced in cl/380475/. cl/383995/ does try to solve the issue for some tests in the Go repo but it doesn't attempt to fix it for anything else.

looking at the contents of the go_sdk directory everything does seem fine, and none of them are symlinks:

~/testproject $ ls bazel-testproject/external/go_sdk/src/crypto/elliptic/                                                                                                                                                                                                      
elliptic.go	  export_generate.go  gen_p256_table.go  p224.go       p256.go	    p256_asm_amd64.s  p256_asm_ppc64le.s  p256_asm_table.bin	  p256_generic.go  p256_s390x.go  p384.go
elliptic_test.go  fuzz_test.go	      internal		 p224_test.go  p256_asm.go  p256_asm_arm64.s  p256_asm_s390x.s	  p256_asm_table_test.go  p256_ppc64le.go  p256_test.go   p521.go

we can also inspect the contents of p256_asm.go in that directory (output trimmed for readability):

//go:build amd64 || arm64

package elliptic

import (
	_ "embed"
	"math/big"
)

//go:generate go run -tags=tablegen gen_p256_table.go

//go:embed p256_asm_table.bin <--- THE EMBEDDED FILE

at a glance this doesn't seem like an issue because none of the files are symlinks (note that go's embed package is expected not to work symlinks). so we tried re-running the build command that the driver ran, with the --sandbox_debug and --verbose_failures flags:

# formatted for readability
bazel build \
  --tool_tag=gopackagesdriver \
  --ui_actions_shown=0 \
  --show_result=0 \
  --build_event_json_file=/var/folders/4v/gx_xc2fn1pj9pj8yd8bdnl9r0000gn/T/gopackagesdriver_bep_252991413 \
  --build_event_json_file_path_conversion=no \
  --experimental_convenience_symlinks=ignore \
  --ui_event_filters=-info,-stderr \
  --noshow_progress \
  --aspects=@io_bazel_rules_go//go/tools/gopackagesdriver:aspect.bzl%go_pkg_info_aspect \
  --output_groups=go_pkg_driver_json_file,go_pkg_driver_stdlib_json_file,go_pkg_driver_srcs \
  --keep_going \
  --sandbox_debug \
  --verbose_failures \
  //example:example

the output for this is pretty useful:

ERROR: /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/external/io_bazel_rules_go/BUILD.bazel:42:7: GoStdlibList external/io_bazel_rules_go/stdlib_/stdlib.pkg.json failed: (Exit 1): sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/sandbox/darwin-sandbox/7/execroot/testproject && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=darwin \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:external/local_config_cc:/bin \
    TMPDIR=/var/folders/4v/gx_xc2fn1pj9pj8yd8bdnl9r0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/sandbox/darwin-sandbox/7/sandbox.sb /var/tmp/_bazel_ricard.sole/install/7a9606eb8f092191f2180bf868dc1b49/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/go_sdk/builder stdliblist -sdk external/go_sdk -installsuffix darwin_amd64 -out bazel-out/darwin-fastbuild/bin/external/io_bazel_rules_go/stdlib_/stdlib.pkg.json)

with this, we cd into the sandbox: /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/sandbox/darwin-sandbox/7/execroot/testproject. if we dig into go_sdk, we'll see that stdliblist is just go list with some extra config. so we tried running go list -json builtin std runtime/cgo, which is roughly what that command above runs... and it works! the only env variable that seems like could make a difference in the outcome is GOROOT, so we re-run it: GOROOT=external/go_sdk go list -json builtin std runtime/cgo and we get the error above. next, we looked at the contents of external/go_sdk in the sandbox (output trimmed for readability) and sure enough, everything is a symlink:

/private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/sandbox/darwin-sandbox/7/execroot/testproject $ ls -la external/go_sdk/src/crypto/elliptic/
# ...
lrwxr-xr-x  1 ricard.sole wheel 137 May 11 09:16 p256_asm.go -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm.go
lrwxr-xr-x  1 ricard.sole wheel 142 May 11 09:16 p256_asm_amd64.s -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm_amd64.s
lrwxr-xr-x  1 ricard.sole wheel 142 May 11 09:16 p256_asm_arm64.s -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm_arm64.s
lrwxr-xr-x  1 ricard.sole wheel 144 May 11 09:16 p256_asm_ppc64le.s -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm_ppc64le.s
lrwxr-xr-x  1 ricard.sole wheel 142 May 11 09:16 p256_asm_s390x.s -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm_s390x.s
lrwxr-xr-x  1 ricard.sole wheel 144 May 11 09:16 p256_asm_table.bin -> /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject/external/go_sdk/src/crypto/elliptic/p256_asm_table.bin
# ...

now we cd into the execroot instead with cd /private/var/tmp/_bazel_ricard.sole/687eea90887650a4f8ab9d09222d1974/execroot/testproject, and re-run GOROOT=external/go_sdk go list -json builtin std runtime/cgo. 🎉 it works! so we determined the issue is that running go list inside a bazel sandbox just won't work (and i doubt go will change this behavior just to accomodate bazel?). so we took an approach similar to the one in aspect-build/rules_swc#31 by adding flags to bazel build using the env variable GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS: --strategy=GoStdlibList=local.

if this seems like a reasonable fix/workaround, i'm happy to send a patch.

note: after running it with strategy local, if you need/want to revert the strategy you will need to add the flag --modify_execution_info=GoStdlibList=+no-cache (see docs).

[brianwolfe]

Thanks for posting your investigation. For what it's worth, when I ran gopackagesdriver with

GOPACKAGESDRIVER_BAZEL_FLAGS=--strategy=GoStdlibList=local

I saw an error on the bazel query step:

Running: [bazel info --tool_tag=gopackagesdriver --ui_actions_shown=0]
Running: [bazel query --tool_tag=gopackagesdriver --ui_actions_shown=0 --strategy=GoStdlibList=local --ui_event_filters=-info,-stderr --noshow_progress --order_output=no --output=label --nodep_deps --noimplicit_deps --notool_deps @io_bazel_rules_go//:stdlib]
ERROR: --strategy=GoStdlibList=local :: Unrecognized option: --strategy=GoStdlibList=local
error: unable to build JSON files: query failed: unable to query: bazel query failed: exit status 2%

Instead, I changed our gopackagesdriver bash script to be:

GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS=--strategy=GoStdlibList=local
exec bazel run -- @io_bazel_rules_go//go/tools/gopackagesdriver "$@"

Where it used to be just the second line. Adding this in case it helps someone else.

[iamricard]

@brianwolfe good catch, I meant to call out the flag for build targets only not the flag for all bazel targets.

[iamricard]

i think this might've been fixed by #3111

edit: i actually meant this PR i think #3157

[vymao]

Is this fixed? I tried adding GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS=--strategy=GoStdlibList=local to the gopackagesdriver bash script but I still get a

Error loading workspace: no packages returned: packages.Load error

error

[ARR4N]

In case it helps others, I found that it only worked when adding the local strategy into my VS settings:

"go.toolsEnvVars": {
	"GOPACKAGESDRIVER": "${workspaceFolder}/devtools/gopackagesdriver.sh",
	"GOPACKAGESDRIVER_BAZEL_BUILD_FLAGS": "--strategy=GoStdlibList=local",
},