x/tools/gopls: high memory consumption when used with a monorepo

[trapgate]

Please answer these questions before submitting your issue. Thanks!

What did you do?

Our code is in a fairly large monorepo containing about 26K files, including vendored packages.
I use VSCode, with gopls enabled, and launched from the root of our monorepo, so the workspace includes the whole repo.

What did you expect to see?

gopls using 1-2GB of memory. That's a wild guess that I can't really validate, but 8-10 seems excessive.

What did you see instead?

The gopls process is using 8-10GB of memory. This number goes up and down with editor usage, but it never goes below about 6GB, and it generally hovers around 8GB. This is a 16GB laptop, so there's quite a bit of memory pressure when gopls is running.

Nearly all the code in our monorepo is intended for linux, and this is a mac, so I've experimented with adding this setting:

    "gopls": {
        "env": {"GOOS": "linux"}
    },

This does affect the number of errors reported by gopls, but the memory consumption is approximately unchanged.

I also tried building the latest master of gopls using the patch mentioned in this bug: #37223. This didn't help either. I'm attaching an SVG heap profile from that version.
pprof.zip

Build info

golang.org/x/tools/gopls v0.3.3
    golang.org/x/tools/[email protected] h1:mTFqRDJQmpSsgDDWvbtGnSva1z9uX2XcDszSWa6DhBQ=
    github.com/BurntSushi/[email protected] h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
    github.com/sergi/[email protected] h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
    golang.org/x/[email protected] h1:WG0RUwxtNT4qqaXX3DPA8zHFNm/D9xaBpxzHt1WcA/E=
    golang.org/x/[email protected] h1:8gQV6CLnAEikrhgkHFbMAEhagSSnXWGV915qUMm9mrU=
    golang.org/x/[email protected] h1:OX66ZzpltgCOuBSGdaeT77hS2z3ub2AB+EuGxvGRBLE=
    golang.org/x/[email protected] h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
    honnef.co/go/[email protected] h1:sXmLre5bzIR6ypkjXCDI3jHPssRhc8KD/Ome589sc3U=
    mvdan.cc/xurls/[email protected] h1:KaMb5GLhlcSX+e+qhbRJODnUUBvlw01jt4yrjFIHAuA=

Go info

go version go1.14 darwin/amd64

GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/geoffhickey/Library/Caches/go-build"
GOENV="/Users/geoffhickey/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="*.internal.digitalocean.com,github.com/digitalocean"
GONOSUMDB="*.internal.digitalocean.com,github.com/digitalocean"
GOOS="darwin"
GOPATH="/Users/geoffhickey/do/cthulhu/docode"
GOPRIVATE="*.internal.digitalocean.com,github.com/digitalocean"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v7/w5lfw5qs1bd9np31z9vv04h80000gn/T/go-build021923438=/tmp/go-build -gno-record-gcc-switches -fno-common"

[gopherbot]

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

[heschi]

There should be some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

[trapgate]

gopls-heap.zip

[heschi]

Thanks.

Unfortunately I don't have good news for you. Everything you've posted points to expected behavior -- I don't see any signs of a memory leak or work pileup. It looks like the monorepo is simply too large to work with all at once.

If you're only interested in one folder, you can start VS Code in that folder. If you're interested in multiple, you can add them piecemeal to your workspace, but note that most gopls features, e.g. find references, will only work within one folder.

[Jacalz]

I am running in to the same issue described here and it is really annoying because my Linux machine with 8GB memory always runs out of memory and I have to pull out the power plug to be able to actually restart the computer.

This often occurs when i am using vscode and gopls with the https://github.com/fyne-io/fyne repository. I usually see memory slowly but steadily rising up to around five or six gigabytes until the memory in my computer is gone and everything stops functioning as expected. Killing of gopls just makes it rise steadily again.

[stamblerre]

@Jacalz: That sounds like a bug with older versions of gopls - can you confirm that you are using v0.4.0 (gopls version)?

It's also possible that you have multiple versions of gopls installed and VS Code is using an old one - you can confirm the path to the gopls binary that VS Code is using the Go: Locate Configured Go Tools command. Once we confirm you are using the correct gopls version, we can continue investigating the memory usage.

[Jacalz]

@stamblerre: I verified that vscode is using the correct installation and this is the path of it: gopls: /home/jacob/go/bin/gopls installed. Looks good to me. The version should be v0.4.0 as long as vscode doesn't update to the wrong release. I tried running Go: Install/Update tools and I still have enormous amounts of memory usage.

[stamblerre]

You can confirm the version by running /home/jacob/go/bin/gopls version.

If the memory usage is high with gopls/v0.4.0, you should have some files named gopls.PID- in your temporary directory, named after heap sizes. Please upload the pair for the largest heap.

[Jacalz]

Thanks. It indeed does appear like I am using the latest version:

jacob@pacman ~/go/src/sparta $ ~/go/bin/gopls version
golang.org/x/tools/gopls 0.4.0
    golang.org/x/tools/[email protected] h1:G4+YP9kaV4dJb79J5MobyApxX493Qa6VoiTceUmxqik=

I tried to find the PID files, but I could not find any. I looked in /tmp, but nothing with the name gopls stood out at all. Am I doing something wrong?

[stamblerre]

What OS are you on? Take a look at the https://pkg.go.dev/os?tab=doc#TempDir docs to see what directory gopls will write temporary files to.

[Jacalz]

I am on Linux. I looks like I just wasn't getting into regions of memory usage that was concidered too much. I did not open the huge monorepo (https://github.com/fyne-io/fyne) and only saw 800MB memory usage due to that. Below are the files that you requested:

gopls.3169-5GiB-goroutines.txt
gopls.3169-5GiB-heap.pb.gz

I must admit that they were quite hard to get. When I tried for the first time, the memory usage jumped to 4GB and then gopls hugged 82% of my cpu and the computer totally froze and that meant that a force restart pruged the temporary files. This was by opening two files from https://github.com/fyne-io/fyne and doing small edits and a two saving of those files.

I restarted, had the system monitor open and ready to kill gopls when I had the chance. My system only has 8GB memory so when gopls starts using over 6GB (I saw a total of 6.7 GB before it was starting to get laggy and I killed it), things get really out of control and in to unplug power adapter territory. Thus I could only get the PID files for 5GB heaps at most. The PID files for 1, 2, 3, 4 GB heaps are saved too in case you need those too.

[heschi]

I was able to reproduce a similar profile by simply opening the root of the Fyne project. Most of the memory is being used by the type checker:

Showing nodes accounting for 853.95MB, 88.04% of 969.98MB total
      flat  flat%   sum%        cum   cum%
  620.02MB 63.92% 63.92%   620.02MB 63.92%  go/types.(*Checker).recordTypeAndValue
   87.50MB  9.02% 72.94%    98.44MB 10.15%  go/parser.(*parser).parseOperand
   43.13MB  4.45% 77.39%   153.17MB 15.79%  go/parser.(*parser).parseElementList

That's a little surprising for Fyne which is a relatively small project. My best guess is that the problem is the GL libraries, which generate tens of thousands of lines of cgo.

Regardless, I don't see a gopls bug here; it appears to be general go/types memory usage as usual.

[huapox]

+1 with version gopls 0.4.0

[huapox]

get worse with gopls 0.4.1

sam @ debian in /_ext/gopath/bin |03:24:50  
$ ./gopls version
golang.org/x/tools/gopls 0.4.1
    golang.org/x/tools/[email protected] h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=
sam @ debian in /_ext/gopath/bin |03:32:22  
$ uname -a
Linux debian 4.9.0-4-amd64 #1 SMP Debian 4.9.51-1 (2017-09-28) x86_64 GNU/Linux

sam @ debian in /_ext/gopath/bin |03:43:33  
$ go version
go version go1.13.7 linux/amd64

[ggilley]

I have 6 projects open in vscode. A gopls process for each is running ranging from 465MB to 900MB. Crazy!

[hunterlxt]

gopls eat 22G memory within just one project. Do you have any experience with large project

using VSCode with the default config

[egorenar]

syzkaller is another Go project which causes gopls on my machine to consume 27GiB RAM.

[luffyao]

i have the same issue .
my gopls version is
golang.org/x/tools/gopls 0.4.1
golang.org/x/tools/[email protected] h1:0e3BPxGV4B3cd0zdMuccwW72SgmHp92lAjOyxX/ScAw=

uname -a

Linux shenjie-VirtualBox 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

go version :
go version go1.13.4 linux/amd64

[tommady]

golang.org/x/tools/gopls 0.4.1
go version go1.14.2 darwin/amd64
34G too crazy 🤪

[agnivade]

Hi everyone, just a quick reminder that just mentioning your Go version and gopls version is not sufficient information to help us debug this further.

If you notice issues with gopls consuming high memory,

• please include your VS Code settings
• the repo that is causing this (if it's open source)
• and a heap profile to go with it.

Thanks.

[waynr]

@heschik thanks for the quick response! If I understand what you are saying correctly alloc_space isn't short for "currently allocated space" but instead for "total allocated during the process's lifetime" and at the point in time when the memory profile is captured that space may have been deallocated.

Without looking carefully, I believe the primary result of your PR is to disable logging.

The debug.Instance struct is what is used to keep track of package statistics; when a user passes for example -debug=localhost:6060 they can then view package statistics gathered by what i assumed to be hooks on the main language server processing bits that are passed to debug.WithInstance. I didn't realize that it also contributes to logging.

[waynr]

I estimate about a 15% reduction in memory consumption.

After looking more carefully at the same profile in the PR I linked above, I see now what @heschik was talking about. Using go tool pprof -png -sample_index=inuse_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz instead of go tool pprof -png -sample_index=alloc_space -nodefraction=0 -nodecount=500 gopls.profile.mem-with-debug-server.gz shows that the debug.Instance consumes an almost negligible amount of memory.

[stamblerre]

There have been a number of memory improvements in gopls at master. Please try out with the instructions on https://github.com/golang/tools/blob/master/gopls/doc/user.md#unstable-versions, or wait for the next release.

[1raghavmahajan]

Great improvement guys. My memory usage dropped from 9gigs to almost 2gb.

[trapgate]

For me at least, gopls 0.4.4 has been much more parsimonious than previous versions - I'm seeing 1-2GB of memory usage on my monorepo, where before I was seeing 8-10GB. I haven't yet tried running with master; I will try to check that out in the next couple of days and report back.

[stamblerre]

Glad to hear that things are working better! There will be more improvements coming in gopls/v0.5.0, so I think I will go ahead and close this issue for now. If you encounter memory usage issues again, please take a look at the instructions in #36943 and file a new issue.

[egorenar]

I also confirm that it got much better, it became usable with large projects, thank you very much.

[rmerry]

Using gopls at master hasn't fixed this issue for me.

I'm working on a monorepo (~40GB in size) and v0.4.4 would regularly cause my remote machine to become unresponsive by using all available memory. I'm now using master which has bought me a little more time but not fixed the issue.

$ gopls version
golang.org/x/tools/gopls master
    golang.org/x/tools/[email protected] h1:/32sBC1LOo43X5JHPUZT+hbLpyamXpL4FgR5eMEYb7w=

Gopls now takes quite a little while longer to eat up all my memory but if left unchecked it will do just that. I'm still having to kill it every 3-4 minutes. My solution now is to use a systemd watchdog to kill gopls when memory usage gets too high---not ideal but far better than losing connection to my remote box.

[Jacalz]

I am having the roughly the same issues as @rmerry, but on a different repository. Using gopls with https://github.com/fyne-io/fyne results in massive memory consumption that easily climbs up to over 8 gigabytes of memory consumption. It climbs a lot more slowly with the latest releases, but it is still largely unusable sadly :(

[heschi]

@rmerry @Jacalz

Please file new issues following the steps at https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#memory-usage. Simply reporting that gopls uses a lot of memory, or describing the workarounds you use to kill it, doesn't help us solve the problem. The memory diagnostics might.

[mrjrieke]

@rmerry @Jacalz

Please file new issues following the steps at https://github.com/golang/tools/blob/master/gopls/doc/troubleshooting.md#memory-usage. Simply reporting that gopls uses a lot of memory, or describing the workarounds you use to kill it, doesn't help us solve the problem. The memory diagnostics might.

I like this suggestion. Additionally, can we please keep this issue open? I regularly try with the latest gopls code (trying since May of 2020) and this issue is certainly not fixed yet as of the latest code checked into master of gopls, so for now, I'm turning it off in VSCode.

Maybe this is just a really hard problem to fix and will take a good long while to fix? Could be that the feature set is too rich and there could be a reduced configuration option for gopls to run in a 'lite' mode. For now, unfortunately, it means I can't use the tool in my development.

[heschi]

Sorry, no. Every open issue with "memory" in the title immediately becomes a lightning rod for everyone with a complaint, regardless of how similar their problem is to the previous ones, and then we can't untangle who's who.

The issue is very hard to fix, but I happen to be working on it now. If you file a new issue with a memory zip I can take a look at whether the work I'm doing is likely to help.

[mrjrieke]

Sorry, no. Every open issue with "memory" in the title immediately becomes a lightning rod for everyone with a complaint, regardless of how similar their problem is to the previous ones, and then we can't untangle who's who.

The issue is very hard to fix, but I happen to be working on it now. If you file a new issue with a memory zip I can take a look at whether the work I'm doing is likely to help.
Thanks @heschi, and good luck! I'd love to be able to start using gopls some day. :)

[mrjrieke]

I guess the thing I don't understand is why it needs to use so much memory. I see it use 6gb every time I click save. Seems just huge for a go app.

[stamblerre]

@mrjrieke: We agree, and we'd love to help investigate. Please follow the instructions in the troubleshooting guide and file a new issue, as @heschi requested. Comments without concrete information are not valuable in resolving the problem.

[marioidival]

+1

golang.org/x/tools/gopls v0.7.5
    golang.org/x/tools/[email protected] h1:8Az52YwcFXTWPvrRomns1C0N+zlgTyyPKWvRazO9GG8=```

[luxuze]

+1

[luxuze]

$ gopls version
golang.org/x/tools/gopls v0.9.0-pre.1
golang.org/x/tools/[email protected] h1:Q/iWw49bdSyLaeXKbUs5mpwrB625Aap0CT/NP4NEh8Y=

[yehun]

golang.org/x/tools/gopls v0.9.4
golang.org/x/tools/[email protected] h1:YhHOxVi++ILnY+QnH9FGtRKZZrunSaR7OW8/dCp7bBk=

[com314159]

it seems every vscode project open will run a gopls process, with each gopls process 2G memory usage, I have to put all the project in one folder

[com314159]