runtime: GC: heap idle is not released / decrease

[tonglil]

This is my first time submitting to Go, apologies in advance if it is a non-helpful issue.

1. What version of Go are you using (go version)?
   1.6
2. What operating system and processor architecture are you using (go env)?
   GOARCH="amd64"
   GOHOSTARCH="amd64"
   GOHOSTOS="darwin"
   GOOS="darwin"
   CC="clang"
   GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
   CXX="clang++"
3. What did you do?
   (Use play.golang.org to provide a runnable example, if possible.)

Sorry in advance, I am not able to extract a runnable example at the moment as it is a larger program I am debugging.

I am having issues with the GC in Go. Specifically, it does not seem to release idle heap memory back to the system, even after +10 minutes of inactivity (based on this) in the code/area that generated the initial garabage in the program.

Heap in use and allocated match one another (makes sense).
Heap sys = heap idle (this does not make sense to me).

1. What did you expect to see?
   Heap idle to decrease.
2. What did you see instead?
   Heap idle does not decrease over time. I even called runtime.GC() and then debug.FreeOSMemory() as well in attempt to release the memory.

Other information
Memory profile information from MemStats logged every 2 seconds:

I did reading of different sources and it one thought I see is that the GC may release memory, but it is up to the OS to collect/ask for it. Is this correct? Does this mean the memory footprint will not decrease without external intervention?

I thought maybe this problem has to do with #13552, however from 1.5 to 1.6 made no difference for me.

Thank you for your time.

[rubycut]

I also called FreeOsMemory(), nothing happends on linux, golang 1.5.3

10:17:52 WARN Allocated memory before GC: 3136 Mb
10:17:52 WARN Allocated memory from OS before FreeOSMemory: 3323 Mb
10:18:02 WARN Allocated memory after GC: 1599 Mb
10:18:02 WARN HeapInusde memory: 1712 Mb
10:18:02 WARN HeapIdle memory: 1471 Mb
10:18:02 WARN HeapReleased memory: 1471 Mb
10:18:02 WARN HeapSys memory: 3184 Mb
10:18:02 WARN Allocated memory from OS: 3323 Mb

[kaypour]

Hmm, I am not sure but afaik runtime.GC() and debug.FreeOSMemory() are not guaranteed to release memory, since Go's GC is conservative, so it keeps memory, even if it is not in use, incase it has to allocate again. (Someone else correct me, if I am wrong. Just my observations)
Does your application crash with out of memory?

[davecheney]

The go runtime can only advise the operating system that a section of
virtual address space is no longer needed, it cannot force the operating
system to take action. Operating systems, especially linux, and especially
if they are not under significant memory pressure can, and regularly do,
ignore this advice.

The best way to determine the memory usage of your application is via
net/http/pprof, or by enabling GODEBUG=gctrace=1

On Fri, Feb 26, 2016 at 10:43 PM, Daniel Kaypour [email protected]
wrote:

Hmm, I am not sure but afaik runtime.GC() and debug.FreeOSMemory() are not
guaranteed to release memory, since Go's GC is conservative, so it keeps
memory, even if it is not in use, incase it has to allocate again. (Someone
else correct me, if I am wrong. Just my observations)
Does your application crash with out of memory?

—
Reply to this email directly or view it on GitHub
#14521 (comment).

[bradfitz]

@Opposition, Go's GC is not conservative. It's a precise GC.

[randall77]

HeapReleased is how much memory Go did give back to the OS. Go has no way of knowing if the OS did actually take the memory back, so it has no stat about that.

HeapIdle includes the HeapReleased stat. So HeapIdle should not go down when Go gives the memory back to the OS.

[rubycut]

Problem is if linux will not take memory back, after some time process get
oom killed.

On Fri, Feb 26, 2016 at 6:04 PM Keith Randall [email protected]
wrote:

HeapReleased is how much memory Go did give back to the OS. Go has no way
of knowing if the OS did actually take the memory back, so it has no stat
about that.

HeapIdle includes the HeapReleased stat. So HeapIdle should not go down
when Go gives the memory back to the OS.

—
Reply to this email directly or view it on GitHub
#14521 (comment).

[ianlancetaylor]

I'm not sure we can make progress on this without an example that we can run ourselves.

[ianlancetaylor]

If possible, you may want to try running your program under truss (or whatever it is called on Darwin) to confirm that the Go program is calling madvise as expected. That is the signal the Go runtime uses to tell the OS that it no longer needs some memory pages.

[tonglil]

Thank you all for your help.

I will run this on Linux and use truss or otherwise to see if I can get the OS to reclaim the memory and reduce the heap idle amount that way.

Please feel free to close issue, I can update and reopen it at a later time with something more concrete for you to go on :)

On Feb 26, 2016, at 9:28 AM, Ian Lance Taylor [email protected] wrote:

If possible, you may want to try running your program under truss (or whatever it is called on Darwin) to confirm that the Go program is calling madvise as expected. That is the signal the Go runtime uses to tell the OS that it no longer needs some memory pages.

—
Reply to this email directly or view it on GitHub.