Persistent high memory usage after using the new files API

[matthewrobertbell]

Hi,

I'm running the 64 bit linux 0.4 from the releases page ona 1GB digital ocean server. I added a bunch of small files using the files API (files/mkdir and files/write). After each batch, the memory usage of IPFS increases, and doesn't drop (I waited over 8 hours and it's stable). It's currently at 800MB, which seems excessive. I've stopped adding files, for fear of running out of memory.

I tried running "ipfs repo gc", but memory usage did not decrease.

I have not tried using other commands, so I don't know if the files API is specifically to blame.

The process is currently running, I'm happy to dump whatever info you wish, none of the data I have added is private.

Cheers

[matthewrobertbell]

No files were added after around 11PM. Maybe the high CPU and high memory are related?

[skmgoldin]

I observe similar memory and CPU usage running go-ipfs v0.4.1 on Ubuntu 14.04. It does seem excessive, v0.3.* ran lighter.

[whyrusleeping]

@mattseh I think youre right, the files api isnt cleaning up after itself as it should.

@skmgoldin If youre not using the files API and seeing excessive CPU and memory usage, please gather the debug information discussed here: https://github.com/ipfs/go-ipfs/blob/master/debug-guide.md and send it to me, It will help me find the source of any leaks and memory/cpu hogs

[matthewrobertbell]

The same script seems to be causing rising ram usage on IPFS 0.4.2, although rising much slower than 0.40, with a single threaded script adding a bunch of mostly small files using the files API. It's currently at about 530MB.

CPU usage is 200% (both cores of 2GB VM), but it does die down when files are no longer added, which IIRC is an improvement.

The daemon did crash once, with:

panic: close of nil channel

goroutine 4093840 [running]:
panic(0xc94860, 0xc828caa140)
/home/whyrusleeping/go/src/runtime/panic.go:464 +0x3e6
gx/ipfs/QmaDNZ4QMdBdku1YZWBysufYyoQt1negQGNav6PLYarbY8/go-log.(*bufWriter).loop(0xc830bdbaa0)
/builds/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmaDNZ4QMdBdku1YZWBysufYyoQt1negQGNav6PLYarbY8/go-log/writer.go:233 +0x2f5
created by gx/ipfs/QmaDNZ4QMdBdku1YZWBysufYyoQt1negQGNav6PLYarbY8/go-log.newBufWriter
/builds/distributions/dists/go-ipfs/gopath/src/gx/ipfs/QmaDNZ4QMdBdku1YZWBysufYyoQt1negQGNav6PLYarbY8/go-log/writer.go:151 +0xbf

I will keep the script and daemon running, and report any changes.

[matthewrobertbell]

It is now 692MB. @whyrusleeping or other IPFS team members, you can have access to this VPS to get data if you wish.

[ghost]

Hey @mattseh thanks for reporting this -- check out debug-guide.md for debugging these kinds of issues and providing useful information. Thanks!

[matthewrobertbell]

Here's the debug info, https://ipfs.io/ipfs/QmedpSbkZxUQQpwRiM2eVdGWUCnDZsr51YtRLdv4UdFANv I'll try and look for stuff myself as well.

The IPFS version is 0.4.2 Linux 64 bit official release.

[matthewrobertbell]

I missed the stack dump: https://ipfs.io/ipfs/QmfTmdf8r7g9X6BW3gEwfKrdvPdBFijdSiCDWAMew7aThz As per the debug guide, there's a bunch of goroutines that have been hung for many minutes.

[ghost]

@mattseh thanks for providing these! I linked them in #1924 and we'll continue to track the issue there