Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

new engine: cannot allocate memory #5341

Closed
ewoudwerkman opened this issue Jan 12, 2016 · 3 comments
Closed

new engine: cannot allocate memory #5341

ewoudwerkman opened this issue Jan 12, 2016 · 3 comments

Comments

@ewoudwerkman
Copy link

Hi all,

I'm having issues restarting influxDB after it had run for almost 2 weeks. I'm running influxDB v0.9.6.1 on a Raspberry Pi 2 (4 ARM cores, 1 GB internal memory). The database files are around 733 MB in total. I'm using telegraf to collect stats at a 30s frequency.

I get the following error immediately (from the influxdb.log):

2016/01/12 11:54:41 InfluxDB starting, version 0.9, branch unknown, commit unknown, built unknown
2016/01/12 11:54:41 Go version go1.5.2, GOMAXPROCS set to 4
2016/01/12 11:54:41 Using configuration at: /etc/influxdb/influxdb.conf
[metastore] 2016/01/12 11:54:41 Using data dir: /pi/influxdb/meta
[metastore] 2016/01/12 11:54:41 Node at localhost:8088 [Follower]
[metastore] 2016/01/12 11:54:41 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2016/01/12 11:54:43 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2016/01/12 11:54:43 spun up monitoring for 1
[metastore] 2016/01/12 11:54:43 Updated node id=1 hostname=localhost:8088
[store] 2016/01/12 11:54:43 Using data dir: /pi/influxdb/data
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/14
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/15
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/16
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/17
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/19
[wal] 2016/01/12 11:54:43 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:43 WAL writing to /pi/influxdb/wal/_internal/monitor/21
[wal] 2016/01/12 11:54:44 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:44 WAL writing to /pi/influxdb/wal/_internal/monitor/22
[wal] 2016/01/12 11:54:44 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:44 WAL writing to /pi/influxdb/wal/_internal/
monitor/23
[wal] 2016/01/12 11:54:44 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:44 WAL writing to /pi/influxdb/wal/nodered/default/20
[wal] 2016/01/12 11:54:44 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2016/01/12 11:54:44 WAL writing to /pi/influxdb/wal/telegraf/default/10
[retention] 2016/01/12 11:54:44 retention policy enforcement terminating
[monitor] 2016/01/12 11:54:44 shutting down monitor system
[handoff] 2016/01/12 11:54:44 shutting down hh service
[metastore] 2016/01/12 11:54:44 RPC listener accept error and closed: network connection closed
[metastore] 2016/01/12 11:54:44 exec listener accept error and closed: network connection closed
[subscriber] 2016/01/12 11:54:44 closed service
run: open server: open tsdb store: failed to open shard 18: new engine: cannot allocate memory

Going through the logs i find this error, before systemd tries to restart influxdb:

[http] 2016/01/08 20:16:22 ::1 - - [08/Jan/2016:20:16:22 +0100] POST /write?consistency=&db=telegraf&precision=s&rp= HTTP/1.1 204 0 - InfluxDBClient 4decf8f8-b63c-11e5-8f15-000000000000 65.497332ms
[wal] 2016/01/08 20:16:28 Flush due to idle. Flushing 352 series with 365 points and 6896 bytes from partition 1
unexpected fault address 0x42f00040
fatal error: fault
[signal 0xb code=0x1 addr=0x42f00040 pc=0x35f8c8]

goroutine 47 [running]:
runtime.throw(0x6cf5c8, 0x5)
        /usr/local/go/src/runtime/panic.go:527 +0x78 fp=0x11db1bc4 sp=0x11db1bb8
runtime.sigpanic()
        /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x11db1bf0 sp=0x11db1bc4
github.com/boltdb/bolt.(*Tx).rollback(0x1106a000)
        /home/ewoud/go/src/github.com/boltdb/bolt/tx.go:235 +0x70 fp=0x11db1c04 sp=0x11db1bf4
github.com/boltdb/bolt.(*Tx).Commit(0x1106a000, 0x0, 0x0)
        /home/ewoud/go/src/github.com/boltdb/bolt/tx.go:163 +0x1e8 fp=0x11db1ca0 sp=0x11db1c04
github.com/boltdb/bolt.(*DB).Update(0x10d45c00, 0x11db1ce0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/boltdb/bolt/db.go:559 +0x110 fp=0x11db1ccc sp=0x11db1ca0
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WriteIndex(0x10edaa80, 0x11170020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:263 +0x68 fp=0x11db1cfc sp=0x11db1ccc
github.com/influxdb/influxdb/tsdb/engine/wal.(*Partition).flushAndCompact(0x10efec60, 0x2, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:1032 +0x5ec fp=0x11db1e58 sp=0x11db1cfc
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).triggerAutoFlush(0x10d306e0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:589 +0xf4 fp=0x11db1e98 sp=0x11db1e58
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d306e0, 0x10f7fcc0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:609 +0x158 fp=0x11db1fac sp=0x11db1e98
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4 fp=0x11db1fac sp=0x11db1fac
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 1 [chan receive, 1287 minutes]:
main.(*Main).Run(0x10d84f88, 0x10c62108, 0x2, 0x3, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:96 +0x948
main.main()
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:46 +0x32c

goroutine 17 [syscall, 1287 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4

goroutine 20 [syscall, 1287 minutes]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 73 [IO wait]:
net.runtime_pollWait(0x75304860, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x10f53d78, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x10f53d78, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x10f53d40, 0x10d3b000, 0x1000, 0x1000, 0x0, 0x75343018, 0x10c68000)
        /usr/local/go/src/net/fd_unix.go:232 +0x1c4
net.(*conn).Read(0x11168d60, 0x10d3b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xc8
net/http.(*liveSwitchReader).Read(0x10f29a04, 0x10d3b000, 0x1000, 0x1000, 0x800, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0x9c
io.(*LimitedReader).Read(0x110d1d50, 0x10d3b000, 0x1000, 0x1000, 0x3c3c90, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0x104
bufio.(*Reader).fill(0x10f8f920)
        /usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).Peek(0x10f8f920, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*conn).readRequest(0x10f299e0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:629 +0x34c
net/http.(*conn).serve(0x10f299e0)
        /usr/local/go/src/net/http/server.go:1319 +0x724
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x360

goroutine 35 [IO wait, 1287 minutes]:
net.runtime_pollWait(0x75304ab8, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x10d052f8, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x10d052f8, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).accept(0x10d052c0, 0x0, 0x75347568, 0x10c69d90)
        /usr/local/go/src/net/fd_unix.go:408 +0x21c
net.(*TCPListener).AcceptTCP(0x10d4e290, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:254 +0x4c
net.(*TCPListener).Accept(0x10d4e290, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:264 +0x34
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0x10d52930, 0x75304bb0, 0x10d4e290, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:48 +0x30
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:361 +0x98c

goroutine 36 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e298, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0x10d4d400, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:2334 +0x54
github.com/hashicorp/raft.(*NetworkTransport).listen(0x10d14320)
        /home/ewoud/go/src/github.com/hashicorp/raft/net_transport.go:362 +0x50
created by github.com/hashicorp/raft.NewNetworkTransportWithLogger
        /home/ewoud/go/src/github.com/hashicorp/raft/net_transport.go:154 +0x270

goroutine 37 [select]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0x10d0e7e0)
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:854 +0xb24
github.com/hashicorp/raft.(*Raft).runLeader(0x10d0e7e0)
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:826 +0x8a0
github.com/hashicorp/raft.(*Raft).run(0x10d0e7e0)
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:602 +0xb8
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:252 +0x1c
github.com/hashicorp/raft.(*raftState).goFunc.func1(0x10d0e7e0, 0x10d4e3a8)
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:152 +0x4c
created by github.com/hashicorp/raft.(*raftState).goFunc
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:153 +0x40

goroutine 38 [select, 1137 minutes]:
github.com/hashicorp/raft.(*Raft).runFSM(0x10d0e7e0)
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:509 +0xd5c
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runFSM)-fm()
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:253 +0x1c
github.com/hashicorp/raft.(*raftState).goFunc.func1(0x10d0e7e0, 0x10d4e3b0)
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:152 +0x4c
created by github.com/hashicorp/raft.(*raftState).goFunc
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:153 +0x40

goroutine 39 [select, 2 minutes]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0x10d0e7e0)
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:1642 +0x380
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runSnapshots)-fm()
        /home/ewoud/go/src/github.com/hashicorp/raft/raft.go:254 +0x1c
github.com/hashicorp/raft.(*raftState).goFunc.func1(0x10d0e7e0, 0x10d4e3b8)
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:152 +0x4c
created by github.com/hashicorp/raft.(*raftState).goFunc
        /home/ewoud/go/src/github.com/hashicorp/raft/state.go:153 +0x40

goroutine 40 [select, 1287 minutes]:
github.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0x10d52b10)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/state.go:192 +0x3cc
created by github.com/influxdb/influxdb/meta.(*localRaft).open
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/state.go:182 +0x1110

goroutine 41 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e2a0, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0x10d1c0d0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:743 +0xa0
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:249 +0x298
goroutine 42 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e2a8, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0x10d1c0d0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:853 +0x94
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:252 +0x2e4

goroutine 43 [select]:
github.com/influxdb/influxdb/meta.(*Store).monitorPeerHealth(0x10d1c0d0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:451 +0x244
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:277 +0x584

goroutine 44 [select, 1287 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0x10d0cf30, 0x10d04c40)
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:536 +0x114
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:367 +0xb10

goroutine 45 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30500, 0x10e02480)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 46 [runnable]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d305a0, 0x10ecdb00)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 48 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30780, 0x11019600)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 50 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30a00, 0x1109c540)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 51 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30aa0, 0x1109d940)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 52 [runnable]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30b40, 0x110fa880)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524
goroutine 53 [runnable]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30be0, 0x110fb900)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 54 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30c80, 0x111508c0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 23 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30d20, 0x10d90540)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 24 [runnable]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d30000, 0x110fb800)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

goroutine 25 [select]:
github.com/influxdb/influxdb/tsdb.(*Store).periodicMaintenance(0x10cf21c0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/store.go:341 +0xf0
created by github.com/influxdb/influxdb/tsdb.(*Store).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/store.go:396 +0x31c

goroutine 26 [select, 27 minutes]:
github.com/influxdb/influxdb/services/hh.(*Service).purgeInactiveProcessors(0x10d0d170)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/hh/service.go:227 +0x168
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/hh/service.go:121 +0x634

goroutine 27 [chan receive]:
github.com/influxdb/influxdb/services/subscriber.(*Service).writePoints(0x10d04e00)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/subscriber/service.go:213 +0xa0
created by github.com/influxdb/influxdb/services/subscriber.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/subscriber/service.go:81 +0x148

goroutine 28 [select, 1137 minutes]:
github.com/influxdb/influxdb/meta.(*Store).WaitForDataChanged(0x10d1c0d0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/meta/store.go:536 +0x1ac
github.com/influxdb/influxdb/services/subscriber.(*Service).waitForMetaUpdates(0x10d04e00)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/subscriber/service.go:107 +0x3c
created by github.com/influxdb/influxdb/services/subscriber.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/subscriber/service.go:83 +0x164

goroutine 29 [select]:
github.com/influxdb/influxdb/monitor.(*Monitor).storeStatistics(0x10d0cea0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/monitor/service.go:367 +0xe14
created by github.com/influxdb/influxdb/monitor.(*Monitor).Open
       /home/ewoud/go/src/github.com/influxdb/influxdb/monitor/service.go:142 +0x350

goroutine 7 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e2b0, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/cluster.(*Service).serve(0x10d141e0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/cluster/service.go:98 +0xd0
created by github.com/influxdb/influxdb/cluster.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/cluster/service.go:75 +0x118

goroutine 8 [select, 7 minutes]:
github.com/influxdb/influxdb/services/precreator.(*Service).runPrecreation(0x10d52390)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/precreator/service.go:76 +0x38c
created by github.com/influxdb/influxdb/services/precreator.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/precreator/service.go:54 +0x1f4

goroutine 9 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e2b8, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/services/snapshotter.(*Service).serve(0x10d52420)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:75 +0x84
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:48 +0x108

goroutine 10 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0x10d4e2c0, 0x0, 0x0, 0x0, 0x0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x5c
github.com/influxdb/influxdb/services/copier.(*Service).serve(0x10d52480)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/copier/service.go:78 +0x84
created by github.com/influxdb/influxdb/services/copier.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/copier/service.go:51 +0x108

goroutine 11 [IO wait, 1287 minutes]:
net.runtime_pollWait(0x75304a40, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x1104e9f8, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x1104e9f8, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).accept(0x1104e9c0, 0x0, 0x75347568, 0x1112a100)
        /usr/local/go/src/net/fd_unix.go:408 +0x21c
net.(*TCPListener).AcceptTCP(0x11098030, 0x6a7f0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:254 +0x4c
net.(*TCPListener).Accept(0x11098030, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:264 +0x34
net/http.(*Server).Serve(0x1109c0c0, 0x75304bb0, 0x11098030, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1887 +0x88
net/http.Serve(0x75304bb0, 0x11098030, 0x7530da60, 0x110f6010, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1765 +0x80
github.com/influxdb/influxdb/services/admin.(*Service).serve(0x10d524e0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/admin/service.go:107 +0x12c
created by github.com/influxdb/influxdb/services/admin.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/admin/service.go:70 +0x374

goroutine 12 [select]:
github.com/influxdb/influxdb/services/continuous_querier.(*Service).backgroundLoop(0x10d14230)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:186 +0x49c
created by github.com/influxdb/influxdb/services/continuous_querier.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:120 +0x220

goroutine 13 [IO wait]:
net.runtime_pollWait(0x753049c8, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x1104eb78, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x1104eb78, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).accept(0x1104eb40, 0x0, 0x75347568, 0x110f9830)
        /usr/local/go/src/net/fd_unix.go:408 +0x21c
net.(*TCPListener).AcceptTCP(0x11098068, 0x10c1b6dc, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:254 +0x4c
net.(*TCPListener).Accept(0x11098068, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:264 +0x34
net/http.(*Server).Serve(0x1109c100, 0x75304bb0, 0x11098068, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1887 +0x88
net/http.Serve(0x75304bb0, 0x11098068, 0x7530da78, 0x10d05080, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1765 +0x80
github.com/influxdb/influxdb/services/httpd.(*Service).serve(0x10d526c0)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/httpd/service.go:133 +0x80
created by github.com/influxdb/influxdb/services/httpd.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/httpd/service.go:101 +0x518

goroutine 14 [select, 27 minutes]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShardGroups(0x10d05100)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/retention/service.go:69 +0x2ec
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x188

goroutine 15 [select, 27 minutes]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShards(0x10d05100)
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/retention/service.go:102 +0x858
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/services/retention/service.go:46 +0x1a4

goroutine 16 [chan receive, 1287 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).startServerReporting(0x10d0cf30)
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:477 +0x1b0
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:405 +0x1150

goroutine 66 [select, 1287 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0x10d14050)
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:148 +0x23c
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run
        /home/ewoud/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:130 +0xc1c

goroutine 67 [select, 1287 minutes, locked to thread]:
runtime.gopark(0x7e372c, 0x1110678c, 0x6d2118, 0x6, 0x45718, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x148
runtime.selectgoImpl(0x1110678c, 0x0, 0xc)
        /usr/local/go/src/runtime/select.go:392 +0x14d4
runtime.selectgo(0x1110678c)
        /usr/local/go/src/runtime/select.go:212 +0x10
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x428
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:1036 +0x4

goroutine 31 [IO wait]:
net.runtime_pollWait(0x75304950, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x1109ccb8, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x1109ccb8, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x1109cc80, 0x11120000, 0x1000, 0x1000, 0x0, 0x75343018, 0x10c68000)
        /usr/local/go/src/net/fd_unix.go:232 +0x1c4
net.(*conn).Read(0x11168d40, 0x11120000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xc8
net/http.(*liveSwitchReader).Read(0x10df1224, 0x11120000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0x9c
io.(*LimitedReader).Read(0x1114da80, 0x11120000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0x104
bufio.(*Reader).fill(0x10cab620)
        /usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).ReadSlice(0x10cab620, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:328 +0x264
bufio.(*Reader).ReadLine(0x10cab620, 0x0, 0x0, 0x0, 0x6b3f00, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:357 +0x60
net/textproto.(*Reader).readLineSlice(0x11483ba0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:55 +0x58
net/textproto.(*Reader).ReadLine(0x11483ba0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/textproto/reader.go:36 +0x34
net/http.ReadRequest(0x10cab620, 0x115a0310, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:653 +0x80
net/http.(*conn).readRequest(0x10df1200, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:633 +0x3a8
net/http.(*conn).serve(0x10df1200)
        /usr/local/go/src/net/http/server.go:1319 +0x724
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x360

goroutine 33 [IO wait, 1286 minutes]:
net.runtime_pollWait(0x753047e8, 0x72, 0x10c68000)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0x1109cef8, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x34
net.(*pollDesc).WaitRead(0x1109cef8, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x30
net.(*netFD).Read(0x1109cec0, 0x111fb000, 0x1000, 0x1000, 0x0, 0x75343018, 0x10c68000)
        /usr/local/go/src/net/fd_unix.go:232 +0x1c4
net.(*conn).Read(0x11098a70, 0x111fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xc8
net/http.(*liveSwitchReader).Read(0x10d941a4, 0x111fb000, 0x1000, 0x1000, 0x10c2ace0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:219 +0x9c
io.(*LimitedReader).Read(0x110c9240, 0x111fb000, 0x1000, 0x1000, 0x10c55cf4, 0x0, 0x0)
        /usr/local/go/src/io/io.go:427 +0x104
bufio.(*Reader).fill(0x10cab7a0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1c4
bufio.(*Reader).Peek(0x10cab7a0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*conn).readRequest(0x10d94180, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:629 +0x34c
net/http.(*conn).serve(0x10d94180)
        /usr/local/go/src/net/http/server.go:1319 +0x724
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:1910 +0x360

goroutine 2582 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0x10d780a0, 0x11765a80)
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:604 +0x470
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /home/ewoud/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x524

Any idea what to do to fix this?
@jwilder jwilder added the bz1 label Jan 15, 2016
@jwilder
Copy link
Contributor

jwilder commented Jan 15, 2016

This may be a go 1.5 issue and might be resolved by #5331. You could try the 0.10.0-beta1 release or a current nightly which was built with go 1.4.3.

@ewoudwerkman
Copy link
Author

I tried 0.10.0-beta1 with go 1.4.3, but that doesn't seem to solve the issue. I removed the database in question and restarted. Lets see if it fails again after two weeks ;-)

@jwilder
Copy link
Contributor

jwilder commented Feb 8, 2016

1GB of memory is likely not enough for the your dataset. Increasing your RAM should avoid the cannot allocate memory.

The fatal error: fault panic is related to the bz1 engine which is deprecated and replaced by the tsm engine in 0.10. Issues with the bz1 engine will not be fixed as the solution is to upgrade to 0.10 and migrate shards to tsm using the conversion tool.

@jwilder jwilder closed this as completed Feb 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jwilder @ewoudwerkman