Segmentation fault when the disk runs out of space

[whyrusleeping]

What version of Go are you using (go version)?

$ go version
go 1.13.6

What version of Badger are you using?

2

Does this issue reproduce with the latest master?

probably

What are the hardware specifications of the machine (RAM, OS, Disk)?

64GB ram, 1TB nvme

What did you do?

used badger, probably ran out of disk space

What did you expect to see?

no crashes

What did you see instead?

unexpected fault address 0x7f7a96d5bb03
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f7a96d5bb03 pc=0x54e1b5]

goroutine 230 [running]:
runtime.throw(0x1c015dc, 0x5)
	/usr/lib/go/src/runtime/panic.go:774 +0x72 fp=0xc00055c948 sp=0xc00055c918 pc=0x51ca42
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:391 +0x455 fp=0xc00055c978 sp=0xc00055c948 pc=0x532545
runtime.memmove(0xc01faf7300, 0x7f7a96d5a2b5, 0x18ce)
	/usr/lib/go/src/runtime/memmove_amd64.s:330 +0x3c5 fp=0xc00055c980 sp=0xc00055c978 pc=0x54e1b5
github.com/dgraph-io/badger/v2/y.SafeCopy(...)
	/home/why/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/y/y.go:101
github.com/dgraph-io/badger/v2.(*Item).ValueCopy(0xc02064a420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/why/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/iterator.go:129 +0x227 fp=0xc00055ca58 sp=0xc00055c980 pc=0xd183d7
github.com/ipfs/go-ds-badger2.(*txn).get(0xc00055cb78, 0xc020251c20, 0x4c, 0xc020251c20, 0xc020251c27, 0xc00055cb70, 0x45, 0x45)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:484 +0xe1 fp=0xc00055cac8 sp=0xc00055ca58 pc=0xd568a1
github.com/ipfs/go-ds-badger2.(*Datastore).Get(0xc02fa04180, 0xc020251c20, 0x4c, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/datastore.go:277 +0x163 fp=0xc00055cba0 sp=0xc00055cac8 pc=0xd54ac3
github.com/ipfs/go-datastore/keytransform.(*Datastore).Get(0xc0002d6300, 0xc020251bd0, 0x45, 0xc020251bd0, 0x45, 0x3d, 0x3e, 0xc0204ea800)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/keytransform/keytransform.go:47 +0x76 fp=0xc00055cbf0 sp=0xc00055cba0 pc=0x9e2326
github.com/ipfs/go-datastore/keytransform.(*Datastore).Get(0xc0002d6320, 0xc0204ea840, 0x3e, 0xc0204ea840, 0x3e, 0x30, 0x20, 0x2a)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/keytransform/keytransform.go:47 +0x76 fp=0xc00055cc40 sp=0xc00055cbf0 pc=0x9e2326
github.com/ipfs/go-ipfs-blockstore.(*blockstore).Get(0xc0002d63e0, 0xc02995fa10, 0x26, 0x0, 0x0, 0x0, 0x0)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/blockstore.go:123 +0xaf fp=0xc00055cd10 sp=0xc00055cc40 pc=0x9e4d1f
github.com/ipfs/go-ipfs-blockstore.(*idstore).Get(0xc02f96fef0, 0xc02995fa10, 0x26, 0xc033347a70, 0xc02995f9b0, 0x26, 0xc02021db98)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/idstore.go:57 +0xd7 fp=0xc00055cd60 sp=0xc00055cd10 pc=0x9e6007
github.com/ipfs/go-ipfs-blockstore.(*gcBlockstore).Get(0xc000392a80, 0xc02995fa10, 0x26, 0x1f0f020, 0xc033347b00, 0x0, 0x0)
	<autogenerated>:1 +0x50 fp=0xc00055cda8 sp=0xc00055cd60 pc=0x9e6d70
github.com/ipfs/go-bitswap/decision.(*Engine).nextEnvelope(0xc02fa04c00, 0x1f0eca0, 0xc02fa5e480, 0x1, 0x0, 0x0)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/decision/engine.go:206 +0x2c2 fp=0xc00055cee0 sp=0xc00055cda8 pc=0xe63582
github.com/ipfs/go-bitswap/decision.(*Engine).taskWorker(0xc02fa04c00, 0x1f0eca0, 0xc02fa5e480)
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/decision/engine.go:176 +0x1bd fp=0xc00055cfc8 sp=0xc00055cee0 pc=0xe6324d
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00055cfd0 sp=0xc00055cfc8 pc=0x54cc41
created by github.com/ipfs/go-bitswap/decision.NewEngine
	/home/why/go/pkg/mod/github.com/ipfs/[email protected]/decision/engine.go:128 +0x3bb

The rest of the software does a good job of handling the case where the system runs out of disk space, but badger seems to not deal with it that well. If we could get errors returned instead of faulting like this that would be fine.

[jarifibrahim]

Hey @whyrusleeping do you have more badger logs? If the machine was out of disk space, badger would've logged it in the logs (the background tasks don't have a way right now to propagate the error)

[jarifibrahim]

This would be fixed by #1187 .

[jarifibrahim]

This has been fixed via 3e25d77