Write batch crash with OOM error

[jarifibrahim]

What version of Badger are you using?

Master

Does this issue reproduce with the latest master?

Yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

N/A

What did you do?

package main

import (
	"fmt"
	"math/rand"
	"os"

	"github.com/dgraph-io/badger"
)

func write(tx *badger.WriteBatch, n int, vsize int) {
	key := make([]byte, 32)
	val := make([]byte, vsize)
	rand.Read(val)
	for j := 0; j <= n; j++ {
		if _, err := rand.Read(key); err != nil {
			panic(err)
		}
		if err := tx.Set(key, val); err != nil {
			panic(err)
		}
		if j%100 == 0 {
			fmt.Printf("set %d items\n", j)
		}
	}
}

func main() {
	path := "./badger.db.2"
	opts := badger.DefaultOptions(path)
	db, err := badger.Open(opts)
	if err != nil {
		fmt.Println("open badger:", err)
		os.Exit(1)
	}
	bt := db.NewWriteBatch()
	defer bt.Cancel()
	// 2,000  16 mb values
	write(bt, 2000, 16000000)
	fmt.Println("flush")
	if err = bt.Flush(); err != nil {
		fmt.Println("flush err:", err)
	}
	fmt.Println("done")
}

What did you expect to see?

I expected the program to run without any errors.

What did you see instead?

The program crashed with out of memory error.

[jarifibrahim]

The OOM happens when we trying to flush write batch
When we insert data into a transaction, we calculate the size of transaction by using the following function

badger/txn.go

Lines 295 to 300 in b20e8b6

	func (txn *Txn) checkSize(e *Entry) error {
	count := txn.count + 1
	// Extra bytes for version in key.
	size := txn.size + int64(e.estimateSize(txn.db.opt.ValueThreshold)) + 10
	if count >= txn.db.opt.maxBatchCount || size >= txn.db.opt.maxBatchSize {
	return ErrTxnTooBig

which uses estimateSize function

badger/structs.go

Lines 134 to 139 in b20e8b6

	func (e *Entry) estimateSize(threshold int) int {
	if len(e.Value) < threshold {
	return len(e.Key) + len(e.Value) + 2 // Meta, UserMeta
	}
	return len(e.Key) + 12 + 2 // 12 for ValuePointer, 2 for metas.
	}

For a key size of 4 and a value size of 1 MB, the estimateSize function would return 4 + 12 + 2 = 18 byte which is valid in case of SSTs but not for a vlog file. The entry size for vlog would be around 1 MB and when all such entries are packed into a single request for vlog, the bytes.buffer in vlog crashes with OOM error.