Add MmapFlags option for MAP_POPULATE (unix)

[gyuho]

/cc @xiang90

This adds MmapFlags to set syscall.MAP_POPULATE in Linux 2.6.23+.
If true, it would do the sequential read-ahead, as discussed in [1].

Xiang helped me on this PR. I benchmarked the READ performance and with MAP_POPULATE flag,
its read performance increased. I did:

1. Generated random data of 2GB.
2. Clean up page cache echo "echo 1 > /proc/sys/vm/drop_caches" | sudo sh
3. Read with and without MAP_POPULATE and compare two results [2].

Results - SSD, Ubuntu, Intel(R) Core(TM) i7-4910MQ CPU @ 2.90GHz, go version go1.5.1 linux/amd64, Linux kernel version: 3.16.0-52-generic

For 2GB, read takes 8 seconds without MAP_POPULATE, 4 seconds WITH MAP_POPULATE.

- HDD, 1 vCPU, 3.75 GB Memory, Google Cloud, Intel(R) Xeon(R) CPU @ 2.30GHz, go version go1.5.1 linux/amd64, Linux kernel version: 3.16.0-0.bpo.4-amd64

For 2GB, read takes 2 minutes 10 seconds seconds without MAP_POPULATE, 8 seconds WITH MAP_POPULATE.

And I added `MAP_POPULATE` and WITHOUT cleaning up the page cache in a separate machine. This takes about the same as the first test case where we cleaned up page cache.

---

[1]:

etcd-io/etcd#3786

[2]:

package main

import (
    "fmt"
    "time"
    "syscall"

    "github.com/boltdb/bolt"
)

const (
    dbPath     = "fake.db"
    bucketName = "fake-bucket"
    writable   = false
)

func main() {
    st := time.Now()
    // Open the dbPath data file in your current directory.
    // It will be created if it doesn't exist.
        // db, err := bolt.Open(dbPath, 0600, &bolt.Options{Timeout: 5 * time.Minute, ReadOnly: true})
    db, err := bolt.Open(dbPath, 0600, &bolt.Options{Timeout: 5 * time.Minute, ReadOnly: true, MmapFlags: syscall.MAP_POPULATE})
    if err != nil {
        panic(err)
    }
    defer db.Close()

    tx, err := db.Begin(writable)
    if err != nil {
        panic(err)
    }
    defer tx.Rollback()

    bk := tx.Bucket([]byte(bucketName))
    c := bk.Cursor()

    for k, v := c.First(); k != nil; k, v = c.Next() {
        // fmt.Printf("%s ---> %s.\n", k, v)
        _ = k
        _ = v
    }
    fmt.Println("took:", time.Since(st))
}

[xiang90]

@gyuho A more meaningful comparison is to compare readahead enabled with cached read.

[xiang90]

This should not be a bool but a int. So people can set the flag directly.

[xiang90]

Then in the comment we can mention that if you want to read entire database fast, you can set MmapFlag to syscall.MAP_POPULATE on linux.

[benbjohnson]

Overall, lgtm. Just some minor clean up with the OR. The perf bump for spinning disks looks great!

[gyuho]

Thanks! I will clean things up tonight.

[gyuho]

@benbjohnson Just updated as you suggest, and ran the same performance tests, for sample 2GB data:

WITHOUT read-ahead: took: 14.328380492s
WITH read-ahead:    took: 3.130086719s

Thanks,

[xiang90]

Let's remove the issue reference from etcd. It does not make a lot of sense to others.

[gyuho]

@xiang90 Just removed it from the db.go. Thanks

[benbjohnson]

👍 lgtm, thanks for all the tweaks and the benchmarks.

[gyuho]

Thanks @benbjohnson !