From a5dbfc80e029c096c29024d2c0b2497fff265ffd Mon Sep 17 00:00:00 2001 From: Ibrahim Jarif Date: Wed, 18 Sep 2019 14:56:32 +0530 Subject: [PATCH] Vendor latest badger (#4007) This change pulls the following commits from badger http://github.com/dgraph-io/badger/commit/cbdef65 Fix windows build for badger http://github.com/dgraph-io/badger/commit/ee70ff2 Store entire L0 in memory http://github.com/dgraph-io/badger/commit/86a77bb Implement PageBuffer http://github.com/dgraph-io/badger/commit/f88aa0c Add FAQ about manifest version mismatch http://github.com/dgraph-io/badger/commits/a1ff348 Minor optimizations http://github.com/dgraph-io/badger/commits/7e99a81 fix small typo ("duing" to "during") --- vendor/github.com/dgraph-io/badger/README.md | 65 +++++-- .../github.com/dgraph-io/badger/appveyor.yml | 2 +- vendor/github.com/dgraph-io/badger/db.go | 42 +++-- vendor/github.com/dgraph-io/badger/go.mod | 6 +- vendor/github.com/dgraph-io/badger/go.sum | 22 +-- .../github.com/dgraph-io/badger/iterator.go | 6 +- .../dgraph-io/badger/level_handler.go | 5 +- vendor/github.com/dgraph-io/badger/levels.go | 35 ++-- .../github.com/dgraph-io/badger/manifest.go | 6 +- vendor/github.com/dgraph-io/badger/options.go | 17 +- .../dgraph-io/badger/stream_writer.go | 3 + .../dgraph-io/badger/table/table.go | 93 ++++++---- vendor/github.com/dgraph-io/badger/value.go | 24 ++- .../github.com/dgraph-io/badger/y/checksum.go | 2 +- .../dgraph-io/badger/y/mmap_windows.go | 3 +- vendor/github.com/dgraph-io/badger/y/y.go | 172 +++++++++++++++++- vendor/vendor.json | 34 ++-- 17 files changed, 413 insertions(+), 124 deletions(-) diff --git a/vendor/github.com/dgraph-io/badger/README.md b/vendor/github.com/dgraph-io/badger/README.md index fe033d9cb58..68d4bfc69a3 100644 --- a/vendor/github.com/dgraph-io/badger/README.md +++ b/vendor/github.com/dgraph-io/badger/README.md @@ -11,7 +11,7 @@ key-value stores like [RocksDB](https://github.com/facebook/rocksdb). Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style bank test runs nightly for -8h, with `--race` flag and ensures maintainance of transactional guarantees. +8h, with `--race` flag and ensures the maintenance of transactional guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure persistence and consistency. @@ -158,7 +158,7 @@ of your application, you have the option to retry the operation if you receive this error. An `ErrTxnTooBig` will be reported in case the number of pending writes/deletes in -the transaction exceed a certain limit. In that case, it is best to commit the +the transaction exceeds a certain limit. In that case, it is best to commit the transaction and start a new transaction immediately. Here is an example (we are not checking for errors in some places for simplicity): @@ -301,7 +301,7 @@ is thread-safe and can be used concurrently via various goroutines. Badger would lease a range of integers to hand out from memory, with the bandwidth provided to `DB.GetSequence`. The frequency at which disk writes are done is determined by this lease bandwidth and the frequency of `Next` -invocations. Setting a bandwith too low would do more disk writes, setting it +invocations. Setting a bandwidth too low would do more disk writes, setting it too high would result in wasted integers if Badger is closed or crashes. To avoid wasted integers, call `Release` before closing Badger. @@ -450,7 +450,7 @@ forward or backward through the keys one at a time. By default, Badger prefetches the values of the next 100 items. You can adjust that with the `IteratorOptions.PrefetchSize` field. However, setting it to -a value higher than GOMAXPROCS (which we recommend to be 128 or higher) +a value higher than `GOMAXPROCS` (which we recommend to be 128 or higher) shouldn’t give any additional benefits. You can also turn off the fetching of values altogether. See section below on key-only iteration. @@ -763,7 +763,7 @@ Below is a list of known projects that use Badger: If you are using Badger in a project please send a pull request to add it to the list. ## Frequently Asked Questions -- **My writes are getting stuck. Why?** +### My writes are getting stuck. Why? **Update: With the new `Value(func(v []byte))` API, this deadlock can no longer happen.** @@ -788,7 +788,7 @@ There are multiple workarounds during iteration: iteration. This might be useful if you just want to delete a lot of keys. 1. Do the writes in a separate transaction after the reads. -- **My writes are really slow. Why?** +### My writes are really slow. Why? Are you creating a new transaction for every single key update, and waiting for it to `Commit` fully before creating a new one? This will lead to very low @@ -813,7 +813,7 @@ handle(wb.Flush()) // Wait for all txns to finish. Note that `WriteBatch` API does not allow any reads. For read-modify-write workloads, you should be using the `Transaction` API. -- **I don't see any disk write. Why?** +### I don't see any disk writes. Why? If you're using Badger with `SyncWrites=false`, then your writes might not be written to value log and won't get synced to disk immediately. Writes to LSM tree are done inmemory first, before they @@ -821,17 +821,17 @@ get compacted to disk. The compaction would only happen once `MaxTableSize` has you're doing a few writes and then checking, you might not see anything on disk. Once you `Close` the database, you'll see these writes on disk. -- **Reverse iteration doesn't give me the right results.** +### Reverse iteration doesn't give me the right results. Just like forward iteration goes to the first key which is equal or greater than the SEEK key, reverse iteration goes to the first key which is equal or lesser than the SEEK key. Therefore, SEEK key would not be part of the results. You can typically add a `0xff` byte as a suffix to the SEEK key to include it in the results. See the following issues: [#436](https://github.com/dgraph-io/badger/issues/436) and [#347](https://github.com/dgraph-io/badger/issues/347). -- **Which instances should I use for Badger?** +### Which instances should I use for Badger? We recommend using instances which provide local SSD storage, without any limit on the maximum IOPS. In AWS, these are storage optimized instances like i3. They provide local SSDs which clock 100K IOPS over 4KB blocks easily. -- **I'm getting a closed channel error. Why?** +### I'm getting a closed channel error. Why? ``` panic: close of closed channel @@ -840,17 +840,50 @@ panic: send on closed channel If you're seeing panics like above, this would be because you're operating on a closed DB. This can happen, if you call `Close()` before sending a write, or multiple times. You should ensure that you only call `Close()` once, and all your read/write operations finish before closing. -- **Are there any Go specific settings that I should use?** +### Are there any Go specific settings that I should use? -We *highly* recommend setting a high number for GOMAXPROCS, which allows Go to +We *highly* recommend setting a high number for `GOMAXPROCS`, which allows Go to observe the full IOPS throughput provided by modern SSDs. In Dgraph, we have set it to 128. For more details, [see this thread](https://groups.google.com/d/topic/golang-nuts/jPb_h3TvlKE/discussion). -- **Are there any linux specific settings that I should use?** - -We recommend setting max file descriptors to a high number depending upon the expected size of you data. - +### Are there any Linux specific settings that I should use? + +We recommend setting `max file descriptors` to a high number depending upon the expected size of +your data. On Linux and Mac, you can check the file descriptor limit with `ulimit -n -H` for the +hard limit and `ulimit -n -S` for the soft limit. A soft limit of `65535` is a good lower bound. +You can adjust the limit as needed. + +### I see "manifest has unsupported version: X (we support Y)" error. + +This error means you have a badger directory which was created by an older version of badger and +you're trying to open in a newer version of badger. The underlying data format can change across +badger versions and users will have to migrate their data directory. +Badger data can be migrated from version X of badger to version Y of badger by following the steps +listed below. +Assume you were on badger v1.5.5 and you wish to migrate to v2.0.0 version +1. Install badger version v1.5.5. + - `cd $GOPATH/src/github.com/dgraph-io/badger` + - `git checkout v1.5.5` + - `cd badger && go install` + + This should install the old badger binary in your $GOBIN. +2. Create Backup + - `badger backup --dir path/to/badger/directory -f badger.backup` +3. Install badger version v2.0.0 + - `cd $GOPATH/src/github.com/dgraph-io/badger` + - `git checkout v2.0.0` + - `cd badger && go install` + + This should install new badger binary in your $GOBIN +4. Install badger version v2.0.0 + - `badger restore --dir path/to/new/badger/directory -f badger.backup` + + This will create a new directory on `path/to/new/badger/directory` and add badger data in + newer format to it. + +NOTE - The above steps shouldn't cause any data loss but please ensure the new data is valid before +deleting the old badger directory. ## Contact - Please use [discuss.dgraph.io](https://discuss.dgraph.io) for questions, feature requests and discussions. - Please use [Github issue tracker](https://github.com/dgraph-io/badger/issues) for filing bugs or feature requests. diff --git a/vendor/github.com/dgraph-io/badger/appveyor.yml b/vendor/github.com/dgraph-io/badger/appveyor.yml index afa54ca0ad0..36853e9d1fd 100644 --- a/vendor/github.com/dgraph-io/badger/appveyor.yml +++ b/vendor/github.com/dgraph-io/badger/appveyor.yml @@ -11,7 +11,7 @@ clone_folder: c:\gopath\src\github.com\dgraph-io\badger # Environment variables environment: - GOVERSION: 1.8.3 + GOVERSION: 1.12 GOPATH: c:\gopath GO111MODULE: on diff --git a/vendor/github.com/dgraph-io/badger/db.go b/vendor/github.com/dgraph-io/badger/db.go index a3703d45196..164546f8f8c 100644 --- a/vendor/github.com/dgraph-io/badger/db.go +++ b/vendor/github.com/dgraph-io/badger/db.go @@ -21,7 +21,6 @@ import ( "context" "encoding/binary" "expvar" - "io" "math" "os" "path/filepath" @@ -200,6 +199,8 @@ func Open(opt Options) (db *DB, err error) { return nil, errors.Errorf("Valuethreshold greater than max batch size of %d. Either "+ "reduce opt.ValueThreshold or increase opt.MaxTableSize.", opt.maxBatchSize) } + // Compact L0 on close if either it is set or if KeepL0InMemory is set. + opt.CompactL0OnClose = opt.CompactL0OnClose || opt.KeepL0InMemory if opt.ReadOnly { // Can't truncate if the DB is read only. @@ -846,8 +847,8 @@ func arenaSize(opt Options) int64 { return opt.MaxTableSize + opt.maxBatchSize + opt.maxBatchCount*int64(skl.MaxNodeSize) } -// WriteLevel0Table flushes memtable. -func writeLevel0Table(ft flushTask, f io.Writer, bopts table.Options) error { +// buildL0Table builds a new table from the memtable. +func buildL0Table(ft flushTask, bopts table.Options) []byte { iter := ft.mt.NewIterator() defer iter.Close() b := table.NewTableBuilder(bopts) @@ -858,8 +859,7 @@ func writeLevel0Table(ft flushTask, f io.Writer, bopts table.Options) error { } b.Add(iter.Key(), iter.Value()) } - _, err := f.Write(b.Finish()) - return err + return b.Finish() } type flushTask struct { @@ -886,28 +886,36 @@ func (db *DB) handleFlushTask(ft flushTask) error { headTs := y.KeyWithTs(head, db.orc.nextTs()) ft.mt.Put(headTs, y.ValueStruct{Value: val}) + bopts := table.Options{ + BlockSize: db.opt.BlockSize, + BloomFalsePositive: db.opt.BloomFalsePositive, + } + tableData := buildL0Table(ft, bopts) + fileID := db.lc.reserveFileID() + if db.opt.KeepL0InMemory { + tbl, err := table.OpenInMemoryTable(tableData, fileID) + if err != nil { + return errors.Wrapf(err, "failed to open table in memory") + } + return db.lc.addLevel0Table(tbl) + } + fd, err := y.CreateSyncedFile(table.NewFilename(fileID, db.opt.Dir), true) if err != nil { return y.Wrap(err) } // Don't block just to sync the directory entry. - dirSyncCh := make(chan error) + dirSyncCh := make(chan error, 1) go func() { dirSyncCh <- syncDir(db.opt.Dir) }() - bopts := table.Options{ - BlockSize: db.opt.BlockSize, - BloomFalsePositive: db.opt.BloomFalsePositive, - } - err = writeLevel0Table(ft, fd, bopts) - dirSyncErr := <-dirSyncCh - - if err != nil { + if _, err = fd.Write(tableData); err != nil { db.elog.Errorf("ERROR while writing to level 0: %v", err) return err } - if dirSyncErr != nil { + + if dirSyncErr := <-dirSyncCh; dirSyncErr != nil { // Do dir sync as best effort. No need to return due to an error there. db.elog.Errorf("ERROR while syncing level directory: %v", dirSyncErr) } @@ -922,7 +930,7 @@ func (db *DB) handleFlushTask(ft flushTask) error { return err } // We own a ref on tbl. - err = db.lc.addLevel0Table(tbl) // This will incrRef (if we don't error, sure) + err = db.lc.addLevel0Table(tbl) // This will incrRef _ = tbl.DecrRef() // Releases our ref. return err } @@ -1030,7 +1038,7 @@ func (db *DB) updateSize(lc *y.Closer) { // RunValueLogGC triggers a value log garbage collection. // // It picks value log files to perform GC based on statistics that are collected -// duing compactions. If no such statistics are available, then log files are +// during compactions. If no such statistics are available, then log files are // picked in random order. The process stops as soon as the first log file is // encountered which does not result in garbage collection. // diff --git a/vendor/github.com/dgraph-io/badger/go.mod b/vendor/github.com/dgraph-io/badger/go.mod index d4a9679f0bb..bbdbd1bd555 100644 --- a/vendor/github.com/dgraph-io/badger/go.mod +++ b/vendor/github.com/dgraph-io/badger/go.mod @@ -4,13 +4,17 @@ go 1.12 require ( github.com/cespare/xxhash v1.1.0 + github.com/cespare/xxhash/v2 v2.1.0 // indirect github.com/dgraph-io/ristretto v0.0.0-20190903064322-eb48d2f7ca30 github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 github.com/dustin/go-humanize v1.0.0 github.com/golang/protobuf v1.3.1 github.com/pkg/errors v0.8.1 + github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/spf13/cobra v0.0.5 - github.com/stretchr/testify v1.3.0 + github.com/stretchr/objx v0.2.0 // indirect + github.com/stretchr/testify v1.4.0 golang.org/x/net v0.0.0-20190620200207-3b0461eec859 golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb + gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect ) diff --git a/vendor/github.com/dgraph-io/badger/go.sum b/vendor/github.com/dgraph-io/badger/go.sum index 1b35a2c416f..f770c2ba616 100644 --- a/vendor/github.com/dgraph-io/badger/go.sum +++ b/vendor/github.com/dgraph-io/badger/go.sum @@ -1,71 +1,61 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= -github.com/OneOfOne/xxhash v1.2.5 h1:zl/OfRA6nftbBK9qTohYBJ5xvw6C/oNKizR7cZGl3cI= github.com/OneOfOne/xxhash v1.2.5/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q= github.com/VictoriaMetrics/fastcache v1.5.1/go.mod h1:+jv9Ckb+za/P1ZRg/sulP5Ni1v49daAVERr0H3CuscE= github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156/go.mod h1:Cb/ax3seSYIx7SuZdm2G2xzfwmv3TPSk2ucNfQESPXM= github.com/allegro/bigcache v1.2.1/go.mod h1:Cb/ax3seSYIx7SuZdm2G2xzfwmv3TPSk2ucNfQESPXM= github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= -github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.0.1-0.20190104013014-3767db7a7e18/go.mod h1:HD5P3vAIAh+Y2GAxg0PrPN1P8WkepXGpjbUPDHJqqKM= +github.com/cespare/xxhash/v2 v2.1.0/go.mod h1:dgIUBU3pDso/gPgZ1osOZ0iQf77oPR28Tjxl5dIMyVM= github.com/coocood/freecache v1.1.0/go.mod h1:ePwxCDzOYvARfHdr1pByNct1at3CoKnsipOHwKlNbzI= github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgraph-io/ristretto v0.0.0-20190903064322-eb48d2f7ca30 h1:2ymRTEtPu60vzv+YTIHHlJ3713Bl3G/MPc2BOc/OkzM= +github.com/dgraph-io/ristretto v0.0.0-20190903064322-eb48d2f7ca30 h1:FkdGlqxPjfHKdVUzS5dOSMeOkGjEG7PnR1RLbcEqw88= github.com/dgraph-io/ristretto v0.0.0-20190903064322-eb48d2f7ca30/go.mod h1:UvZmzj8odp3S1nli6yEb1vLME8iJFBrRcw8rAJEiu9Q= -github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA= github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= -github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/goburrow/cache v0.1.0/go.mod h1:8oxkfud4hvjO4tNjEKZfEd+LrpDVDlBIauGYsWGEzio= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= -github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= -github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= -github.com/spaolacci/murmur3 v1.0.1-0.20190317074736-539464a789e9 h1:5Cp3cVwpQP4aCQ6jx6dNLP3IarbYiuStmIzYu+BjQwY= github.com/spaolacci/murmur3 v1.0.1-0.20190317074736-539464a789e9/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= +github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= -github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s= github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= -github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= -github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb h1:fgwFCsaw9buMuxNd6+DQfAuSFqbNiQZpcgJQAgJsK6k= golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/vendor/github.com/dgraph-io/badger/iterator.go b/vendor/github.com/dgraph-io/badger/iterator.go index d9d135bea4d..78ff7f65503 100644 --- a/vendor/github.com/dgraph-io/badger/iterator.go +++ b/vendor/github.com/dgraph-io/badger/iterator.go @@ -27,6 +27,7 @@ import ( "github.com/dgraph-io/badger/options" "github.com/dgraph-io/badger/table" + "github.com/dgryski/go-farm" "github.com/dgraph-io/badger/y" ) @@ -361,7 +362,7 @@ func (opt *IteratorOptions) pickTable(t table.TableInterface) bool { } // Bloom filter lookup would only work if opt.Prefix does NOT have the read // timestamp as part of the key. - if opt.prefixIsKey && t.DoesNotHave(opt.Prefix) { + if opt.prefixIsKey && t.DoesNotHave(farm.Fingerprint64(opt.Prefix)) { return false } return true @@ -394,6 +395,7 @@ func (opt *IteratorOptions) pickTables(all []*table.Table) []*table.Table { } var out []*table.Table + hash := farm.Fingerprint64(opt.Prefix) for _, t := range filtered { // When we encounter the first table whose smallest key is higher than // opt.Prefix, we can stop. @@ -402,7 +404,7 @@ func (opt *IteratorOptions) pickTables(all []*table.Table) []*table.Table { } // opt.Prefix is actually the key. So, we can run bloom filter checks // as well. - if t.DoesNotHave(opt.Prefix) { + if t.DoesNotHave(hash) { continue } out = append(out, t) diff --git a/vendor/github.com/dgraph-io/badger/level_handler.go b/vendor/github.com/dgraph-io/badger/level_handler.go index e30e4c842c5..310afaa2442 100644 --- a/vendor/github.com/dgraph-io/badger/level_handler.go +++ b/vendor/github.com/dgraph-io/badger/level_handler.go @@ -21,6 +21,8 @@ import ( "sort" "sync" + "github.com/dgryski/go-farm" + "github.com/dgraph-io/badger/table" "github.com/dgraph-io/badger/y" "github.com/pkg/errors" @@ -231,9 +233,10 @@ func (s *levelHandler) get(key []byte) (y.ValueStruct, error) { tables, decr := s.getTableForKey(key) keyNoTs := y.ParseKey(key) + hash := farm.Fingerprint64(keyNoTs) var maxVs y.ValueStruct for _, th := range tables { - if th.DoesNotHave(keyNoTs) { + if th.DoesNotHave(hash) { y.NumLSMBloomHits.Add(s.strLevel, 1) continue } diff --git a/vendor/github.com/dgraph-io/badger/levels.go b/vendor/github.com/dgraph-io/badger/levels.go index 0deaab59b33..f23ec8c1b52 100644 --- a/vendor/github.com/dgraph-io/badger/levels.go +++ b/vendor/github.com/dgraph-io/badger/levels.go @@ -52,7 +52,7 @@ var ( ) // revertToManifest checks that all necessary table files exist and removes all table files not -// referenced by the manifest. idMap is a set of table file id's that were read from the directory +// referenced by the manifest. idMap is a set of table file id's that were read from the directory // listing. func revertToManifest(kv *DB, mf *Manifest, idMap map[uint64]struct{}) error { // 1. Check all files in manifest exist. @@ -237,7 +237,10 @@ func (s *levelsController) dropTree() (int, error) { // Generate the manifest changes. changes := []*pb.ManifestChange{} for _, table := range all { - changes = append(changes, newDeleteChange(table.ID())) + // Add a delete change only if the table is not in memory. + if !table.IsInmemory { + changes = append(changes, newDeleteChange(table.ID())) + } } changeSet := pb.ManifestChangeSet{Changes: changes} if err := s.kv.manifest.addChanges(changeSet.Changes); err != nil { @@ -647,7 +650,10 @@ func buildChangeSet(cd *compactDef, newTables []*table.Table) pb.ManifestChangeS newCreateChange(table.ID(), cd.nextLevel.level)) } for _, table := range cd.top { - changes = append(changes, newDeleteChange(table.ID())) + // Add a delete change only if the table is not in memory. + if !table.IsInmemory { + changes = append(changes, newDeleteChange(table.ID())) + } } for _, table := range cd.bot { changes = append(changes, newDeleteChange(table.ID())) @@ -869,15 +875,19 @@ func (s *levelsController) doCompact(p compactionPriority) error { } func (s *levelsController) addLevel0Table(t *table.Table) error { - // We update the manifest _before_ the table becomes part of a levelHandler, because at that - // point it could get used in some compaction. This ensures the manifest file gets updated in - // the proper order. (That means this update happens before that of some compaction which - // deletes the table.) - err := s.kv.manifest.addChanges([]*pb.ManifestChange{ - newCreateChange(t.ID(), 0), - }) - if err != nil { - return err + // Add table to manifest file only if it is not opened in memory. We don't want to add a table + // to the manifest file if it exists only in memory. + if !t.IsInmemory { + // We update the manifest _before_ the table becomes part of a levelHandler, because at that + // point it could get used in some compaction. This ensures the manifest file gets updated in + // the proper order. (That means this update happens before that of some compaction which + // deletes the table.) + err := s.kv.manifest.addChanges([]*pb.ManifestChange{ + newCreateChange(t.ID(), 0), + }) + if err != nil { + return err + } } for !s.levels[0].tryAddLevel0Table(t) { @@ -993,6 +1003,7 @@ func (s *levelsController) getTableInfo(withKeysCount bool) (result []TableInfo) for it.Rewind(); it.Valid(); it.Next() { count++ } + it.Close() } info := TableInfo{ diff --git a/vendor/github.com/dgraph-io/badger/manifest.go b/vendor/github.com/dgraph-io/badger/manifest.go index cdcf8be0e32..9c07ba7bdea 100644 --- a/vendor/github.com/dgraph-io/badger/manifest.go +++ b/vendor/github.com/dgraph-io/badger/manifest.go @@ -344,7 +344,11 @@ func ReplayManifestFile(fp *os.File) (ret Manifest, truncOffset int64, err error version := y.BytesToU32(magicBuf[4:8]) if version != magicVersion { return Manifest{}, 0, - fmt.Errorf("manifest has unsupported version: %d (we support %d)", version, magicVersion) + //nolint:lll + fmt.Errorf("manifest has unsupported version: %d (we support %d).\n"+ + "Please see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error"+ + " on how to fix this.", + version, magicVersion) } build := createManifest() diff --git a/vendor/github.com/dgraph-io/badger/options.go b/vendor/github.com/dgraph-io/badger/options.go index 3a4f6cfdf21..fc72b95801d 100644 --- a/vendor/github.com/dgraph-io/badger/options.go +++ b/vendor/github.com/dgraph-io/badger/options.go @@ -54,6 +54,7 @@ type Options struct { NumMemtables int BlockSize int BloomFalsePositive float64 + KeepL0InMemory bool NumLevelZeroTables int NumLevelZeroTablesStall int @@ -104,6 +105,7 @@ func DefaultOptions(path string) Options { SyncWrites: true, NumVersionsToKeep: 1, CompactL0OnClose: true, + KeepL0InMemory: true, // Nothing to read/write value log using standard File I/O // MemoryMap to mmap() the value log files // (2^30 - 1)*2 when mmapping < 2^31 - 1, max int32. @@ -387,7 +389,7 @@ func (opt Options) WithNumCompactors(val int) Options { // // CompactL0OnClose determines whether Level 0 should be compacted before closing the DB. // This ensures that both reads and writes are efficient when the DB is opened later. -// +// CompactL0OnClose is set to true if KeepL0InMemory is set to true. // The default value of CompactL0OnClose is true. func (opt Options) WithCompactL0OnClose(val bool) Options { opt.CompactL0OnClose = val @@ -407,3 +409,16 @@ func (opt Options) WithLogRotatesToFlush(val int32) Options { opt.LogRotatesToFlush = val return opt } + +// WithKeepL0InMemory returns a new Options value with KeepL0InMemory set to the given value. +// +// When KeepL0InMemory is set to true we will keep all Level 0 tables in memory. This leads to +// better performance in writes as well as compactions. In case of DB crash, the value log replay +// will take longer to complete since memtables and all level 0 tables will have to be recreated. +// This option also sets CompactL0OnClose option to true. +// +// The default value of KeepL0InMemory is true. +func (opt Options) WithKeepL0InMemory(val bool) Options { + opt.KeepL0InMemory = val + return opt +} diff --git a/vendor/github.com/dgraph-io/badger/stream_writer.go b/vendor/github.com/dgraph-io/badger/stream_writer.go index 730d78f219d..e763f1b0aaa 100644 --- a/vendor/github.com/dgraph-io/badger/stream_writer.go +++ b/vendor/github.com/dgraph-io/badger/stream_writer.go @@ -331,6 +331,7 @@ func (w *sortedWriter) createTable(data []byte) error { if _, err := fd.Write(data); err != nil { return err } + opts := table.Options{ LoadingMode: w.db.opt.TableLoadingMode, ChkMode: w.db.opt.ChecksumVerificationMode, @@ -375,6 +376,8 @@ func (w *sortedWriter) createTable(data []byte) error { if err := lhandler.replaceTables([]*table.Table{}, []*table.Table{tbl}); err != nil { return err } + // Release the ref held by OpenTable. + _ = tbl.DecrRef() w.db.opt.Infof("Table created: %d at level: %d for stream: %d. Size: %s\n", fileID, lhandler.level, w.streamId, humanize.Bytes(uint64(tbl.Size()))) return nil diff --git a/vendor/github.com/dgraph-io/badger/table/table.go b/vendor/github.com/dgraph-io/badger/table/table.go index 859ffd77d0d..f808d06b1f4 100644 --- a/vendor/github.com/dgraph-io/badger/table/table.go +++ b/vendor/github.com/dgraph-io/badger/table/table.go @@ -27,7 +27,6 @@ import ( "sync" "sync/atomic" - "github.com/dgryski/go-farm" "github.com/golang/protobuf/proto" "github.com/pkg/errors" @@ -62,7 +61,7 @@ type Options struct { type TableInterface interface { Smallest() []byte Biggest() []byte - DoesNotHave(key []byte) bool + DoesNotHave(hash uint64) bool } // Table represents a loaded table file with the info we have about it @@ -84,7 +83,8 @@ type Table struct { bf *z.Bloom Checksum []byte - opt *Options + IsInmemory bool // Set to true if the table is on level 0 and opened in memory. + opt *Options } // IncrRef increments the refcount (having to do with whether the file should be deleted) @@ -99,11 +99,17 @@ func (t *Table) DecrRef() error { // We can safely delete this file, because for all the current files, we always have // at least one reference pointing to them. - // It's necessary to delete windows files + // It's necessary to delete windows files. if t.opt.LoadingMode == options.MemoryMap { if err := y.Munmap(t.mmap); err != nil { return err } + t.mmap = nil + } + // fd can be nil if the table belongs to L0 and it is opened in memory. See + // OpenTableInMemory method. + if t.fd == nil { + return nil } if err := t.fd.Truncate(0); err != nil { // This is very important to let the FS know that the file is deleted. @@ -157,32 +163,15 @@ func OpenTable(fd *os.File, opts Options) (*Table, error) { return nil, errors.Errorf("Invalid filename: %s", filename) } t := &Table{ - fd: fd, - ref: 1, // Caller is given one reference. - id: id, - opt: &opts, + fd: fd, + ref: 1, // Caller is given one reference. + id: id, + opt: &opts, + IsInmemory: false, } t.tableSize = int(fileInfo.Size()) - if err := t.readIndex(); err != nil { - return nil, y.Wrap(err) - } - - it := t.NewIterator(false) - defer it.Close() - it.Rewind() - if it.Valid() { - t.smallest = it.Key() - } - - it2 := t.NewIterator(true) - defer it2.Close() - it2.Rewind() - if it2.Valid() { - t.biggest = it2.Key() - } - switch opts.LoadingMode { case options.LoadToRAM: if _, err := t.fd.Seek(0, io.SeekStart); err != nil { @@ -211,24 +200,64 @@ func OpenTable(fd *os.File, opts Options) (*Table, error) { panic(fmt.Sprintf("Invalid loading mode: %v", opts.LoadingMode)) } - if t.opt.ChkMode == options.OnTableRead || t.opt.ChkMode == options.OnTableAndBlockRead { + if err := t.initBiggestAndSmallest(); err != nil { + return nil, errors.Wrapf(err, "failed to initialize table") + } + if opts.ChkMode == options.OnTableRead || opts.ChkMode == options.OnTableAndBlockRead { if err := t.VerifyChecksum(); err != nil { _ = fd.Close() - return nil, err + return nil, errors.Wrapf(err, "failed to verify checksum") } } return t, nil } +// OpenInMemoryTable is similar to OpenTable but it opens a new table from the provided data. +// OpenInMemoryTable is used for L0 tables. +func OpenInMemoryTable(data []byte, id uint64) (*Table, error) { + t := &Table{ + ref: 1, // Caller is given one reference. + opt: &Options{LoadingMode: options.LoadToRAM}, + mmap: data, + tableSize: len(data), + IsInmemory: true, + id: id, // It is important that each table gets a unique ID. + } + + if err := t.initBiggestAndSmallest(); err != nil { + return nil, err + } + return t, nil +} + +func (t *Table) initBiggestAndSmallest() error { + if err := t.readIndex(); err != nil { + return errors.Wrapf(err, "failed to read index.") + } + + t.smallest = t.blockIndex[0].Key + + it2 := t.NewIterator(true) + defer it2.Close() + it2.Rewind() + if it2.Valid() { + t.biggest = it2.Key() + } + return nil +} + // Close closes the open table. (Releases resources back to the OS.) func (t *Table) Close() error { if t.opt.LoadingMode == options.MemoryMap { if err := y.Munmap(t.mmap); err != nil { return err } + t.mmap = nil + } + if t.fd == nil { + return nil } - return t.fd.Close() } @@ -348,9 +377,9 @@ func (t *Table) Filename() string { return t.fd.Name() } // ID is the table's ID number (used to make the file name). func (t *Table) ID() uint64 { return t.id } -// DoesNotHave returns true if (but not "only if") the table does not have the key. It does a -// bloom filter lookup. -func (t *Table) DoesNotHave(key []byte) bool { return !t.bf.Has(farm.Fingerprint64(key)) } +// DoesNotHave returns true if (but not "only if") the table does not have the key hash. +// It does a bloom filter lookup. +func (t *Table) DoesNotHave(hash uint64) bool { return !t.bf.Has(hash) } // VerifyChecksum verifies checksum for all blocks of table. This function is called by // OpenTable() function. This function is also called inside levelsController.VerifyChecksum(). diff --git a/vendor/github.com/dgraph-io/badger/value.go b/vendor/github.com/dgraph-io/badger/value.go index 2478f1935f1..8917aaf54f9 100644 --- a/vendor/github.com/dgraph-io/badger/value.go +++ b/vendor/github.com/dgraph-io/badger/value.go @@ -97,9 +97,13 @@ func (lf *logFile) munmap() (err error) { // Nothing to do return nil } + if err := y.Munmap(lf.fmap); err != nil { return errors.Wrapf(err, "Unable to munmap value log: %q", lf.path) } + // This is important. We should set the map to nil because ummap + // system call doesn't change the length or capacity of the fmap slice. + lf.fmap = nil return nil } @@ -137,12 +141,22 @@ func (lf *logFile) doneWriting(offset uint32) error { return errors.Wrapf(err, "Unable to sync value log: %q", lf.path) } + // Unmap file before we truncate it. Windows cannot truncate a file that is mmapped. + if err := lf.munmap(); err != nil { + return errors.Wrapf(err, "failed to mumap vlog file %s", lf.fd.Name()) + } + // TODO: Confirm if we need to run a file sync after truncation. // Truncation must run after unmapping, otherwise Windows would crap itself. if err := lf.fd.Truncate(int64(offset)); err != nil { return errors.Wrapf(err, "Unable to truncate file: %q", lf.path) } + // Reinitialize the log file. This will mmap the entire file. + if err := lf.init(); err != nil { + return errors.Wrapf(err, "failed to initialize file %s", lf.fd.Name()) + } + // Previously we used to close the file after it was written and reopen it in read-only mode. // We no longer open files in read-only mode. We keep all vlog files open in read-write mode. return nil @@ -722,12 +736,12 @@ func (vlog *valueLog) createVlogFile(fid uint32) (*logFile, error) { if err = syncDir(vlog.dirPath); err != nil { return nil, errFile(err, vlog.dirPath, "Sync value log dir") } - if err = lf.mmap(2 * vlog.opt.ValueLogFileSize); err != nil { - return nil, errFile(err, lf.path, "Mmap value log file") - } if err = bootstrapLogfile(lf.fd); err != nil { return nil, err } + if err = lf.mmap(2 * vlog.opt.ValueLogFileSize); err != nil { + return nil, errFile(err, lf.path, "Mmap value log file") + } // writableLogOffset is only written by write func, by read by Read func. // To avoid a race condition, all reads and updates to this variable must be @@ -848,6 +862,10 @@ func (vlog *valueLog) open(db *DB, ptr valuePointer, replayFn logEntry) error { // Log file is corrupted. Delete it. if err == errDeleteVlogFile { delete(vlog.filesMap, fid) + // Close the fd of the file before deleting the file otherwise windows complaints. + if err := lf.fd.Close(); err != nil { + return errors.Wrapf(err, "failed to close vlog file %s", lf.fd.Name()) + } path := vlog.fpath(lf.fid) if err := os.Remove(path); err != nil { return y.Wrapf(err, "failed to delete empty value log file: %q", path) diff --git a/vendor/github.com/dgraph-io/badger/y/checksum.go b/vendor/github.com/dgraph-io/badger/y/checksum.go index 6bceada4c5e..d7993542ca5 100644 --- a/vendor/github.com/dgraph-io/badger/y/checksum.go +++ b/vendor/github.com/dgraph-io/badger/y/checksum.go @@ -44,7 +44,7 @@ func CalculateChecksum(data []byte, ct pb.Checksum_Algorithm) uint64 { func VerifyChecksum(data []byte, expected *pb.Checksum) error { actual := CalculateChecksum(data, expected.Algo) if actual != expected.Sum { - return Wrapf(ErrChecksumMismatch, "actual: %d, expected: %d", actual, expected) + return Wrapf(ErrChecksumMismatch, "actual: %d, expected: %d", actual, expected.Sum) } return nil } diff --git a/vendor/github.com/dgraph-io/badger/y/mmap_windows.go b/vendor/github.com/dgraph-io/badger/y/mmap_windows.go index 0efb2d0f8dc..b1f64c21f36 100644 --- a/vendor/github.com/dgraph-io/badger/y/mmap_windows.go +++ b/vendor/github.com/dgraph-io/badger/y/mmap_windows.go @@ -38,7 +38,8 @@ func Mmap(fd *os.File, write bool, size int64) ([]byte, error) { return nil, err } - // Truncate the database to the size of the mmap. + // In windows, we cannot mmap a file more than it's actual size. + // So truncate the file to the size of the mmap. if fi.Size() < size { if err := fd.Truncate(size); err != nil { return nil, fmt.Errorf("truncate: %s", err) diff --git a/vendor/github.com/dgraph-io/badger/y/y.go b/vendor/github.com/dgraph-io/badger/y/y.go index 87facc42d17..df3b8520c0b 100644 --- a/vendor/github.com/dgraph-io/badger/y/y.go +++ b/vendor/github.com/dgraph-io/badger/y/y.go @@ -21,6 +21,7 @@ import ( "encoding/binary" "fmt" "hash/crc32" + "io" "math" "os" "reflect" @@ -128,7 +129,6 @@ func ParseTs(key []byte) uint64 { // a would be sorted higher than aa if we use bytes.compare // All keys should have timestamp. func CompareKeys(key1, key2 []byte) int { - AssertTrue(len(key1) > 8 && len(key2) > 8) if cmp := bytes.Compare(key1[:len(key1)-8], key2[:len(key2)-8]); cmp != 0 { return cmp } @@ -141,7 +141,6 @@ func ParseKey(key []byte) []byte { return nil } - AssertTrue(len(key) > 8) return key[:len(key)-8] } @@ -340,3 +339,172 @@ func BytesToU32Slice(b []byte) []uint32 { hdr.Data = uintptr(unsafe.Pointer(&b[0])) return u32s } + +// page struct contains one underlying buffer. +type page struct { + buf []byte +} + +// PageBuffer consists of many pages. A page is a wrapper over []byte. PageBuffer can act as a +// replacement of bytes.Buffer. Instead of having single underlying buffer, it has multiple +// underlying buffers. Hence it avoids any copy during relocation(as happens in bytes.Buffer). +// PageBuffer allocates memory in pages. Once a page is full, it will allocate page with double the +// size of previous page. Its function are not thread safe. +type PageBuffer struct { + pages []*page + + length int // Length of PageBuffer. + nextPageSize int // Size of next page to be allocated. +} + +// NewPageBuffer returns a new PageBuffer with first page having size pageSize. +func NewPageBuffer(pageSize int) *PageBuffer { + b := &PageBuffer{} + b.pages = append(b.pages, &page{buf: make([]byte, 0, pageSize)}) + b.nextPageSize = pageSize * 2 + return b +} + +// Write writes data to PageBuffer b. It returns number of bytes written and any error encountered. +func (b *PageBuffer) Write(data []byte) (int, error) { + dataLen := len(data) + for { + cp := b.pages[len(b.pages)-1] // Current page. + + n := copy(cp.buf[len(cp.buf):cap(cp.buf)], data) + cp.buf = cp.buf[:len(cp.buf)+n] + b.length += n + + if len(data) == n { + break + } + data = data[n:] + + b.pages = append(b.pages, &page{buf: make([]byte, 0, b.nextPageSize)}) + b.nextPageSize *= 2 + } + + return dataLen, nil +} + +// WriteByte writes data byte to PageBuffer and returns any encountered error. +func (b *PageBuffer) WriteByte(data byte) error { + _, err := b.Write([]byte{data}) + return err +} + +// Len returns length of PageBuffer. +func (b *PageBuffer) Len() int { + return b.length +} + +// pageForOffset returns pageIdx and startIdx for the offset. +func (b *PageBuffer) pageForOffset(offset int) (int, int) { + AssertTrue(offset < b.length) + + var pageIdx, startIdx, sizeNow int + for i := 0; i < len(b.pages); i++ { + cp := b.pages[i] + + if sizeNow+len(cp.buf)-1 < offset { + sizeNow += len(cp.buf) + } else { + pageIdx = i + startIdx = offset - sizeNow + break + } + } + + return pageIdx, startIdx +} + +// Truncate truncates PageBuffer to length n. +func (b *PageBuffer) Truncate(n int) { + pageIdx, startIdx := b.pageForOffset(n) + // For simplicity of the code reject extra pages. These pages can be kept. + b.pages = b.pages[:pageIdx+1] + cp := b.pages[len(b.pages)-1] + cp.buf = cp.buf[:startIdx] + b.length = n +} + +// Bytes returns whole Buffer data as single []byte. +func (b *PageBuffer) Bytes() []byte { + buf := make([]byte, b.length) + written := 0 + for i := 0; i < len(b.pages); i++ { + written += copy(buf[written:], b.pages[i].buf) + } + + return buf +} + +// WriteTo writes whole buffer to w. It returns number of bytes written and any error encountered. +func (b *PageBuffer) WriteTo(w io.Writer) (int64, error) { + written := int64(0) + for i := 0; i < len(b.pages); i++ { + n, err := w.Write(b.pages[i].buf) + written += int64(n) + if err != nil { + return written, err + } + } + + return written, nil +} + +// NewReaderAt returns a reader which starts reading from offset in page buffer. +func (b *PageBuffer) NewReaderAt(offset int) *PageBufferReader { + pageIdx, startIdx := b.pageForOffset(offset) + + return &PageBufferReader{ + buf: b, + pageIdx: pageIdx, + startIdx: startIdx, + } +} + +// PageBufferReader is a reader for PageBuffer. +type PageBufferReader struct { + buf *PageBuffer // Underlying page buffer. + pageIdx int // Idx of page from where it will start reading. + startIdx int // Idx inside page - buf.pages[pageIdx] from where it will start reading. +} + +// Read reads upto len(p) bytes. It returns number of bytes read and any error encountered. +func (r *PageBufferReader) Read(p []byte) (int, error) { + // Check if there is enough to Read. + pc := len(r.buf.pages) + + read := 0 + for r.pageIdx < pc && read < len(p) { + cp := r.buf.pages[r.pageIdx] // Current Page. + endIdx := len(cp.buf) // Last Idx up to which we can read from this page. + + n := copy(p[read:], cp.buf[r.startIdx:endIdx]) + read += n + r.startIdx += n + + // Instead of len(cp.buf), we comparing with cap(cp.buf). This ensures that we move to next + // page only when we have read all data. Reading from last page is an edge case. We don't + // want to move to next page until last page is full to its capacity. + if r.startIdx >= cap(cp.buf) { + // We should move to next page. + r.pageIdx++ + r.startIdx = 0 + continue + } + + // When last page in not full to its capacity and we have read all data up to its + // length, just break out of the loop. + if r.pageIdx == pc-1 { + break + } + } + + if read == 0 { + return read, io.EOF + } + + return read, nil +} diff --git a/vendor/vendor.json b/vendor/vendor.json index aa79c2385ff..6b886fdcb1f 100644 --- a/vendor/vendor.json +++ b/vendor/vendor.json @@ -465,46 +465,46 @@ "revisionTime": "2016-09-07T16:21:46Z" }, { - "checksumSHA1": "O8YSP9f+dH/BlJMH5yK5hff0VG8=", + "checksumSHA1": "nztM5OYLZA5UhtlAkVemis1eRP4=", "path": "github.com/dgraph-io/badger", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { "checksumSHA1": "On87ObyZ4RLZWWmDtW2M5gY997I=", "path": "github.com/dgraph-io/badger/options", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { "checksumSHA1": "SHnr7a5ZSRsyrAT8Nw1AxDs1UXw=", "path": "github.com/dgraph-io/badger/pb", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { "checksumSHA1": "zWLyKFwJcRNWzpvE+QDrsqi9UlE=", "path": "github.com/dgraph-io/badger/skl", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { - "checksumSHA1": "X0uJEOHMIiJ/jUiGqhl62o+f0z0=", + "checksumSHA1": "cnBwJqnn60NE2pa1hvffhbg2E8Q=", "path": "github.com/dgraph-io/badger/table", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { "checksumSHA1": "JxUxrgSrNNTqbX3tqzh3dQIG+uU=", "path": "github.com/dgraph-io/badger/trie", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { - "checksumSHA1": "pfK0K0RIzPJ4NZ+B27kgq6as/ew=", + "checksumSHA1": "d7+Tg5zhOvzE1lUzK2a+NGLTRAc=", "path": "github.com/dgraph-io/badger/y", - "revision": "fbcd608981982d2d8c1612d310ac491d01dbde96", - "revisionTime": "2019-09-03T19:46:48Z" + "revision": "cbdef65095c7c34544963f31eb1f7867c49d6725", + "revisionTime": "2019-09-17T13:39:22Z" }, { "checksumSHA1": "gM1B8ixG+lBNstwPe0UDzOA2gcA=",