Potential Read Performance Regression

[connorgorman]

What version of Go are you using (go version)?

$ go version
1.13.4

What version of Badger are you using?

master (bce069c)

Does this issue reproduce with the latest master?

yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

8GB, Ubuntu, 100GB HDD

What did you do?

Running 1.6.0 vs master with a mixed workload doing mostly writes and random reads
Value sizes range from 400 to 1000 bytes

What did you expect to see?

I'd expect the performance of master to be better than the performance of 1.6.0 across the board. Instead, it seems that random reads have more variability in speed.

What did you see instead?

I've seen slow downs of up to 200x. Unfortunately, I haven't been able to provide an isolated code sample that I could share, but was curious if anything has shown up in any benchmarks.

[jarifibrahim]

@connorgorman were you running badger with default options? You might want to disable compression, encryption, and cache and then run your benchmarks.

[connorgorman]

@jarifibrahim I was running with the default options, but with NumLevelZeroTables = 2 and NumLevelZeroTablesStall = 5. I believe those other features are off by default (except the cache), but I will set the flags and comment again. Thanks

[connorgorman]

@jarifibrahim So I ran it again with

badger.DefaultOptions(path).
		WithTruncate(true).
		WithNumLevelZeroTables(2).
		WithNumLevelZeroTablesStall(5).
		WithMaxCacheSize(0).
		WithChecksumVerificationMode(options.NoVerification).
		WithCompression(options.None).
		WithVerifyValueChecksum(false)

and I'm getting similar results in terms of performance regressions. In fact, it looks like reads and writes are generally slower (though some cases appear faster).

Are there any internal metrics in Badger I can use? Do all those options make sense?

[jarifibrahim]

@connorgorman The options make sense. Let me try to run some benchmarks and see what results I get. Ideally, v2 should be faster than v1.6.

It would be useful if you can share your benchmarks.

[jarifibrahim]

Hey @connorgorman, I've raised #1256 to fix this.
Major performance improvement comes from #1256 and #1257 also improves the performance.

Please try out #1257 and let me know how it goes.

[connorgorman]

Running with the above benchmarks and taking the performance improvements, it looks like v1 and v2 are fairly comparable. V2 may be slightly slower, but hard to tell with any certainty. Feel free to close this issue. I'll reopen or open a new one if I have any more insights! Thanks for fixing this!

[jarifibrahim]

@connorgorman The code in #1257 is definitely faster than v1.6.0
with #1257 read bench tool (badger/cmd/read_bench.go) read 3.3 GB in 1 minute

grep elapsed read-new.txt | tail
Time elapsed: 50s, bytes read: 2.9 GB, speed: 58 MB/sec, entries read: 18703030, speed: 374060/sec
Time elapsed: 51s, bytes read: 2.9 GB, speed: 57 MB/sec, entries read: 19013253, speed: 372808/sec
Time elapsed: 52s, bytes read: 3.0 GB, speed: 57 MB/sec, entries read: 19316184, speed: 371465/sec
Time elapsed: 53s, bytes read: 3.0 GB, speed: 57 MB/sec, entries read: 19615634, speed: 370106/sec
Time elapsed: 54s, bytes read: 3.1 GB, speed: 57 MB/sec, entries read: 19863860, speed: 367849/sec
Time elapsed: 55s, bytes read: 3.1 GB, speed: 57 MB/sec, entries read: 20224828, speed: 367724/sec
Time elapsed: 56s, bytes read: 3.2 GB, speed: 57 MB/sec, entries read: 20595347, speed: 367774/sec
Time elapsed: 57s, bytes read: 3.2 GB, speed: 57 MB/sec, entries read: 20967933, speed: 367858/sec
Time elapsed: 58s, bytes read: 3.3 GB, speed: 57 MB/sec, entries read: 21335377, speed: 367851/sec
Time elapsed: 59s, bytes read: 3.3 GB, speed: 57 MB/sec, entries read: 21672915, speed: 367337/sec

For v1.6.0, it read 2.8 GB

grep elapsed read-1.6.txt | tail
Time elapsed: 50s, bytes read: 2.4 GB, speed: 48 MB/sec, entries read: 14321488, speed: 286429/sec
Time elapsed: 51s, bytes read: 2.4 GB, speed: 48 MB/sec, entries read: 14625924, speed: 286782/sec
Time elapsed: 52s, bytes read: 2.5 GB, speed: 48 MB/sec, entries read: 14912857, speed: 286785/sec
Time elapsed: 53s, bytes read: 2.5 GB, speed: 48 MB/sec, entries read: 15159867, speed: 286035/sec
Time elapsed: 54s, bytes read: 2.6 GB, speed: 48 MB/sec, entries read: 15476701, speed: 286605/sec
Time elapsed: 55s, bytes read: 2.6 GB, speed: 48 MB/sec, entries read: 15778034, speed: 286873/sec
Time elapsed: 56s, bytes read: 2.7 GB, speed: 48 MB/sec, entries read: 16077903, speed: 287105/sec
Time elapsed: 57s, bytes read: 2.7 GB, speed: 48 MB/sec, entries read: 16378622, speed: 287344/sec
Time elapsed: 58s, bytes read: 2.8 GB, speed: 48 MB/sec, entries read: 16681041, speed: 287604/sec
Time elapsed: 59s, bytes read: 2.8 GB, speed: 48 MB/sec, entries read: 16976639, speed: 287739/sec

[jarifibrahim]

Hey @connorgorman I've merged PRs #1257 and #1260 which have fixed the performance issue in latest badger. We'll soon release v2.0.3 with these fixes.

Thanks for filing the issue :)