BulkLoader becomes very slow after updating Badger to V2.0.0

[xiangzhao632]

What version of Go are you using (go version)?

go1.13.5

What version of Badger are you using?

v2.0.0

Does this issue reproduce with the latest master?

yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

linux/amd64

What did you do?

I run bulk loader

What did you expect to see?

bulk loader runs fast as usual

What did you see instead?

Before BadgerV2.0.0，reduce stage of bulk loader were about 60w rdf/s at our machine，now is about 4w/s。I used go tool pprof，and found that Cgo call cost 85% of cpu time。I guess its caused by the new compress algo。

[ashish-goswami]

@xiangzhao632 Badger V2.0.0, has compression enabled and it uses ZSTD compression by default. Badger also exposes one option called ZSTDCompressionLevel, which is 15 by default(This option is not exposed at bulk loader). Having higher compression level means more compressed data at the expense of more computation. This is the reason for slowness. You can try to reduce compression level and check the speed.

[xiangzhao632]

@ashish-goswami thank you ! Have you planned to expose compression option at bulk loader ?

[jarifibrahim]

Hey @xiangzhao632 I've created an issue on dgraph to track this hypermodeinc/dgraph#4439 .