Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Use sync.Pool for MapEntries in bulk loader (#3763)
This PR has a significant impact on the allocated objects and space. For addMapEntry it went down from 7.35GB to 5.48GB. For addIndexMapEntries, it went down from 15.67GB to 10.11GB. The profiles were taken 30s into bulk loader. The other allocations line up nicely to verify that (lexer.Emit and rdfChunker.Parse). Current master: ``` Showing nodes accounting for 45.97GB, 82.78% of 55.53GB total Dropped 112 nodes (cum <= 0.28GB) Showing top 10 nodes out of 54 flat flat% sum% cum cum% 13.61GB 24.51% 24.51% 13.61GB 24.51% github.com/dgraph-io/dgraph/lex.(*Lexer).Emit 7.35GB 13.23% 37.74% 7.35GB 13.23% github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).addMapEntry 6.82GB 12.29% 50.02% 15.67GB 28.21% github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).addIndexMapEntries 3.62GB 6.51% 56.54% 24.84GB 44.74% github.com/dgraph-io/dgraph/chunker.rdfChunker.Parse 3.04GB 5.48% 62.02% 3.04GB 5.48% github.com/dgraph-io/dgraph/posting.NewPosting 2.95GB 5.32% 67.34% 2.95GB 5.32% bytes.makeSlice 2.80GB 5.04% 72.38% 2.80GB 5.04% github.com/dgraph-io/dgraph/gql.NQuad.createEdgePrototype 2.71GB 4.87% 77.25% 2.71GB 4.87% github.com/dgraph-io/dgraph/lex.NewLexer 1.62GB 2.92% 80.18% 1.62GB 2.92% strings.makeCutsetFunc 1.45GB 2.61% 82.78% 1.45GB 2.61% github.com/dgraph-io/dgraph/x.generateKey ``` This PR: ``` Showing nodes accounting for 39.94GB, 80.85% of 49.40GB total Dropped 148 nodes (cum <= 0.25GB) Showing top 10 nodes out of 54 flat flat% sum% cum cum% 13.47GB 27.26% 27.26% 13.47GB 27.26% github.com/dgraph-io/dgraph/lex.(*Lexer).Emit 4.40GB 8.90% 36.16% 10.11GB 20.47% github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).addIndexMapEntries 3.63GB 7.36% 43.52% 24.81GB 50.23% github.com/dgraph-io/dgraph/chunker.rdfChunker.Parse 3.47GB 7.03% 50.54% 3.47GB 7.03% github.com/dgraph-io/dgraph/dgraph/cmd/bulk.newMapper.func1 2.97GB 6.01% 56.55% 2.97GB 6.01% bytes.makeSlice 2.92GB 5.91% 62.46% 2.92GB 5.91% github.com/dgraph-io/dgraph/posting.NewPosting 2.76GB 5.59% 68.05% 2.76GB 5.59% github.com/dgraph-io/dgraph/lex.NewLexer 2.69GB 5.45% 73.51% 2.69GB 5.45% github.com/dgraph-io/dgraph/gql.NQuad.createEdgePrototype 2GB 4.05% 77.56% 5.48GB 11.09% github.com/dgraph-io/dgraph/dgraph/cmd/bulk.(*mapper).addMapEntry 1.63GB 3.29% 80.85% 1.63GB 3.29% strings.makeCutsetFunc ``` (cherry picked from commit 3e52a8f)
- Loading branch information