Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sync.Pool for posting lists in getInternal and getNew. #3773

Closed
wants to merge 2 commits into from

Conversation

martinmr
Copy link
Contributor

@martinmr martinmr commented Aug 7, 2019

Before:

ROUTINE ======================== github.com/dgraph-io/dgraph/posting.(*LocalCache).getInternal in /home/martinmr/go/src/github.com/dgraph-io/dgraph/posting/lists.go
    6.49GB    15.49GB (flat, cum) 11.68% of Total
         .          .    223:
         .          .    224:func (lc *LocalCache) getInternal(key []byte, readFromDisk bool) (*List, error) {
         .          .    225:   if lc == nil {
         .          .    226:           return getNew(key, pstore)
         .          .    227:   }
    1.13GB     1.13GB    228:   skey := string(key)
         .          .    229:   if pl := lc.getNoStore(skey); pl != nil {
         .          .    230:           return pl, nil
         .          .    231:   }
         .          .    232:
         .          .    233:   var pl *List
         .          .    234:   if readFromDisk {
         .          .    235:           var err error
         .     6.52GB    236:           pl, err = getNew(key, pstore)
         .          .    237:           if err != nil {
         .          .    238:                   return nil, err
         .          .    239:           }
         .          .    240:   } else {
         .          .    241:           pl = &List{
         .          .    242:                   key:         key,
    1.02GB     1.02GB    243:                   mutationMap: make(map[uint64]*pb.PostingList),
    4.34GB     4.34GB    244:                   plist:       new(pb.PostingList),
         .          .    245:           }
         .          .    246:   }
         .          .    247:
         .          .    248:   // If we just brought this posting list into memory and we already have a delta for it, let's
         .          .    249:   // apply it before returning the list.
         .          .    250:   lc.RLock()
         .          .    251:   if delta, ok := lc.deltas[skey]; ok && len(delta) > 0 {
         .          .    252:           pl.setMutation(lc.startTs, delta)
         .          .    253:   }
         .          .    254:   lc.RUnlock()
         .     2.48GB    255:   return lc.SetIfAbsent(skey, pl), nil
         .          .    256:}
         .          .    257:
         .          .    258:// Get retrieves the cached version of the list associated with the given key.
         .          .    259:func (lc *LocalCache) Get(key []byte) (*List, error) {
         .          .    260:   return lc.getInternal(key, true)

After:

ROUTINE ======================== github.com/dgraph-io/dgraph/posting.(*LocalCache).getInternal in /home/martinmr/go/src/github.com/dgraph-io/dgraph/posting/lists.go
    3.23GB     9.65GB (flat, cum) 11.14% of Total
         .          .    223:
         .          .    224:func (lc *LocalCache) getInternal(key []byte, readFromDisk bool) (*List, error) {
         .          .    225:   if lc == nil {
         .          .    226:           return getNew(key, pstore)
         .          .    227:   }
  851.02MB   851.02MB    228:   skey := string(key)
         .          .    229:   if pl := lc.getNoStore(skey); pl != nil {
         .          .    230:           return pl, nil
         .          .    231:   }
         .          .    232:
         .          .    233:   var pl *List
         .          .    234:   if readFromDisk {
         .          .    235:           var err error
         .     4.50GB    236:           pl, err = getNew(key, pstore)
         .          .    237:           if err != nil {
         .          .    238:                   return nil, err
         .          .    239:           }
         .          .    240:   } else {
         .          .    241:           pl = &List{
         .          .    242:                   key:         key,
  859.04MB   859.04MB    243:                   mutationMap: make(map[uint64]*pb.PostingList),
    1.56GB     1.60GB    244:                   plist: postingListPool.Get().(*pb.PostingList),
         .          .    245:           }
         .          .    246:   }
         .          .    247:
         .          .    248:   // If we just brought this posting list into memory and we already have a delta for it, let's
         .          .    249:   // apply it before returning the list.
         .          .    250:   lc.RLock()
         .          .    251:   if delta, ok := lc.deltas[skey]; ok && len(delta) > 0 {
         .          .    252:           pl.setMutation(lc.startTs, delta)
         .          .    253:   }
         .          .    254:   lc.RUnlock()
         .     1.88GB    255:   return lc.SetIfAbsent(skey, pl), nil
         .          .    256:}
         .          .    257:
         .          .    258:// Get retrieves the cached version of the list associated with the given key.
         .          .    259:func (lc *LocalCache) Get(key []byte) (*List, error) {
         .          .    260:   return lc.getInternal(key, true)

This change is Reviewable

Before:
```
ROUTINE ======================== github.com/dgraph-io/dgraph/posting.(*LocalCache).getInternal in /home/martinmr/go/src/github.com/dgraph-io/dgraph/posting/lists.go
    6.49GB    15.49GB (flat, cum) 11.68% of Total
         .          .    223:
         .          .    224:func (lc *LocalCache) getInternal(key []byte, readFromDisk bool) (*List, error) {
         .          .    225:   if lc == nil {
         .          .    226:           return getNew(key, pstore)
         .          .    227:   }
    1.13GB     1.13GB    228:   skey := string(key)
         .          .    229:   if pl := lc.getNoStore(skey); pl != nil {
         .          .    230:           return pl, nil
         .          .    231:   }
         .          .    232:
         .          .    233:   var pl *List
         .          .    234:   if readFromDisk {
         .          .    235:           var err error
         .     6.52GB    236:           pl, err = getNew(key, pstore)
         .          .    237:           if err != nil {
         .          .    238:                   return nil, err
         .          .    239:           }
         .          .    240:   } else {
         .          .    241:           pl = &List{
         .          .    242:                   key:         key,
    1.02GB     1.02GB    243:                   mutationMap: make(map[uint64]*pb.PostingList),
    4.34GB     4.34GB    244:                   plist:       new(pb.PostingList),
         .          .    245:           }
         .          .    246:   }
         .          .    247:
         .          .    248:   // If we just brought this posting list into memory and we already have a delta for it, let's
         .          .    249:   // apply it before returning the list.
         .          .    250:   lc.RLock()
         .          .    251:   if delta, ok := lc.deltas[skey]; ok && len(delta) > 0 {
         .          .    252:           pl.setMutation(lc.startTs, delta)
         .          .    253:   }
         .          .    254:   lc.RUnlock()
         .     2.48GB    255:   return lc.SetIfAbsent(skey, pl), nil
         .          .    256:}
         .          .    257:
         .          .    258:// Get retrieves the cached version of the list associated with the given key.
         .          .    259:func (lc *LocalCache) Get(key []byte) (*List, error) {
         .          .    260:   return lc.getInternal(key, true)
```

After:
```
ROUTINE ======================== github.com/dgraph-io/dgraph/posting.(*LocalCache).getInternal in /home/martinmr/go/src/github.com/dgraph-io/dgraph/posting/lists.go
    3.23GB     9.65GB (flat, cum) 11.14% of Total
         .          .    223:
         .          .    224:func (lc *LocalCache) getInternal(key []byte, readFromDisk bool) (*List, error) {
         .          .    225:   if lc == nil {
         .          .    226:           return getNew(key, pstore)
         .          .    227:   }
  851.02MB   851.02MB    228:   skey := string(key)
         .          .    229:   if pl := lc.getNoStore(skey); pl != nil {
         .          .    230:           return pl, nil
         .          .    231:   }
         .          .    232:
         .          .    233:   var pl *List
         .          .    234:   if readFromDisk {
         .          .    235:           var err error
         .     4.50GB    236:           pl, err = getNew(key, pstore)
         .          .    237:           if err != nil {
         .          .    238:                   return nil, err
         .          .    239:           }
         .          .    240:   } else {
         .          .    241:           pl = &List{
         .          .    242:                   key:         key,
  859.04MB   859.04MB    243:                   mutationMap: make(map[uint64]*pb.PostingList),
    1.56GB     1.60GB    244:                   plist: postingListPool.Get().(*pb.PostingList),
         .          .    245:           }
         .          .    246:   }
         .          .    247:
         .          .    248:   // If we just brought this posting list into memory and we already have a delta for it, let's
         .          .    249:   // apply it before returning the list.
         .          .    250:   lc.RLock()
         .          .    251:   if delta, ok := lc.deltas[skey]; ok && len(delta) > 0 {
         .          .    252:           pl.setMutation(lc.startTs, delta)
         .          .    253:   }
         .          .    254:   lc.RUnlock()
         .     1.88GB    255:   return lc.SetIfAbsent(skey, pl), nil
         .          .    256:}
         .          .    257:
         .          .    258:// Get retrieves the cached version of the list associated with the given key.
         .          .    259:func (lc *LocalCache) Get(key []byte) (*List, error) {
         .          .    260:   return lc.getInternal(key, true)
```
@martinmr martinmr requested review from manishrjain and a team August 7, 2019 22:41
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ A review job has been created and sent to the PullRequest network.


@martinmr you can click here to see the review status or cancel the code review job.

@martinmr
Copy link
Contributor Author

martinmr commented Aug 7, 2019

This PR depends on #3767

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants