Skip to content

Commit

Permalink
mon: don't panic in prod if no bytes to release
Browse files Browse the repository at this point in the history
Previously, we'd unconditionally panic the server if a memory monitor
was asked to release more bytes than it had allocated. This can happen
when there's a bug in memory accounting, but doesn't indicate an
actual memory leak.

This commit changes the panic to be a ReportOrPanic, which will panic in
test environments and send a report to sentry in production
environments. The memory monitor then resets itself to 0 allocated
bytes.

Release note (bug fix): prevent production server crashes on certain
assertion errors
  • Loading branch information
jordanlewis committed Mar 27, 2019
1 parent 15913f5 commit 087996b
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions pkg/util/mon/bytes_usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -625,8 +625,10 @@ func (mm *BytesMonitor) releaseBytes(ctx context.Context, sz int64) {
mm.mu.Lock()
defer mm.mu.Unlock()
if mm.mu.curAllocated < sz {
panic(fmt.Sprintf("%s: no bytes to release, current %d, free %d",
mm.name, mm.mu.curAllocated, sz))
sz = mm.mu.curAllocated
log.ReportOrPanic(ctx, &mm.settings.SV,
"%s: no bytes to release, current %d, free %d",
mm.name, mm.mu.curAllocated, sz)
}
mm.mu.curAllocated -= sz
if mm.curBytesCount != nil {
Expand Down

0 comments on commit 087996b

Please sign in to comment.