runtime: bound small object sweeping to 100 spans when allocating

Currently, the small object sweeper will sweep until it finds a free
slot or there are no more spans of that size class to sweep. In dense
heaps, this can cause sweeping for a given size class to take
unbounded time, and gets worse with larger heaps.

This CL limits the small object sweeper to try at most 100 spans
before giving up and allocating a fresh span. Since it's already shown
that 100 spans are completely full at that point, the space overhead
of this fresh span is at most 1%.

This CL is based on an experimental CL by Austin Clements (CL 187817)
and is updated to be part of the mcentral implementation, gated by
go115NewMCentralImpl.

Updates #18155.

Change-Id: I37a72c2dcc61dd6f802d1d0eac3683e6642b6ef8
Reviewed-on: https://go-review.googlesource.com/c/go/+/229998
Run-TryBot: Michael Knyszek <[email protected]>
Reviewed-by: Austin Clements <[email protected]>

src/runtime/mcentral.go

@@ -106,14 +106,31 @@ func (c *mcentral) cacheSpan() *mspan {
 	if trace.enabled {
 		traceGCSweepStart()
 	}
+
+	// If we sweep spanBudget spans without finding any free
+	// space, just allocate a fresh span. This limits the amount
+	// of time we can spend trying to find free space and
+	// amortizes the cost of small object sweeping over the
+	// benefit of having a full free span to allocate from. By
+	// setting this to 100, we limit the space overhead to 1%.
+	//
+	// TODO(austin,mknyszek): This still has bad worst-case
+	// throughput. For example, this could find just one free slot
+	// on the 100th swept span. That limits allocation latency, but
+	// still has very poor throughput. We could instead keep a
+	// running free-to-used budget and switch to fresh span
+	// allocation if the budget runs low.
+	spanBudget := 100
+
 	var s *mspan
 
 	// Try partial swept spans first.
 	if s = c.partialSwept(sg).pop(); s != nil {
 		goto havespan
 	}
+
 	// Now try partial unswept spans.
-	for {
+	for ; spanBudget >= 0; spanBudget-- {
 		s = c.partialUnswept(sg).pop()
 		if s == nil {
 			break
@@ -132,7 +149,7 @@ func (c *mcentral) cacheSpan() *mspan {
 	}
 	// Now try full unswept spans, sweeping them and putting them into the
 	// right list if we fail to get a span.
-	for {
+	for ; spanBudget >= 0; spanBudget-- {
 		s = c.fullUnswept(sg).pop()
 		if s == nil {
 			break