util/metric: v23.2.0-alpha: child already exists

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4456806246/?referrer=webhooks_plugin

Panic Message:

agg_metric.go:107: child [× × ×] already exists
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric.(*childSet).add
  | 	github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric/agg_metric.go:107
  | github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric.(*AggGauge).AddChild
  | 	github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric/gauge.go:87
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Metrics).acquire
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/metrics.go:189
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).newPeer
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/peer.go:178
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2157
  | github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCUnvalidatedDial
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:2096
  | github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1.2
  | 	github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/client.go:103
  | github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1
  | 	github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/client.go:115
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (4) child [× × ×] already exists
Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: child [× × ×] already exists
agg_metric.go:107: *withstack.withStack (top exception)
*assert.withAssertionFailure
panic.go:884: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 483 to 485 in 1382b26

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/gossip/pkg/gossip/client.go#L114-L116
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/gossip/pkg/gossip/client.go#L102-L104
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/context.go#L2095-L2097
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/context.go#L2156-L2158
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/peer.go#L177-L179
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/metrics.go#L188-L190

cockroach/pkg/util/metric/aggmetric/gauge.go

Lines 86 to 88 in 1382b26

	}
	g.add(child)
	return child

cockroach/pkg/util/metric/aggmetric/agg_metric.go

Lines 106 to 108 in 1382b26

	if cs.mu.tree.Has(metric) {
	panic(errors.AssertionFailedf("child %v already exists", metric.labelValues()))
	}

GOROOT/src/runtime/panic.go#L883-L885
GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 483 to 485 in 1382b26

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/gossip/pkg/gossip/client.go#L114-L116
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/gossip/pkg/gossip/client.go#L102-L104
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/context.go#L2095-L2097
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/context.go#L2156-L2158
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/peer.go#L177-L179
https://github.com/cockroachdb/cockroach/blob/1382b26a97bf6f70a07d363dc319283c173359eb/pkg/rpc/pkg/rpc/metrics.go#L188-L190

cockroach/pkg/util/metric/aggmetric/gauge.go

Lines 86 to 88 in 1382b26

	}
	g.add(child)
	return child

cockroach/pkg/util/metric/aggmetric/agg_metric.go

Lines 106 to 108 in 1382b26

	if cs.mu.tree.Has(metric) {
	panic(errors.AssertionFailedf("child %v already exists", metric.labelValues()))
	}

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 484
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).startLocked.func1 at line 115
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).startLocked.func1.2 at line 103
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).GRPCUnvalidatedDial at line 2096
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).grpcDialNodeInternal at line 2157
pkg/rpc/pkg/rpc/peer.go in pkg/rpc.(*Context).newPeer at line 178
pkg/rpc/pkg/rpc/metrics.go in pkg/rpc.(*Metrics).acquire at line 189
pkg/util/metric/aggmetric/gauge.go in pkg/util/metric/aggmetric.(*AggGauge).AddChild at line 87
pkg/util/metric/aggmetric/agg_metric.go in pkg/util/metric/aggmetric.(*childSet).add at line 107
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 484
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).startLocked.func1 at line 115
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).startLocked.func1.2 at line 103
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).GRPCUnvalidatedDial at line 2096
pkg/rpc/pkg/rpc/context.go in pkg/rpc.(*Context).grpcDialNodeInternal at line 2157
pkg/rpc/pkg/rpc/peer.go in pkg/rpc.(*Context).newPeer at line 178
pkg/rpc/pkg/rpc/metrics.go in pkg/rpc.(*Metrics).acquire at line 189
pkg/util/metric/aggmetric/gauge.go in pkg/util/metric/aggmetric.(*AggGauge).AddChild at line 87
pkg/util/metric/aggmetric/agg_metric.go in pkg/util/metric/aggmetric.(*childSet).add at line 107

Tags

Tag	Value
Command	server
Environment	v23.2.0-alpha.00000000
Go Version	go1.19.10
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.0-alpha.7-5180-g1382b26a97
Cockroach SHA	1382b26
# of CPUs	16
# of Goroutines	1001

Jira issue: CRDB-31257

[knz]

@aliher1911 @pavelkalinnikov this seems related to #108841. Can you have a look?

[blathers-crl]

cc @cockroachdb/replication

[pav-kv]

The SHA 1382b26 committed on Aug 8, which precedes another instance of this failure in #108499 on Aug 10. It was fixed by #108841.