tracing: panic due to "getting first from empty ring buffer" #75045
Assignees
Labels
branch-master
Failures and bugs on the master branch.
C-test-failure
Broken test (automatically or manually discovered).
O-robot
Originated from a bot.
T-kv
KV Team
T-sql-queries
SQL Queries Team
Comments
cockroach-teamcity
added
branch-master
|
sql/tests.TestCreateRandomSchema failed with artifacts on master @ 365b4da8bd02c06ee59d2130a56dec74ffc9ce21: Help
See also: How To Investigate a Go Test Failure (internal)
|
|
Actually, was only interested in the last failure, the first looks unrelated ("panic: getting first from empty ring buffer"). |
|
+cc @andreimatei, might have something to do with tracing. |
|
Also seen in I can repro with this test under stress. Somehow the maintained size of the structured recording buffer seems to get desynchronized from the buffer's contents. Nothing in this area seems to have changed recently, so I'm not sure why we're seeing these failures all of a sudden. |
rytaft
changed the title
|
Surprisingly or not, this bisected to #73876. |
andreimatei
added a commit
to andreimatei/cockroach
that referenced
this issue
Jan 18, 2022
This patch fixes a bug leading to corruption of one of the Span's
recording buffers.
We had broken code like the following:
```
for _, e := range c.StructuredRecords {
s.recordInternalLocked(&e, &s.mu.recording.structured)
}
```
`recordInternalLocked(ptr)` takes ownership of ptr, so the `&e` is
broken because `e` keep being reassigned.
The bug was causing buffer elements to essentially be overwritten. In
turn, this was leading to inconsistencies in the accounting of the size
of the buffer's elements - we were maintaining a size sum that was
diverging from the overwritten reality.
Fixes cockroachdb#74994
Fixes cockroachdb#75045
Release note: None
|
Fixing in #75099 |
craig bot
pushed a commit
that referenced
this issue
Jan 20, 2022
75099: tracing: fix corruption of structured recording buffer r=andreimatei a=andreimatei
This patch fixes a bug leading to corruption of one of the Span's
recording buffers.
We had broken code like the following:
```
for _, e := range c.StructuredRecords {
s.recordInternalLocked(&e, &s.mu.recording.structured)
}
```
`recordInternalLocked(ptr)` takes ownership of ptr, so the `&e` is
broken because `e` keep being reassigned.
The bug was causing buffer elements to essentially be overwritten. In
turn, this was leading to inconsistencies in the accounting of the size
of the buffer's elements - we were maintaining a size sum that was
diverging from the overwritten reality.
Fixes #74994
Fixes #75045
Release note: None
Co-authored-by: Andrei Matei <[email protected]>
gtr
pushed a commit
to gtr/cockroach
that referenced
this issue
Jan 24, 2022
This patch fixes a bug leading to corruption of one of the Span's
recording buffers.
We had broken code like the following:
```
for _, e := range c.StructuredRecords {
s.recordInternalLocked(&e, &s.mu.recording.structured)
}
```
`recordInternalLocked(ptr)` takes ownership of ptr, so the `&e` is
broken because `e` keep being reassigned.
The bug was causing buffer elements to essentially be overwritten. In
turn, this was leading to inconsistencies in the accounting of the size
of the buffer's elements - we were maintaining a size sum that was
diverging from the overwritten reality.
Fixes cockroachdb#74994
Fixes cockroachdb#75045
Release note: None
23 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
sql/tests.TestCreateRandomSchema failed with artifacts on master @ 365b4da8bd02c06ee59d2130a56dec74ffc9ce21:
Fatal error:
Stack:
Log preceding fatal error
Help
See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:
This test on roachdash | Improve this report!
The text was updated successfully, but these errors were encountered: