Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli/democluster: TestTransientClusterSimulateLatencies failed (nil panic in maybeRejectUnsafeProposalLocked) #83308

Closed
cockroach-teamcity opened this issue Jun 24, 2022 · 3 comments · Fixed by #83520
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jun 24, 2022

cli/democluster.TestTransientClusterSimulateLatencies failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154:

=== RUN   TestTransientClusterSimulateLatencies
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/b296432f77c11d14af9987c5728df5a9/logTestTransientClusterSimulateLatencies3489558533
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTransientClusterSimulateLatencies
    panic.go:1047: -- test log scope end --
    panic.go:1047: runtime error: invalid memory address or nil pointer dereference
        goroutine 10493 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()
        	github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:110 +0x92
        panic({0x5734e80, 0x87004f0})
        	GOROOT/src/runtime/panic.go:1047 +0x266
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).maybeRejectUnsafeProposalLocked(0xc00475aab8, {0x64f8bd8, 0xc00526e390}, {0x0, 0x0}, 0xc006450b40)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:690 +0x41f
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc00475aab8, {0x64f8bd8, 0xc00526e390}, {0x0, 0x0})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:428 +0x4d5
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithoutProposing(0xc00475aab8, {0x64f8bd8, 0xc00526e390})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:921 +0x65
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2.1(0xc00475a400)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:711 +0x97
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc009abd2c0, 0xc01334d6b8)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:416 +0x25c
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).VisitReplicas(0xc001aa3500, 0xc01334d6b8, {0x0, 0x0, 0x0})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2633 +0xc5
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2()
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:709 +0x85
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc005737be0)
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc000bc26c0, {0x64f8b68, 0xc00013a000})
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x3ae
        github.com/cockroachdb/cockroach/pkg/server.(*Server).Stop(0xc001082400)
        	github.com/cockroachdb/cockroach/pkg/server/server.go:1608 +0x4a
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc00087f1d0)
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc000bc2480, {0x64f8b68, 0xc00013a000})
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x3ae
        github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Close(0xc000412ff0, {0x64f8b68, 0xc00013a000})
        	github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:812 +0x4c
        github.com/cockroachdb/cockroach/pkg/cli/democluster.TestTransientClusterSimulateLatencies(0xc001098340)
        	github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster_test.go:256 +0x846
        testing.tRunner(0xc001098340, 0x5f48808)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
--- FAIL: TestTransientClusterSimulateLatencies (11.46s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-experience @cockroachdb/server @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-16989

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jun 24, 2022
@blathers-crl blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Jun 24, 2022
@otan
Copy link
Contributor

otan commented Jun 24, 2022

panic.go:1047: runtime error: invalid memory address or nil pointer dereference
        goroutine 10493 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        [github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()](http://github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1())
        	[github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:110](http://github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:110) +0x92
        panic({0x5734e80, 0x87004f0})
        	GOROOT/src/runtime/panic.go:1047 +0x266
        [github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).maybeRejectUnsafeProposalLocked(0xc00475aab8](http://github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).maybeRejectUnsafeProposalLocked(0xc00475aab8), {0x64f8bd8, 0xc00526e390}, {0x0, 0x0}, 0xc006450b40)
        	[github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:690](http://github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:690) +0x41f

maybeRejectUnsafeProposalLocked nil ptr'ing seems bad, maybe KV should take a look!

@blathers-crl blathers-crl bot added the T-kv KV Team label Jun 24, 2022
@otan otan changed the title cli/democluster: TestTransientClusterSimulateLatencies failed cli/democluster: TestTransientClusterSimulateLatencies failed (nil panic in maybeRejectUnsafeProposalLocked) Jun 24, 2022
@rafiss rafiss removed the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Jun 25, 2022
@cockroach-teamcity
Copy link
Member Author

cli/democluster.TestTransientClusterSimulateLatencies failed with artifacts on master @ a611410e3babe97a14152381b89060e1bafa2e27:

=== RUN   TestTransientClusterSimulateLatencies
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/b296432f77c11d14af9987c5728df5a9/logTestTransientClusterSimulateLatencies2570579511
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestTransientClusterSimulateLatencies
    panic.go:1047: -- test log scope end --
    panic.go:1047: runtime error: invalid memory address or nil pointer dereference
        goroutine 82 [running]:
        runtime/debug.Stack()
        	GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()
        	github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:110 +0x92
        panic({0x573f4a0, 0x8711500})
        	GOROOT/src/runtime/panic.go:1047 +0x266
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).maybeRejectUnsafeProposalLocked(0xc004a572b8, {0x6508718, 0xc0041f98f0}, {0x0, 0x0}, 0xc0052c0000)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:690 +0x41f
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc004a572b8, {0x6508718, 0xc0041f98f0}, {0x0, 0x0})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:428 +0x4d5
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithoutProposing(0xc004a572b8, {0x6508718, 0xc0041f98f0})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:921 +0x65
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2.1(0xc004a56c00)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:711 +0x97
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc00e003a10, 0xc012cdf6b8)
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:416 +0x25c
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).VisitReplicas(0xc0007ec000, 0xc012cdf6b8, {0x0, 0x0, 0x0})
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2633 +0xc5
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2()
        	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:709 +0x85
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc003311660)
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00038ccf0, {0x65086a8, 0xc000080048})
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x3ae
        github.com/cockroachdb/cockroach/pkg/server.(*Server).Stop(0xc000b25b00)
        	github.com/cockroachdb/cockroach/pkg/server/server.go:1608 +0x4a
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc0009722b0)
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1f
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00038cb40, {0x65086a8, 0xc000080048})
        	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x3ae
        github.com/cockroachdb/cockroach/pkg/cli/democluster.(*transientCluster).Close(0xc00026d950, {0x65086a8, 0xc000080048})
        	github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster.go:812 +0x4c
        github.com/cockroachdb/cockroach/pkg/cli/democluster.TestTransientClusterSimulateLatencies(0xc000603a00)
        	github.com/cockroachdb/cockroach/pkg/cli/democluster/demo_cluster_test.go:256 +0x846
        testing.tRunner(0xc000603a00, 0x5f54888)
        	GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
        	GOROOT/src/testing/testing.go:1306 +0x35a
--- FAIL: TestTransientClusterSimulateLatencies (3.72s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@nvanbenschoten
Copy link
Member

Duplicate of #83498.

craig bot pushed a commit that referenced this issue Jun 28, 2022
82352: server, sql: surface session txnCount, recent txn fingerprints, active time r=xinhaoz a=xinhaoz

Finishing up Gerardo's PR, original review here: #80717

--------------------------------------------------
Partially addresses #74257.

Previously, the status server did not provide session details such as
total number of transactions executed, transaction fingerprint
IDs, and total active time. This change adds the aforementioned session
details to the `serverpb.Session` struct.

To track recently executed transaction fingerprint IDs, a FIFO cache
`TxnFingerprintIDCache` is introduced with its corresponding cluster
setting `TxnFingerprintIDBufferCapacity` to control the capacity. The
default capacity is set at 100 fingerprints.

The total number of transactions executed is filled using the existing
`txnCounter` from the `extraTxnState` in `connExecutor`. The total active
time is calculated by introducing a `timeutil.StopWatch` to the connection
executor, which is started and stopped when a transaction is started and
finished respectively.

Release note (api change): the `serverpb.Session` struct now has three
new fields: number of transactions executed, transaction fingerprint
IDs, and total active time.

82623: backupinfo: introduce a backupinfo package r=stevendanna a=adityamaru

The backupinfo package contains logic related to interacting
with information and metadata describing the backup. After this
change we have `backupdest` depending on `backupinfo`.

Release note: None

82718: kvserver: emit MVCC range tombstones over rangefeeds r=aliher1911 a=erikgrinaker

This patch adds MVCC range tombstone support in rangefeeds. Whenever an
MVCC range tombstone is written, a new `MVCCDeleteRangeOp` logical op
is recorded and emitted across the rangefeed as a `RangeFeedDeleteRange`
event. MVCC range tombstones will only be written when the
`MVCCRangeTombstones` version gate has been enabled.

Changefeeds will emit an error for these events. We do not expect to see
these in online spans with changefeeds, since they are initially only
planned for use with schema GC and import rollbacks.

The rangefeed client library has been extended with support for these
events, but no existing callers handle them for the same reason as
changefeeds. Initial scans do not emit regular tombstones, and thus not
range tombstones either, but catchup scans will emit them if
encountered.

This patch has rudimentary testing of MVCC range tombstones in
rangefeeds. A later patch will add a data-driven test harness for
rangefeeds with more exhaustive tests.

Resolves #82449.
Touches #70433.

Release note: None

82936: sql/schemachanger: implement DROP OWNED BY r=jasonmchan a=jasonmchan

Previously, we did not support the DROP OWNED BY statement (#55381).
This commit adds partial support for DROP OWNED BY in the declarative
schema changer. Followup work is needed to support the CASCADE modifier.

Release note (sql change): Support `DROP OWNED BY`.

83229: ui: remove option 10/30 min from SQL Activity page r=maryliag a=maryliag

Note to reviewers: only 2nd commit is relevant to this PR

Previously we had the options for 10 and 30min on
SQL Activity pages, which created some confusion, since
we would always show the last 1h info.
This commit remove those 2 options.
If the user select any of those options on the Metrics
page, it will get updated to 1h on the SQL Activity
pages.

<img width="444" alt="Screen Shot 2022-06-22 at 5 43 53 PM" src="https://user-images.githubusercontent.com/1017486/175144243-2f084e0b-5e09-4874-9640-e7eea6179343.png">

https://www.loom.com/share/226e54322df6456aa2039b5c54f72eb1


Fixes #82914

Release note (ui change): Removal of the 10 and 30min options
on the SQL Activity page.

83420: ui: improve tooltip UX with text updates r=ericharmeling a=ericharmeling

Fixes #81374.
Fixes #83256.
Fixes #81248.
Fixes #79018.

Note the following:

- The updates resolving #79018 effectively revert the tooltip text for Rows Read to the original wording (which [was updated for accuracy](e379e9d#diff-492398441e971e355a687a4ce333a9766e2195287d0227682444d5dc0eb7ee1a)). I assume this is okay. `@kevin-v-ngo`
- The updates resolving #81248 do not in fact refer to the time intervals as date ranges, as this language is misleading (a 1h interval is an interval and not a date range). Instead, this update just removes the anchor and the link to the non-existent Interval Range section of https://www.cockroachlabs.com/docs/stable/ui-statements-page.html. We may want to consider updating the docs to call the "time picker" data type a time interval and not a date range. This appears to have been the case in previous releases (https://www.cockroachlabs.com/docs/v21.1/ui-statements-page#time-interval). `@stbof` 

Release note (ui change): Updated tooltips on the Statements and
Transactions pages in the DB Console for improved UX.

83428: sql: rename anonymizedStmt in sqlstats pkg to stmtNoConstants r=ericharmeling a=ericharmeling

Note that this commit does not change any files outside the sqlstats package.

Fixes #80725.

Release note: None

83468: ui: update all dates to use same format r=maryliag a=maryliag

Update all dates to use the same format.

Fixes #81159

Release note: None

83520: kv: don't try to reject lease transfer when flushing proposal buffer r=nvanbenschoten a=nvanbenschoten

Fixes #83498.
Fixes #83402.
Fixes #83308.

This was fallout from #82758.

This commit adds logic to `propBuf.maybeRejectUnsafeProposalLocked` to avoid
trying to reject proposals based on the state of the raft group when the group
is not provided (e.g. when flushing the buffer). We already had this logic for
`RequestLease` (indirectly), but did not for `TransferLease`.

Co-authored-by: Gerardo Torres <[email protected]>
Co-authored-by: Aditya Maru <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Jason Chan <[email protected]>
Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: Eric Harmeling <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in #83520 Jun 28, 2022
@craig craig bot closed this as completed in 5d4c0d7 Jun 28, 2022
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Jan 13, 2023
Fixes cockroachdb#83498.
Fixes cockroachdb#83402.
Fixes cockroachdb#83308.

This was fallout from cockroachdb#82758.

This commit adds logic to `propBuf.maybeRejectUnsafeProposalLocked` to avoid
trying to reject proposals based on the state of the raft group when the group
is not provided (e.g. when flushing the buffer). We already had this logic for
`RequestLease` (indirectly), but did not for `TransferLease`.
nathanstilwell pushed a commit that referenced this issue Feb 2, 2023
Fixes #83498.
Fixes #83402.
Fixes #83308.

This was fallout from #82758.

This commit adds logic to `propBuf.maybeRejectUnsafeProposalLocked` to avoid
trying to reject proposals based on the state of the raft group when the group
is not provided (e.g. when flushing the buffer). We already had this logic for
`RequestLease` (indirectly), but did not for `TransferLease`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants