kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ ce9d16042e04534c6d50727993661febd9c41789:

            b.Del(tk(9145994268991428698) /* @s28 */) // <nil>
            txn.Run(ctx, b) // @1732664893.382627674,0 <nil>
          }
          txn.ScanForShareSkipLocked(ctx, tk(9597347846416037427), tk(15835677158805487820), 0) // @1732664893.433471340,0 (/Table/100/"97882a30b5b42902":v27, <nil>)
          txn.GetForUpdateSkipLocked(ctx, tk(16170792340023286261)) // @1732664893.491658201,0 (<nil>, <nil>)
          {
            b := &kv.Batch{}
            b.GetForUpdate(tk(13660889541681518250)) // (<nil>, <nil>)
            b.GetForShare(tk(12595367538787500034)) // (<nil>, <nil>)
            b.GetForUpdate(tk(11414968790134163113)) // (<nil>, <nil>)
            txn.Run(ctx, b) // @1732664893.537647748,0 <nil>
          }
          txn.ReverseScanForUpdateGuaranteedDurability(ctx, tk(2366192350843884703), tk(14763842834420616581), 0) // TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=0626f107 key=/Table/100/"1885ed73b413caf8" iso=ReadCommitted pri=0.03156494 epo=0 ts=1732664893.588766222,0 min=1732664892.541333862,0 seq=2} lock=true stat=ABORTED rts=1732664893.588766222,0 wto=false gul=1732664892.551333862,0
          txn.DelRange(ctx, tk(10159571435349630440), tk(10404373276203250708), true /* @s29 */) // omitted
          txn.ReverseScanForShareSkipLockedGuaranteedDurability(ctx, tk(7106046230720532286), tk(10043145979000319449), 0) // omitted
          {
            b := &kv.Batch{}
            b.ScanForUpdate(tk(4144092209510484391), tk(16348789955210874526)) // omitted
            b.ScanForShareGuaranteedDurability(tk(1836852335987230440), tk(12356198369059853600)) // omitted
            b.ScanForShare(tk(1953096453474226557), tk(10187785147640933441)) // omitted
            txn.Run(ctx, b) // omitted
          }
          txn.Get(ctx, tk(923493776545345850)) // omitted
          return errors.New("rollback")
        }) // have retried transaction: unnamed (id: 0626f107-84c4-4891-bcc5-e3a677069373) 101 times, most recently because of the retryable error: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=0626f107 key=/Table/100/"1885ed73b413caf8" iso=ReadCommitted pri=0.03156494 epo=0 ts=1732664893.588766222,0 min=1732664892.541333862,0 seq=2} lock=true stat=ABORTED rts=1732664893.588766222,0 wto=false gul=1732664892.551333862,0. Terminating retry loop and returning error due to cluster setting kv.transaction.internal.max_auto_retries (100). Rollback error: <nil>.: have retried transaction: unnamed (id: 0626f107-84c4-4891-bcc5-e3a677069373) 101 times, most recently because of the retryable error: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=0626f107 key=/Table/100/"1885ed73b413caf8" iso=ReadCommitted pri=0.03156494 epo=0 ts=1732664893.588766222,0 min=1732664892.541333862,0 seq=2} lock=true stat=ABORTED rts=1732664893.588766222,0 wto=false gul=1732664892.551333862,0. Terminating retry loop and returning error due to cluster setting kv.transaction.internal.max_auto_retries (100). Rollback error: <nil>.
    kvnemesis.go:208: failures(verbose): /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis3117748181/failures
        repro steps: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis3117748181/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis3117748181/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/kvnemesis3117748181/kvs-scan.txt
    kvnemesis_test.go:373: [0] proposed: 4927
    kvnemesis_test.go:375: [0] reproposed unchanged: 73
    kvnemesis_test.go:377: [0] reproposed with new LAI: 1
    kvnemesis_test.go:373: [1] proposed: 1823
    kvnemesis_test.go:375: [1] reproposed unchanged: 0
    kvnemesis_test.go:377: [1] reproposed with new LAI: 0
    kvnemesis_test.go:373: [2] proposed: 760
    kvnemesis_test.go:375: [2] reproposed unchanged: 0
    kvnemesis_test.go:377: [2] reproposed with new LAI: 1
    kvnemesis_test.go:373: [3] proposed: 2763
    kvnemesis_test.go:375: [3] reproposed unchanged: 0
    kvnemesis_test.go:377: [3] reproposed with new LAI: 11
    kvnemesis_test.go:382: 
        	Error Trace:	pkg/kv/kvnemesis/kvnemesis_test.go:382
        	            				pkg/kv/kvnemesis/kvnemesis_test.go:306
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/4aeb9c412c16a003ed8ddc43788d2bba/logTestKVNemesisMultiNode2520231409
--- FAIL: TestKVNemesisMultiNode (147.98s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvnemesis: TestKVNemesisMultiNode failed #134950 kv/kvnemesis: TestKVNemesisMultiNode failed [C-bug C-test-failure O-robot P-3 T-kv branch-release-23.1]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-44946

[arulajmani]

This seems similar to #133431 at first glance. We had thought #135173 would help here, but evidently not.

As next steps, I'll dig in a bit more to make sure we're seeing the same issue. If we are, we should be able to remove the release blocker label and think more about how we can deflake this.