19.2 release blockers list

[andy-kimball]

Backport patrol

• release: backport patrol for 19.2 #41629 @tbg

Possible blockers for the 19.2 release:

Build State

• build: Update x/net/html dependency #40876 Go dependency updates @bdarnell
• ui: JavaScript updates to address vulnerabilities #41291 JS dependency updates @nathanstilwell

KV

• BETA BLOCKER storage: non-zero HardState.Commit on uninitialized replica #40213 (Non-zero HardState.Commit on uninitialized replica) @nvanbenschoten

• stability: Rebalances must be atomic #12768 (Rebalances must be atomic) @tbg

• storage: learner snap can fail when interacting with raft snap #40207 (Learner snap can fail when interacting with raft snap) @danhhz

• teamcity: failed test: TestParallel invalid retryable err #40328 (teamcity: failed test: TestParallel invalid retryable err) @nvanbenschoten

• roachtest: tpcc/mixed-headroom/n5cpu16 failed #39460 (roachtest: tpcc/mixed-headroom/n5cpu16 failed) @tbg

• Slack link (intents ignored when in a read's uncertainty interval, can cause stale reads) @nvanbenschoten

• roachtest: acceptance/bank/cluster-recovery failed #38785 (roachtest: acceptance/bank/cluster-recovery failed) @irfansharif

• perf: regression since 20190805 release #41206 performance regression (logging memory leak + stats collection inefficiency) @nvanbenschoten

• changefeedccl: bug in poller causes it to emit a resolved timestamp before emitting row updates at that timestamp. #41415 bug in poller causes it to emit a resolved timestamp before emitting row updates at that timestamp @aayushshah15. Update: PR for this is in now (changefeedccl: fix bug in poller that allows it to emit a resolved ts before emitting row updates at the ts #41665). Pending backport.

• changefeedccl: bug in table history wrt fetching old table descriptor versions after a restart #41694 bug in table history/poller when fetching old table descriptor versions after a restart @aayushshah15.

• Incorrect count of total replicas for AllocatorAdd #41328 Under-replicated ranges when performing sequence of node removes/adds -- @darinpp

• storage: system.replication_stats shows faux overreplication #41446 system.replication_stats shows faux overreplication @andreimatei

• storage: replication reports generation takes 20% of a node's CPU for 200k ranges #41609 replication reports generation takes 20% of a node's CPU for 200k ranges @andreimatei

• BETA BLOCKERS The "old replicaID state reuse" fallout cluster:

  • storage: must never reuse state from old replicaIDs #40367 (must never reuse snapshots from old replicaIDs) @ajwerner @tbg
  • roachtest: clearrange/checks=false failed #38772 (comment) (merge missing RHS during clearrange)
  • storage: avoid (one) fatal error from splitPostApply #39796 (Avoid fatal error from splitPostApply) @tbg
  • storage: splitPostApply can see tombstone for RHS #40470 (splitPostApply sees tombstone for RHS which should be impossible) @tbg
  • storage: crash with snapshot widens existing replica, but no replica exists for subsumed key #40257 roachtest: clearrange/checks=false failed #40934 (Crash with snapshot widens existing replica, but no replica exists for subsumed key) @ajwerner

• BETA BLOCKER roachtest: clearrange/checks=false failed #38772 (roachtest: clearrange/checks=false failed) @nvanbenschoten

• teamcity: failed test: bank/zerosum-splits #41096 (had non-voter replicas assertion fired) @danhhz

• BETA BLOCKER roachtest: jepsen/monotonic/majority-ring-start-kill-2 failed #41364 Jepsen/monotonic failures

• storage: check whether the acquiring replica can receive the lease #42068 leases can be transferred to replicas which are being removed @ajwerner

Probably not blockers, but here for visibility and discussion:

• teamcity: failed test: TestChangefeedNemeses [skipped] #38368 cloud storage cdc sinks are marked as experimental but there's a known correctness problem (that we know how to fix) -- @danhhz @aayushshah15
• Sentry events do not file github issues any more #40412 Sentry events do not file github issues any more -- @knz
• roachtest: clearrange/checks=true failed #38720 (comment) node crashed during roachtest due to consistency checker -- @tbg
• cli: Make unsafe-remove-dead-replicas work with multiple stores #41266 unsafe-remove-replicas needs to work in 19.2 with learners -- @darinpp
• roachtest: schemachange/mixed/tpcc failed #40935 schemachange/mixed/tpcc reports a FK violation @lucy-zhang

CRDB on Windows has severe issues

• storage: "Failed to link" error creating index on windows #38789 "Failed to link" error creating index on windows -- @darinpp
• storage: fatal error on windows related to rocksdb sst files while handling sideload event  #37427 fatal error running on Windows -- @darinpp
• sentry: windows: io error failed to remove dir #37819 sentry: windows: io error failed to remove dir -- @darinpp
• Appveyor hangs during shutdown of CRDB on Windows #40936 Appveyor hangs during shutdown of CRDB on Windows -- @darinpp
• unable to commit Raft entry batch: non-deterministic failure: unable to commit Raft entry batch: Corruption: Out-of-order key insertion into table #40918 (unable to commit Raft entry batch: Corruption: Out-of-order key insertion into table) @ajkr

Note, #32415 is a lower priority than the above

BulkIO

• teamcity: failed test: TestLogic #40504 (comment) (schema change transactions taking >30s will fail in 19.1+) @ajwerner
• roachtest: import/tpcc/warehouses=1000/nodes=32 failed #39072 descriptor changed (during IMPORT) @pbardea
• backup: "invalid header size" error during restore #40670 Failed to restore a registration cluster. @pbardea
• roachtest: import/tpcc/warehouses=1000/nodes=4 failed #39726 roachtest: import/tpcc/warehouses=1000/nodes=4 failed @lucy-zhang (not a release blocker)
• roachtest: clearrange/checks=true failed #38720 roachtest: clearrange/checks=true failed @dt
• OOM panic during fixtures import #41009 OOM when importing TPCC 10k @pbardea

SQL

• Combined SQL/KV issue kv: scan doesn't return row added in the same txn) kv: scan doesn't return row added in the same txn #40487 @andreimatei
• Combined SQL/KV issue jepsen-monotonic failure when using RootTxn on the gateway jepsen-monotonic failure when using RootTxn on the gateway #41424 @andreimatei
• Combined SQL/KV issue write skew in distributed DistSQL flows sql: race between gateway refresh and reads performed by remote flows leads to write skew #41173 @andreimatei
• Malformed planNode tree generated for TPCH variant sql: a crash with a lookup join #40562
• Insufficient privilege checking for ALTER CONFIGURE ZONE sql: users with select permissions can alter tables and partitions with zone configurations  #40693
• BETA BLOCKER Incorrect span creation for plans with column families sql: DELETE FROM is broken for system.jobs table #40890
• BETA BLOCKER panic in TPCCBench due to something about getRangesInfo roachtest: tpccbench/nodes=3/cpu=16 failed #39013
• Index/lookup join has correctness problem in the face of column families sql: SELECT difference with column families #41007
• issues with AND and OR expressions with vectorized colexec: problems with AND and OR operators #40913
• Timestamps are truncated to precision 0 after upgrading to 19.2 sql: timestamps are truncated to precision 0 after 19.2 upgrade #42283

Storage

• Fix concurrent full purge and WAL recycling rocksdb#64 Fix concurrent full purge and WAL recycling -- @ajkr
• Intra-LO compaction+ingestion bug Fix corruption with intra-L0 on ingested files facebook/rocksdb#5958

Other Discussion

• Should we add new stability period criteria: change allowed if it's going to be back-ported to previous release, and there's at least 1-2 weeks left for testing?

[andy-kimball]

We'll need to add Isaac as an Assignee once he's in the system.

[irfansharif]

Can we have a tracking issue for this:

Slack link (intents ignored when in a read's uncertainty interval, can cause stale reads) @nvanbenschoten

[nvanbenschoten]

Can we have a tracking issue for this:

We never opened an issue for it. It was fixed by #40600.

[andreimatei]

Heads up that I have a new horse in the race: #41173

[maddyblue]

I'm adding #41206 to the list.

[knz]

added #41222 from andrei pending further assessment

[thoszhang]

I added #38720 (comment) from @dt (a node crashed during the clearrange roachtest due to a consistency checker failure) to the KV section. (Note that this is a new failure.)

[bdarnell]

Added jepsen failure #41364, and marked it and #38720 as beta blockers.

[aayushshah15]

Added #41415 to the list (changefeed poller bug).

[aayushshah15]

Added #41694 to the list (bug in table history/poller when fetching old table descriptor versions after a job restart).

[solongordon]

Added #42068 (leases can be transferred to replicas which are being removed)

[thoszhang]

Added #42283.

[andreimatei]

Long live 19.2