release: v19.2.10

[celiala]

Release Candidate Overview:

Candidate SHA: 5a1eb98cf9e0139f58b138f2743a3e0f55408b2f (5a1eb98, https://github.com/cockroachdb/cockroach/releases/tag/provisional_202008191705_v19.2.10)
Deployment status: Qualifying
Qualification Suite: #166 - started 8/19, 4:30p
Nightly Suite: #2313 - started 8/19, 4:30p

• #1967 (19 Aug 20 21:26) - Roachtest Nightly - GCE
  • 👍 Completed: Tests failed: 9 (4 new), passed: 210, ignored: 74

Admin UI for Qualification Clusters:

Created	v19.2.9	v19.2.10	+2 day	8/19	+1 day	+4 day
8/15, 2:30p	v19.2.9	v19.2.10	8/17, 12:30p	new SHA	8/20, 3pm	8/24, 10:30a
Overview (+QPS, Latency)	Overview	Overview	✅	-	✅	✅
SQL	SQL	SQL	✅	-	✅	✅
Replication	Replication	Replication	✅	-	✅	✅

Release process checklist

Prep date: WIP fill in date to roll out: usually Monday, a week before the release WIP

• Pick a SHA
  • fill in Candidate SHA above
  • email thread on releases@
• Tag the provisional SHA
• Publish provisional binaries
• Ack security@ and release-engineering-team@ on the generated AWS S3 bucket write Alert to confirm these writes were part of a planned release (Just reply on the email received alert email acking that this was part of the release process)

Release Qualification

• Check binaries - Update v19.2.10 bincheck#217
• Deploy to test clusters
• Verify node crash reports
• Start release qualification suite
• Start nightly suite
• fill in all WIP elements above (Nightly Suite, Admin UI, etc.)

One day after prep date:

• Get signoff on roachtest failures

• Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: Tentatively Monday, 24 Aug, 2020

• Check cluster status

• Tag release

• Bless provisional binaries

• If applicable, update the map that roachtests use to map a version to a previous version, to reference the newly tagged version

• For production or stable releases in the latest major release series

  • Update the Homebrew Formula
  • Update the image tag in our Orchestrator configurations

• For production or stable releases

  • Announce version to the registration cluster

• Update docs

• External communications for release

Cleanup:

• Clean up provisional tag from repository
• Destroy roachprod clusters

(deprecated 8/19 -- previous candidate details below)

Release Blocker Details:

• Issue: sql: panic caused by DEALLOCATE ALL / DISCARD ALL inside of a prepared statement sql: panic caused by DEALLOCATE ALL / DISCARD ALL inside of a prepared statement #52915
  • Backport PR: release-19.2: sql: allow DEALLOCATE ALL with a prepared statement #52992 (master PR sql: allow DEALLOCATE ALL with a prepared statement #52940)
  • See #sql-execution channel for latest discussions

Release Details:

Candidate SHA: 13350e2bebf99997730262bce26b843ed3ecb1be (13350e2, provisional_202008151325_v19.2.10)
Deployment status: ⛔ Release Blocked ⛔
Qualification Suite: #165, ✅ passed
Nightly Suite:

• #2306 (16 Aug 20 15:31) - Nightly Suite
  • #1960 (16 Aug 20 16:23) - Roachtest Nightly - GCE
    • Tests failed: 12 (9 new), passed: 208, ignored: 72 (👍 completed with failures)

Admin UI for Qualification Clusters:

Created	v19.2.9	v19.2.10	2 day check
8/15, 2:30p	v19.2.9	v19.2.10	8/17, 12:30p
Overview (+QPS, Latency)	Overview	Overview	✅
SQL	SQL	SQL	✅
Replication	Replication	Replication	✅

[blathers-crl]

Hi @celiala, please add a C-ategory label to your issue. Check out the label system docs.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[celiala]

🙈🙈 Deprecated / Ignore 🙈🙈

Nightly Tests - Sign-Off checklist

Roachtest Nightly - GCE (Tests failed: 12 (9 new), passed: 208, ignored: 72) - https://teamcity.cockroachdb.com/viewLog.html?buildId=2185654&tab=buildResultsDiv&buildTypeId=Cockroach_Nightlies_WorkloadNightly&branch_Cockroach_Nightlies=provisional_202008151325_v19.2.10

AppDev (🙈 deprecated / ignore)

• django | [appdev]
• pgjdbc | [appdev] (⛔ release blocker sql: panic caused by DEALLOCATE ALL / DISCARD ALL inside of a prepared statement #52915)

CDC (🙈 deprecated / ignore)

• cdc/cloud-sink-gcs/rangefeed=true | [cdc]
• cdc/ledger | [cdc]
• cdc/poller/rangefeed=false | [cdc]
• cdc/sink-chaos | [cdc]
• cdc/tpcc-1000 | [cdc]
• cdc/initial-scan | [cdc] (skipped: roachtest: cdc/crdb-chaos/rangefeed=true failed [skipped] #37716)

KV (🙈 deprecated / ignore)

• follower-reads/nodes=3 | [kv]
• kv/contention/nodes=4 | [kv]
• tpcc-nowait/nodes=3/w=1 | [kv]
• tpccbench/nodes=9/cpu=4/multi-region | [kv]

[rafiss]

Signed off on the Django test -- it's a temporary flake.

The PGJDBC test failure is quite concerning. The PGJDBC test suite is causing a panic in CockroachDB. It is caused by issuing a DEALLOCATE ALL or DISCARD ALL statement from within a prepared statement. The details are in #52915. I think this qualifies as a release blocker until we understand things further.

[nvanbenschoten]

Signing off on all CDC tests. They're all testing failures that will be fixed by #52928.

[nvanbenschoten]

Signing off on follower-reads/nodes=3. This was a test breakage due to the removal of the "experimental" prefix on the experimental_follower_read_timestamp() builtin. This will be fixed by #52925.

[nvanbenschoten]

Signing off on kv/contention/nodes=4. This has known failure modes that we're ok with in v19.2. I've bumped the min version for the test in #52921.

[nvanbenschoten]

Signing off on tpcc-nowait/nodes=3/w=1. The error is concerning, but not overly so as nothing bad will happen if someone hits it and it seems exceedingly rare. I've improved logging here to help us debug in the future in #52923.

[celiala]

Roachtest Nightly - GCE: Test Failure Sign-Off

(Completed: Tests failed: 9 (4 new), passed: 210, ignored: 74)
Build 1967 (19 Aug 20 21:26) - Roachtest Nightly - GCE:
https://teamcity.cockroachdb.com/viewLog.html?buildId=2198863&tab=buildResultsDiv&buildTypeId=Cockroach_Nightlies_WorkloadNightly

appdev

• django | [appdev]
• hibernate | [appdev]
• psycopg | [appdev]

bulkio

• backupTPCC | [bulkio]
• import/tpcc/warehouses=1000/nodes=32 | [bulkio]
• import/tpcc/warehouses=1000/nodes=4 | [bulkio]

kv

• version/mixed/nodes=5 | [kv]
• tpccbench/nodes=9/cpu=4/multi-region | [kv]
• version/mixed/nodes=3 | [kv]

[rafiss]

Signed off on appdev tests. django has a new flaky test that I will ignore in the tests for now. hibernate failed due to a new flake that hasn't happened before, but is not concerning. psycopg is failing because of a setup issue that I'll address today.

[irfansharif]

Signed off on the KV tests.

[adityamaru]

Signed off on BulkIO tests. TPCC tests failed to run due to a workload/version mismatch due to recent changes made to workload.

[andreimatei]

closing as 19.2.10 was released