roachtest: acceptance/version-upgrade failed

[cockroach-teamcity]

roachtest.acceptance/version-upgrade failed with artifacts on release-23.1.0 @ 358e0d87912365b8976c55ab9b3292e999cf720d:

test artifacts and logs in: /artifacts/acceptance/version-upgrade/run_1
(mixedversion.go:366).Run: mixed-version test failure while running step 30 (run "test schema change step"): output in run_053232.261078662_n2_workload-run-schemac: ./workload run schemachange {pgurl:1-4} --concurrency 2 --max-ops 10 --verbose 1 returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_053233.007234233_n2_workload-run-schemac.log: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-27910

[renatolabs]

Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: write-job-info-delete: relation "system.job_info" does not exist (SQLSTATE 42P01)

Dup of #103239.

[msbutler]

saw this in CI here: https://teamcity.cockroachdb.com/viewLog.html?buildId=10246723&problemId=11989

[adityamaru]

@renatolabs is the CI link the same failure mode as system.job_info not being available?

I see:

Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: index element col4_12 of type jsonb is not indexable in a non-inverted index (SQLSTATE 42P16)
run_150321.596784424_n1_workload-run-schemac: 15:03:22 cluster.go:2081: > result: ./workload run schemachange {pgurl:1-4} --concurrency 2 --max-ops 10 --verbose 1 returned: COMMAND_PROBLEM: exit status 1
(1) ./workload run schemachange {pgurl:1-4} --concurrency 2 --max-ops 10 --verbose 1 returned
  | stderr:
  | I230525 15:03:21.764863 1 workload/cli/run.go:432  [-] 1  creating load generator...
  | I230525 15:03:21.868917 1 workload/cli/run.go:471  [-] 2  creating load generator... done (took 104.052151ms)
  | Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: index element col4_12 of type jsonb is not indexable in a non-inverted index (SQLSTATE 42P16)
  |
  | stdout:
  | <... some data truncated by circular buffer; go to artifacts for details ...>
  | 0)) AND (name = $1)), descriptor_ids AS (SELECT nsp.id FROM system.namespace AS nsp, schema_id, database_id WHERE (nsp.\"parentID\" = database_id.id) AND (nsp.\"parentSchemaID\" = schema_id.id)), descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor) AS descriptor FROM system.descriptor AS descriptors JOIN descriptor_ids ON descriptors.id = descriptor_ids.id), types AS (SELECT descriptor FROM descriptors WHERE (descriptor-\u003e'type') IS NOT NULL), table_references AS (SELECT json_array_elements((descriptor-\u003e'table')-\u003e'dependedOnBy') AS ref FROM descriptors WHERE (descriptor-\u003e'table') IS NOT NULL), dependent AS (SELECT (ref-\u003e\u003e'id')::INT8 AS id FROM table_references), referenced_descriptors AS (SELECT json_array_elements_text((descriptor-\u003e'type')-\u003e'referencingDescriptorIds')::INT8 AS id FROM types) SELECT EXISTS (SELECT * FROM system.namespace WHERE ((id IN (SELECT id FROM referenced_descriptors)) AND (\"parentSchemaID\" NOT IN (SELECT id FROM schema_id))) AND (id NOT IN (SELECT id FROM dependent)))",
  |     "queryArgs": [
  |      "public"
  |     ],
  |     "result": false
  |    },
  |    {
  |     "query": "SELECT EXISTS (SELECT schema_name FROM information_schema.schemata WHERE schema_name = $1)",
  |     "queryArgs": [
  |      "schema2"
  |     ],
  |     "result": false
  |    },
  |    {
  |     "query": "WITH database_id AS (SELECT id FROM system.namespace WHERE ((\"parentID\" = 0) AND (\"parentSchemaID\" = 0)) AND (name = current_database())), schema_id AS (SELECT nsp.id FROM system.namespace AS nsp JOIN database_id ON ((\"parentID\" = database_id.id) AND (\"parentSchemaID\" = 0)) AND (name = $1)), descriptor_ids AS (SELECT nsp.id FROM system.namespace AS nsp, schema_id, database_id WHERE (nsp.\"parentID\" = database_id.id) AND (nsp.\"parentSchemaID\" = schema_id.id)), descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor) AS descriptor FROM system.descriptor AS descriptors JOIN descriptor_ids ON descriptors.id = descriptor_ids.id), types AS (SELECT descriptor FROM descriptors WHERE (descriptor-\u003e'type') IS NOT NULL), table_references AS (SELECT json_array_elements((descriptor-\u003e'table')-\u003e'dependedOnBy') AS ref FROM descriptors WHERE (descriptor-\u003e'table') IS NOT NULL), dependent AS (SELECT (ref-\u003e\u003e'id')::INT8 AS id FROM table_references), referenced_descriptors AS (SELECT json_array_elements_text((descriptor-\u003e'type')-\u003e'referencingDescriptorIds')::INT8 AS id FROM types) SELECT EXISTS (SELECT * FROM system.namespace WHERE ((id IN (SELECT id FROM referenced_descriptors)) AND (\"parentSchemaID\" NOT IN (SELECT id FROM schema_id))) AND (id NOT IN (SELECT id FROM dependent)))",
  |     "queryArgs": [
  |      "schema2"
  |     ],
  |     "result": false
  |    },
  |    {
  |     "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
  |     "result": false
  |    },
  |    {
  |     "query": "SELECT EXISTS (SELECT table_name FROM information_schema.tables WHERE (table_schema = $1) AND (table_name = $2))",
  |     "queryArgs": [
  |      "public",
  |      "table4"
  |     ],
  |     "result": false
  |    },
  |    {
  |     "query": "SELECT EXISTS (SELECT schema_name FROM information_schema.schemata WHERE schema_name = $1)",
  |     "queryArgs": [
  |      "public"
  |     ],
  |     "result": true
  |    }
  |   ],
  |   "previousStatements": [
  |    "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
  |    "CREATE TABLE IF NOT EXISTS public.table4 (col4_5 FLOAT4 NOT NULL, col4_6 CHAR NOT NULL, col4_7 TIMETZ NOT NULL, \"\"\"col4_8\" INT4 NOT NULL, \"col 4_9\" GEOGRAPHY NOT NULL, \"col\"\"4_10\" GEOMETRY NOT NULL, \"c.ol4_11\" TIMETZ NOT NULL, col4_12 JSONB NOT NULL, col4_13 STRING NOT NULL AS (lower(CAST(col4_7 AS STRING))) STORED, PRIMARY KEY (\"c.ol4_11\" ASC, col4_6 DESC, col4_12), INDEX (col4_7, col4_12 ASC), INDEX (col4_12 DESC, \"\"\"col4_8\", col4_13 DESC, col4_5 ASC) STORING (col4_7), FAMILY (col4_5), FAMILY (\"col\"\"4_10\"), FAMILY (col4_13), FAMILY (col4_12), FAMILY (\"\"\"col4_8\", col4_6), FAMILY (col4_7), FAMILY (\"col 4_9\"), FAMILY (\"c.ol4_11\"))"
  |   ]
  |  }
  | }
Wraps: (2) Node 1. Command with error:
  | ```
  | ./workload run schemachange {pgurl:1-4} --concurrency 2 --max-ops 10 --verbose 1
  | ```
Wraps: (3) COMMAND_PROBLEM
Wraps: (4) exit status 1
Error types: (1) *cluster.WithCommandDetails (2) *hintdetail.withDetail (3) errors.Cmd (4) *exec.ExitError

but maybe I'm looking in the wrong place.

[renatolabs]

Indeed, seems like the link @msbutler shared is a different issue (which is fixed on master: #103679). The original roachtest failure, however, is the system.job_info failure.

[blathers-crl]

cc @cockroachdb/disaster-recovery

[dt]

23.1.0 dupe of #105032 / #103239