roachtest: unoptimized-query-oracle/disable-rules=half failed

[cockroach-teamcity]

roachtest.unoptimized-query-oracle/disable-rules=half failed with artifacts on release-22.2.0 @ 77667a1b0101cd323090011f50cf910aaa933654:

test artifacts and logs in: /artifacts/unoptimized-query-oracle/disable-rules=half/run_1
	query_comparison_util.go:237,query_comparison_util.go:66,unoptimized_query_oracle.go:54,test_runner.go:930: failed to set random seed. 26002 statements run: dial tcp 35.185.126.19:26257: connect: connection refused
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*queryComparisonHelper).makeError
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/query_comparison_util.go:345
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runUnoptimizedQueryOracleImpl
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/unoptimized_query_oracle.go:102
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerUnoptimizedQueryOracle.func1.1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/unoptimized_query_oracle.go:57
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runOneRoundQueryComparison
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/query_comparison_util.go:236
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runQueryComparison
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/query_comparison_util.go:66
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerUnoptimizedQueryOracle.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/unoptimized_query_oracle.go:54
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) failed to set random seed. 26002 statements run
		Wraps: (3) dial tcp 35.185.126.19:26257
		Wraps: (4) connect
		Wraps: (5) connection refused
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *net.OpError (4) *os.SyscallError (5) syscall.Errno

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: unoptimized-query-oracle/disable-rules=half/rand-tables failed #91917 roachtest: unoptimized-query-oracle/disable-rules=half/rand-tables failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master]
• roachtest: debug zip cpu profile collection fails #91203 roachtest: unoptimized-query-oracle/disable-rules=half/seed-multi-region failed [C-test-failure O-roachtest O-robot T-sql-queries branch-master]

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-21592

[DrewKimball]

cockroach-health log shows ~16GB total go memory right before the end. The last heap profile doesn't reflect all of this, but it probably shows the problem nonetheless:

Last query looks like this:

SELECT
	'254.239.239.254/0':::INET AS col_108350,
	tab_33887.col1_11 AS col_108351,
	tab_33887.col1_3 AS col_108352,
	tab_33887.col1_5 AS col_108353,
	'0104000020E610000004000000010100000000000000000024400000000000004440010100000000000000000044400000000000003E4001010000000000000000003440000000000000344001010000000000000000003E400000000000002440':::GEOGRAPHY
		AS col_108354,
	'|':::STRING AS col_108355,
	tab_33887.col1_8 AS col_108356
FROM
	defaultdb.public.table1@[0] AS tab_33887
WHERE
	((tab_33887.col1_11::TIME IN (SELECT tab_33887.col1_11::TIME AS col_108347 FROM defaultdb.public.table3@[0] AS tab_33888 WHERE 'm6]':::STRING NOT LIKE rpad('=9vm':::STRING::STRING, tab_33888.col3_8::INT8, e'\x15\x18g':::STRING::STRING)::STRING ORDER BY tab_33888.col3_6 DESC, tab_33888.col3_8 DESC)) AND EXISTS (SELECT tab_33889.col2_8 AS col_108348, 0:::OID AS col_108349 FROM defaultdb.public.table2@[0] AS tab_33889 ORDER BY tab_33889.col2_4 DESC, tab_33889.col2_0 DESC, tab_33889.col2_5));

So likely more rpad/lpad inefficient regexp problems.

[DrewKimball]

Dup of #89749