roachperf: regression around 2022-07-07

[renatolabs]

kv95:

Similar drops are seen in kv95/nodes=3 and some ycsb benchmarks.

% GOOD_SHA=41228d10ac5e326328507b98245fc076d67d2ecd
% BAD_SHA=7405c568e482fbfff4ad2bcba101ef42678d0b1a
% git log $BAD_SHA --not $GOOD_SHA^ --topo-order --merges --pretty="format:%h %s (%ch)" | sed 's/^/- [ ] /g'

• ab9eabf Merge server: pass AC soft slot granter to Pebble #84006 builtins: require admin to use crdb_internal.compact_engine_spans #84036 vendor: bump Pebble to d68c152233af #84051 (Fri Jul 8 16:53)
• 4d8d0b5 Merge DOC-4629: Update SQL diagrams surrounding storage parameters #83529 sql: Use one BytesMonitor for all IE created by IEFactory  #83678 ci: ensure jemalloc is configured _without_ MADV_FREE #83795 changefeedccl: Add timeout to testfeed library. #84007 json: make checkLength assertion error "regular" #84046 (Fri Jul 8 15:05)
• 18f6450 Merge storage: link to MVCC range tombstones tech note #84062 (Fri Jul 8 13:53)
• 672f201 Merge storage: add nextKeyIgnoringTime() to MVCCIncrementalIterator #83729 (Fri Jul 8 06:11)
• cd1abbb Merge dev,genbzl: add more dev generate subcommands #84005 (Fri Jul 8 04:58)
• 88789fe Merge rfcs: mark cluster_locks RFC as completed #83484 (Fri Jul 8 03:49)
• 8edc7ba Merge sql/catalog/descs: fix typo #83941 (Fri Jul 8 01:41)
• 365b9c4 Merge kvserver: allow voter demoting to get a lease, in case there's an incoming voter #83686 (Fri Jul 8 00:20)
• 2dd8e76 Merge ui: add internal app filter to active statements and transactions pages #83014 roachtest: run workload from the tenant node #83707 storage: close pebble iter gracefully when NewPebbleSSTIterator fails #84003 prometheus: use older node_exporter #84039 (Thu Jul 7 22:04)
• 6b7082a Merge opt: fix memo cycle caused by join ordering #83875 storage/metamorphic: Add MVCC delete range using tombstone #83945 server: log stacks on 500 errors #83992 bazel: clear configurations when running git grep in check.sh #84015 (Thu Jul 7 20:57)
• e67e47f Merge admission,kvserver: broadcast per-store IO overload status #83720 sql: move NullableArgs function property to overload level #83732 sql: add identifiers to sampled query #83938 reducesql: reduce index STORING columns #83953 (Thu Jul 7 18:12)
• bce2267 Merge sql/stats: conversion of datums to and from quantile function values #83123 sql: fix and rename sql stats session transaction received time #83590 outliers: protocol buffer adjustments #83943 cmd/dev: add generate execgen subcommand #83961 streamingccl: unskip TestTenantStreaming #83975 streamingccl: small logging and tracing cleanups #83979 (Thu Jul 7 17:02) 🔴🔴 (mean: 63677.4, stdev: 523.7)
• 4dfbe65 Merge stmtdiagnostics: remove the usage of gossip #83547 ui/cluster-ui: filter out closed sessions from active exec pages  #83896 ui: persist stmt view selection in sql activity page #83903 kvstreamer: reuse the truncation helper and introduce a fast path for a single range case #83926 (Thu Jul 7 15:44)
• 3b22cdd Merge Update row level TTL RFC #83908 (Thu Jul 7 13:03)
• 5fb4478 Merge docs: add volatility to built-in functions table #83331 kvstreamer: remove some redundant function calls #83685 (Thu Jul 7 07:25)
• f636444 Merge roachtest: add unoptimized-query-oracle test to validate query results #83095 sql: add access control checks to multi-region related builtins #83918 (Thu Jul 7 05:41)
• f93ca3a Merge sql/schemachanger: extend metadata updater to support for updating zone configs #83804 (Thu Jul 7 04:33)
• 03d5260 Merge pgwire: fix buffer size limiting behavior for addBatch #83870 tree: support row_to_json on VOID type #83876 parser: require single-quotes when setting password #83924 changefeedccl: Skip some flaky tests. #83947 builtins: check privileges for crdb_internal builtins #83952 (Thu Jul 7 02:58)
• 1abab4c Merge sql/schemachanger: check nil table name for comment on column #83812 roachprod: fix target path when getting cluster artifacts. #83923 storage: clarify and test overlapping SST range keys in Export #83927 coexec: normalize the timestamp subtraction like postgres #83944 (Thu Jul 7 01:52) 🔴🔴 (mean: 62634.5, stdev: 1940.1)
• a0b1222 Merge pgwire: allow Flush message during portal exeuction #83675 sql/builtins: tenant-related builtins require admin role #83800 workload/schemachanger: stabilize schema changer workload #83838 (Thu Jul 7 00:15) 🟡🟡 (mean: 64224.3, stdev: 704.4)
• ce1b42b Merge sql, sqlstats: create temporary stats container for all txns #83616 docs: add MVCC range tombstones tech note #83762 tenantsettingswatcher: remove the version gate #83873 server: remove TLS cert data retrieval over HTTP #83902 (Wed Jul 6 22:45) 🔴🔴 (mean: 64129.5, stdev: 1555.2)
• 0848d90 Merge sql: remove ttl_automatic_column storage parameter #83134 sql: don't clear the memory account of the prepared stmt #83615 (Wed Jul 6 21:37) 🔴🔴 (mean: 62893.2, stdev: 2932.7) -- second run: 🟢🟢 (mean: 67211.4, stdev: 1107.4).
  • Reverting sql: don't clear the memory account of the prepared stmt #83615: 🟢🟢 (mean: 68235.1, stdev: 269.6)
  • Reverting sql: remove ttl_automatic_column storage parameter #83134: 🟢🟢 (mean: 68038.3, stdev: 286.4)
  • ❓❓
• 41228d1 Merge kvstreamer: reuse incomplete Get requests on resume batches #83684 pkg/util/tracing: Add hidden tag group, make server responsible for sorting. #83709 outliers: extract a Registry interface. #83834 (Wed Jul 6 20:27) 🟢🟢 (mean: 67750.42, stdev: 742.2)
• 264cd04 Merge admission: introduce soft slots for speculative concurrency #78519 sql: fix error log spam from SHOW CREATE #83421 (Wed Jul 6 19:20) 🟢 🟢 (mean: 68207.9, stdev: 413.9)
• 7cfa5d4 Merge bazel: in check.sh, restrict grep to pkg #83866 (Wed Jul 6 14:04)
• 8b6e3c3 Merge roachtest,roachprod: detect tenant-scope certs via help, unskip tests #83703 (Wed Jul 6 12:57)
• d239a2f Merge kv: fix conflict resolution for high-priority, non-txn'al requests #83366 util/mon: pass reserved account by reference #83629 (Wed Jul 6 06:37)
• 90b15cf Merge kvcoord: optimize batch truncation loop #82865 sql: initialize sql instance during instance provider start #83358 kvstreamer: perform more memory accounting #83683 (Wed Jul 6 05:27)
• 33d7099 Merge kv: add debug information to track down #77663 #82151 (Wed Jul 6 02:25)
• 9c5472e Merge bazel: delete bespoke string_test target #83711 roachtest: unskip large decommissionBench test #83733 sql/sem/builtins: deflake TestSerialNormalizationWithUniqueUnorderedID #83809 (Tue Jul 5 19:22) 🟢🟢 (mean: 68017, stdev: 1618.3)

Jira issue: CRDB-17923

[smg260]

This regression seems to coincide with #88122 which singles out sql, sqlstats: create temporary stats container for all txns as the probable cause. Merge commit: ce1b42b

[srosenberg]

Root cause appears to be sqlstats, closing.