Merge #93836

93836: *: lower default GC TTL to 4h r=irfansharif a=irfansharif

Fixes #89233.

Release note (general change): The GC TTL previously defaulted to 25h. This value was configurable using `ALTER RANGE DEFAULT CONFIGURE ZONE USING gc.ttlseconds = <whatever>`, but also possible to scope to specific schema objects using `ALTER {DATABASE,TABLE,INDEX} CONFIGURE ZONE USING ...`. This value determine how long overwritten values were retained. The `RANGE DEFAULT` value is now lowered to 4h but only for freshly created clusters. When existing clusters upgrade onto this release, we will respect whatever value they were using before the upgrade for all their schema objects. This will be 25h if the GC TTL was never altered, or some specific value if set explicitly. Full cluster backups taken on earlier version clusters, when restored to clusters that started off at v23.1, will use the GC TTL recorded in the backup image.

We've found the 25h value to translate to higher-than-necessary storage costs, especially for workloads where rows are deleted frequently. It can also make for costlier reads with respect to CPU since we currently have to scan over overwritten values to get to the one of interest. Finally, we've also observed cluster instability due to large unsplittable ranges that have accumulated an excessive amount of MVCC garbage. We chose a default of 25h originally to accommodate daily incremental backups with revision history. But with the introduction of scheduled backups introduced in 22.2, we no longer need a large GC TTL. Scheduled backups "chain together" and prevent garbage collection of relevant data to ensure coverage of revision history across backups, decoupling it from whatever value is used for GC TTL. So we no longer need a 25h default, hence this change.

The GC TTL determines how far back AS OF SYSTEM TIME queries can go, which now if going past `now()-4h`, will start failing informatively. To support larger windows for AS OF SYSTEM TIME queries, users are encouraged to pick a more appropriate GC TTL and set it using `ALTER ... CONFIGURE ZONE using gc.ttlseconds = <whatever>`. The earlier considerations around storage use, read costs, and stability still apply.

Release note (backward-incompatible change): See release note above. Technically this is not a backwards-incompatible change since we're only changing the default value that new clusters are initialized with -- existing clusters will remain unaffected. But it might be worth highlighting this change more prominently to our users for added scrutiny.

Co-authored-by: irfan sharif <[email protected]>

pkg/ccl/logictestccl/testdata/logic_test/crdb_internal

@@ -80,37 +80,37 @@ query TT
 SELECT * FROM [SHOW ALL ZONE CONFIGURATIONS] ORDER BY 1
 ----
 INDEX test.public.t4@myt4index  ALTER INDEX test.public.t4@myt4index CONFIGURE ZONE USING
-                                num_replicas = 5
+                                  num_replicas = 5
 RANGE default                   ALTER RANGE default CONFIGURE ZONE USING
-                                range_min_bytes = 134217728,
-                                range_max_bytes = 536870912,
-                                gc.ttlseconds = 90000,
-                                num_replicas = 3,
-                                constraints = '[]',
-                                lease_preferences = '[]'
+                                  range_min_bytes = 134217728,
+                                  range_max_bytes = 536870912,
+                                  gc.ttlseconds = 14400,
+                                  num_replicas = 3,
+                                  constraints = '[]',
+                                  lease_preferences = '[]'
 RANGE liveness                  ALTER RANGE liveness CONFIGURE ZONE USING
-                                range_min_bytes = 134217728,
-                                range_max_bytes = 536870912,
-                                gc.ttlseconds = 600,
-                                num_replicas = 5,
-                                constraints = '[]',
-                                lease_preferences = '[]'
+                                  range_min_bytes = 134217728,
+                                  range_max_bytes = 536870912,
+                                  gc.ttlseconds = 600,
+                                  num_replicas = 5,
+                                  constraints = '[]',
+                                  lease_preferences = '[]'
 RANGE meta                      ALTER RANGE meta CONFIGURE ZONE USING
-                                range_min_bytes = 134217728,
-                                range_max_bytes = 536870912,
-                                gc.ttlseconds = 3600,
-                                num_replicas = 5,
-                                constraints = '[]',
-                                lease_preferences = '[]'
+                                  range_min_bytes = 134217728,
+                                  range_max_bytes = 536870912,
+                                  gc.ttlseconds = 3600,
+                                  num_replicas = 5,
+                                  constraints = '[]',
+                                  lease_preferences = '[]'
 RANGE system                    ALTER RANGE system CONFIGURE ZONE USING
-                                range_min_bytes = 134217728,
-                                range_max_bytes = 536870912,
-                                gc.ttlseconds = 90000,
-                                num_replicas = 5,
-                                constraints = '[]',
-                                lease_preferences = '[]'
+                                  range_min_bytes = 134217728,
+                                  range_max_bytes = 536870912,
+                                  gc.ttlseconds = 14400,
+                                  num_replicas = 5,
+                                  constraints = '[]',
+                                  lease_preferences = '[]'
 TABLE test.public.t4            ALTER TABLE test.public.t4 CONFIGURE ZONE USING
-                                num_replicas = 7
+                                  num_replicas = 7
 
 query error pq: user testuser has no privileges on database db2
 SHOW ZONE CONFIGURATION FOR DATABASE db2
@@ -128,34 +128,34 @@ query TT
 SHOW ZONE CONFIGURATION FOR INDEX myt4index
 ----
 INDEX test.public.t4@myt4index  ALTER INDEX test.public.t4@myt4index CONFIGURE ZONE USING
-                                range_min_bytes = 134217728,
-                                range_max_bytes = 536870912,
-                                gc.ttlseconds = 90000,
-                                num_replicas = 5,
-                                constraints = '[]',
-                                lease_preferences = '[]'
+                                  range_min_bytes = 134217728,
+                                  range_max_bytes = 536870912,
+                                  gc.ttlseconds = 14400,
+                                  num_replicas = 5,
+                                  constraints = '[]',
+                                  lease_preferences = '[]'
 
 query TT
 SHOW ZONE CONFIGURATION FOR TABLE t4
 ----
 TABLE t4  ALTER TABLE t4 CONFIGURE ZONE USING
-          range_min_bytes = 134217728,
-          range_max_bytes = 536870912,
-          gc.ttlseconds = 90000,
-          num_replicas = 7,
-          constraints = '[]',
-          lease_preferences = '[]'
+            range_min_bytes = 134217728,
+            range_max_bytes = 536870912,
+            gc.ttlseconds = 14400,
+            num_replicas = 7,
+            constraints = '[]',
+            lease_preferences = '[]'
 
 query TT
 SHOW ZONE CONFIGURATION FOR RANGE default
 ----
 RANGE default  ALTER RANGE default CONFIGURE ZONE USING
-               range_min_bytes = 134217728,
-               range_max_bytes = 536870912,
-               gc.ttlseconds = 90000,
-               num_replicas = 3,
-               constraints = '[]',
-               lease_preferences = '[]'
+                 range_min_bytes = 134217728,
+                 range_max_bytes = 536870912,
+                 gc.ttlseconds = 14400,
+                 num_replicas = 3,
+                 constraints = '[]',
+                 lease_preferences = '[]'
 
 # This test checks that generator backed tables do not concurrently
 # access transactions. It does this by scanning two virtual tables at a time.