Skip to content

Commit

Permalink
Update all GC TTL and range size mentions to DRY
Browse files Browse the repository at this point in the history
... where DRY = Don't Repeat Yourself

Addresses:

DOC-6539
DOC-6540
DOC-6820
DOC-6834

Summary of changes:

- Update all mentions in v23.1 docs of the default GC
  TTL (`gc.ttlseconds`) and default/max range size (`range_max_bytes`)
  to point to their respective entries on the 'Configure Replication
  Zones' page.

- Update the description of the `gc.ttlseconds` zcfg variable to link to
  more salient info re: MVCC, range splits, etc. to provide the reader
  with add'l context

- Make a few small text updates to weave in the above, as well as a few
  opportunistic link additions (to range splitting docs, for example)
  • Loading branch information
rmloveland committed Mar 16, 2023
1 parent 6a31ab9 commit 3b549dc
Show file tree
Hide file tree
Showing 18 changed files with 26 additions and 26 deletions.
2 changes: 1 addition & 1 deletion _includes/v23.1/faq/auto-generate-unique-ids.html
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
(3 rows)
~~~

In either case, generated IDs will be 128-bit, large enough for there to be virtually no chance of generating non-unique values. Also, once the table grows beyond a single key-value range (more than 512 MiB by default), new IDs will be scattered across all of the table's ranges and, therefore, likely across different nodes. This means that multiple nodes will share in the load.
In either case, generated IDs will be 128-bit, large enough for there to be virtually no chance of generating non-unique values. Also, once the table grows beyond a single key-value range's [default size](configure-replication-zones.html#range-max-bytes), new IDs will be scattered across all of the table's ranges and, therefore, likely across different nodes. This means that multiple nodes will share in the load.

This approach has the disadvantage of creating a primary key that may not be useful in a query directly, which can require a join with another table or a secondary index.

Expand Down
2 changes: 1 addition & 1 deletion _includes/v23.1/misc/basic-terms.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ An individual instance of CockroachDB. One or more nodes form a cluster.
<a name="architecture-range"></a>
CockroachDB stores all user data (tables, indexes, etc.) and almost all system data in a sorted map of key-value pairs. This keyspace is divided into contiguous chunks called _ranges_, such that every key is found in one range.

From a SQL perspective, a table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the _primary index_ because the table is sorted by the primary key) or a single row in a secondary index. As soon as the size of a range reaches 512 MiB ([the default](../configure-replication-zones.html#range-max-bytes)), it is split into two ranges. This process continues for these new ranges as the table and its indexes continue growing.
From a SQL perspective, a table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the _primary index_ because the table is sorted by the primary key) or a single row in a secondary index. As soon as the size of a range reaches [the default range size](../configure-replication-zones.html#range-max-bytes), it is [split into two ranges](distribution-layer.html#range-splits). This process continues for these new ranges as the table and its indexes continue growing.

### Replica
<a name="architecture-replica"></a>
Expand Down
2 changes: 1 addition & 1 deletion _includes/v23.1/sql/range-splits.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CockroachDB breaks data into ranges. By default, CockroachDB attempts to keep ranges below a size of 512 MiB. To do this, the system will automatically [split a range](architecture/distribution-layer.html#range-splits) if it grows larger than this limit. For most use cases, this automatic range splitting is sufficient, and you should never need to worry about when or where the system decides to split ranges.
CockroachDB breaks data into ranges. By default, CockroachDB attempts to keep ranges below [the default range size](configure-replication-zones.html#range-max-bytes). To do this, the system will automatically [split a range](architecture/distribution-layer.html#range-splits) if it grows larger than this limit. For most use cases, this automatic range splitting is sufficient, and you should never need to worry about when or where the system decides to split ranges.

However, there are reasons why you may want to perform manual splits on the ranges that store tables or indexes:

Expand Down
4 changes: 2 additions & 2 deletions _includes/v23.1/zone-configs/variables.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
Variable | Description
------|------------
`range_min_bytes` | <a name="range-min-bytes"></a> The minimum size, in bytes, for a range of data in the zone. When a range is less than this size, CockroachDB will merge it with an adjacent range.<br><br>**Default:** `134217728` (128 MiB)
`range_max_bytes` | <a name="range-max-bytes"></a> The maximum size, in bytes, for a range of data in the zone. When a range reaches this size, CockroachDB will split it into two ranges.<br><br>**Default:** `536870912` (512 MiB)
`gc.ttlseconds` | <a name="gc-ttlseconds"></a> The number of seconds overwritten values will be retained before garbage collection. Smaller values can save disk space if values are frequently overwritten; larger values increase the range allowed for `AS OF SYSTEM TIME` queries, also know as [Time Travel Queries](select-clause.html#select-historical-data-time-travel).<br><br>It is not recommended to set this below `600` (10 minutes); doing so will cause problems for long-running queries. Also, since all versions of a row are stored in a single range that never splits, it is not recommended to set this so high that all the changes to a row in that time period could add up to more than 512 MiB; such oversized ranges could contribute to the server running out of memory or other problems. {{site.data.alerts.callout_info}} Ensure that you set `gc.ttlseconds` long enough to accommodate your [backup schedule](create-schedule-for-backup.html), otherwise your incremental backups will fail with [this error](common-errors.html#protected-ts-verification-error). For example, if you set up your backup schedule to recur daily, but you set `gc.ttlseconds` to less than one day, all your incremental backups will fail.{{site.data.alerts.end}} **Default:** `90000` (25 hours)
`range_max_bytes` | <a name="range-max-bytes"></a> The maximum size, in bytes, for a [range]({{link_prefix}}architecture/glossary.html#architecture-range) of data in the zone. When a range reaches this size, CockroachDB will [split it]({{link_prefix}}architecture/distribution-layer.html#range-splits) into two ranges.<br><br>**Default:** `536870912` (512 MiB)
`gc.ttlseconds` | <a name="gc-ttlseconds"></a> The number of seconds overwritten [MVCC values]({{link_prefix}}architecture/storage-layer.html#mvcc) will be retained before [garbage collection]({{link_prefix}}architecture/storage-layer.html#garbage-collection). Smaller values can save disk space if values are frequently overwritten; larger values increase the interval allowed for [`AS OF SYSTEM TIME`](as-of-system-time.html) queries, also known as [Time Travel Queries](select-clause.html#select-historical-data-time-travel).<br><br>It is not recommended to set this below `600` (10 minutes); doing so will cause problems for [long-running queries](manage-long-running-queries.html). Also, since all versions of a row are stored in a single range that never splits, it is not recommended to set this so high that all the changes to a row in that time period could add up to more than [the default range size](#range-max-bytes); such oversized ranges could contribute to the server [running out of memory](cluster-setup-troubleshooting.html#out-of-memory-oom-crash) or other problems. {{site.data.alerts.callout_info}} Ensure that you set `gc.ttlseconds` long enough to accommodate your [backup schedule](create-schedule-for-backup.html), otherwise your incremental backups will fail with [the error message `protected ts verification error`](common-errors.html#protected-ts-verification-error). For example, if you set up your backup schedule to recur at an interval greater than `gc.ttlseconds`, all your incremental backups will fail.{{site.data.alerts.end}} **Default:** `14400` (4 hours)
`num_replicas` | <a name="num_replicas"></a> The number of replicas in the zone, also called the "replication factor".<br><br>**Default:** `3`<br><br>For the `system` database and `.meta`, `.liveness`, and `.system` ranges, the default value is `5`.<br /><br />For [multi-region databases configured to survive region failures](multiregion-overview.html#surviving-region-failures), the default value is `5`; this will include both [voting](#num_voters) and [non-voting replicas](architecture/replication-layer.html#non-voting-replicas).
`constraints` | <a name="constraints"></a> An array of required (`+`) and/or prohibited (`-`) constraints influencing the location of replicas. See [Types of Constraints](configure-replication-zones.html#types-of-constraints) and [Scope of Constraints](configure-replication-zones.html#scope-of-constraints) for more details.<br/><br/>To prevent hard-to-detect typos, constraints placed on [store attributes and node localities](configure-replication-zones.html#descriptive-attributes-assigned-to-nodes) must match the values passed to at least one node in the cluster. If not, an error is signalled. To prevent this error, make sure at least one active node is configured to match the constraint. For example, apply `constraints = '[+region=west]'` only if you had set `--locality=region=west` for at least one node while starting the cluster.<br/><br/>**Default:** No constraints, with CockroachDB locating each replica on a unique node and attempting to spread replicas evenly across localities.
`lease_preferences` <a name="lease_preferences"></a> | An ordered list of required and/or prohibited constraints influencing the location of [leaseholders](architecture/glossary.html#architecture-leaseholder). Whether each constraint is required or prohibited is expressed with a leading `+` or `-`, respectively. Note that lease preference constraints do not have to be shared with the `constraints` field. For example, it's valid for your configuration to define a `lease_preferences` field that does not reference any values from the `constraints` field. It's also valid to define a `lease_preferences` field with no `constraints` field at all. <br /><br /> If the first preference cannot be satisfied, CockroachDB will attempt to satisfy the second preference, and so on. If none of the preferences can be met, the lease will be placed using the default lease placement algorithm, which is to base lease placement decisions on how many leases each node already has, trying to make all the nodes have around the same amount.<br /><br />Each value in the list can include multiple constraints. For example, the list `[[+zone=us-east-1b, +ssd], [+zone=us-east-1a], [+zone=us-east-1c, +ssd]]` means "prefer nodes with an SSD in `us-east-1b`, then any nodes in `us-east-1a`, then nodes in `us-east-1c` with an SSD."<br /><br /> For a usage example, see [Constrain leaseholders to specific availability zones](configure-replication-zones.html#constrain-leaseholders-to-specific-availability-zones).<br /><br />**Default**: No lease location preferences are applied if this field is not specified.
Expand Down
8 changes: 4 additions & 4 deletions v23.1/architecture/distribution-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ Each node caches values of the `meta2` range it has accessed before, which optim

After the node's meta ranges is the KV data your cluster stores.

Each table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as a range reaches 512 MiB in size, it splits into two ranges. This process continues as a table and its indexes continue growing. Once a table is split across multiple ranges, it's likely that the table and secondary indexes will be stored in separate ranges. However, a range can still contain data for both the table and a secondary index.
Each table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as a range reaches [the default range size](../configure-replication-zones.html#range-max-bytes), it splits into two ranges. This process continues as a table and its indexes continue growing. Once a table is split across multiple ranges, it's likely that the table and secondary indexes will be stored in separate ranges. However, a range can still contain data for both the table and a secondary index.

The default 512 MiB range size represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability.
The [default range size](../configure-replication-zones.html#range-max-bytes) represents a sweet spot for us between a size that's small enough to move quickly between nodes, but large enough to store a meaningfully contiguous set of data whose keys are more likely to be accessed together. These ranges are then shuffled around your cluster to ensure survivability.

These table ranges are replicated (in the aptly named replication layer), and have the addresses of each replica stored in the `meta2` range.

Expand Down Expand Up @@ -186,13 +186,13 @@ All of these updates to the range descriptor occur locally on the range, and the
### Range splits
By default, CockroachDB attempts to keep ranges/replicas at the default range size (currently 512 MiB). Once a range reaches that limit, we split it into two smaller ranges (composed of contiguous key spaces).
By default, CockroachDB attempts to keep ranges/replicas at [the default range size](../configure-replication-zones.html#range-max-bytes). Once a range reaches that limit, we split it into two smaller ranges (composed of contiguous key spaces).
During this range split, the node creates a new Raft group containing all of the same members as the range that was split. The fact that there are now two ranges also means that there is a transaction that updates `meta2` with the new keyspace boundaries, as well as the addresses of the nodes using the range descriptor.
### Range merges
By default, CockroachDB automatically merges small ranges of data together to form fewer, larger ranges (up to the default range size). This can improve both query latency and cluster survivability.
By default, CockroachDB automatically merges small ranges of data together to form fewer, larger ranges (up to [the default range size](../configure-replication-zones.html#range-max-bytes)). This can improve both query latency and cluster survivability.
#### How range merges work
Expand Down
2 changes: 1 addition & 1 deletion v23.1/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Once the CockroachDB cluster is initialized, developers interact with CockroachD

After receiving SQL remote procedure calls (RPCs), nodes convert them into key-value (KV) operations that work with our [distributed, transactional key-value store](transaction-layer.html).

As these RPCs start filling your cluster with data, CockroachDB starts [algorithmically distributing your data among the nodes of the cluster](distribution-layer.html), breaking the data up into 512 MiB chunks that we call ranges. Each range is replicated to at least 3 nodes by default to ensure survivability. This ensures that if any nodes go down, you still have copies of the data which can be used for:
As these RPCs start filling your cluster with data, CockroachDB starts [algorithmically distributing your data among the nodes of the cluster](distribution-layer.html), breaking the data up into chunks that we call ranges. Each range is replicated to at least 3 nodes by default to ensure survivability. This ensures that if any nodes go down, you still have copies of the data which can be used for:

- Continuing to serve reads and writes.
- Consistently replicating the data to other nodes.
Expand Down
2 changes: 1 addition & 1 deletion v23.1/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ CockroachDB stores full backups in a backup collection. Each full backup in a co
`LATEST` | Append an incremental backup to the latest completed full backup's subdirectory.
<a name="collectionURI-param"></a> `collectionURI` | The URI where you want to store the backup. (Or, the default locality for a locality-aware backup.)<br/><br/>For information about this URL structure, see [Backup File URLs](#backup-file-urls).
`localityURI` | The URI containing the `COCKROACH_LOCALITY` parameter for a non-default locality that is part of a single locality-aware backup.
`timestamp` | Back up data as it existed as of [`timestamp`](as-of-system-time.html). The `timestamp` must be more recent than your cluster's last garbage collection (which defaults to occur every 25 hours, but is [configurable per table](configure-replication-zones.html#replication-zone-variables)).
`timestamp` | Back up data as it existed as of [`timestamp`](as-of-system-time.html). The `timestamp` must be more recent than your cluster's last garbage collection (which occurs at a [configurable interval](configure-replication-zones.html#gc-ttlseconds)).
`backup_options` | Control the backup behavior with a comma-separated list of [these options](#options).

### Targets
Expand Down
2 changes: 1 addition & 1 deletion v23.1/common-errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ When running a multi-node CockroachDB cluster, if you see an error like the one

## split failed while applying backpressure; are rows updated in a tight loop?

In CockroachDB, a table row is stored on disk as a key-value pair. Whenever the row is updated, CockroachDB also stores a distinct version of the key-value pair to enable concurrent request processing while guaranteeing consistency (see [multi-version concurrency control (MVCC)](architecture/storage-layer.html#mvcc)). All versions of a key-value pair belong to a larger ["range"](architecture/overview.html#architecture-range) of the total key space, and the historical versions remain until the garbage collection period defined by the `gc.ttlseconds` variable in the applicable [zone configuration](configure-replication-zones.html#gc-ttlseconds) has passed (25 hours by default). Once a range reaches a size threshold (512 MiB by default), CockroachDB splits the range into two ranges. However, this message indicates that a range cannot be split as intended.
In CockroachDB, a table row is stored on disk as a key-value pair. Whenever the row is updated, CockroachDB also stores a distinct version of the key-value pair to enable concurrent request processing while guaranteeing consistency (see [multi-version concurrency control (MVCC)](architecture/storage-layer.html#mvcc)). All versions of a key-value pair belong to a larger ["range"](architecture/overview.html#architecture-range) of the total key space, and the historical versions remain until the garbage collection period defined by the `gc.ttlseconds` variable in the applicable [zone configuration](configure-replication-zones.html#gc-ttlseconds) has passed. Once a range reaches a [size threshold](configure-replication-zones.html#range-max-bytes), CockroachDB [splits the range](architecture/distribution-layer.html#range-splits) into two ranges. However, this message indicates that a range cannot be split as intended.

One possible cause is that the range consists only of MVCC version data due to a row being repeatedly updated, and the range cannot be split because doing so would spread MVCC versions for a single row across multiple ranges.

Expand Down
Loading

0 comments on commit 3b549dc

Please sign in to comment.