diff --git a/TOC.md b/TOC.md index f26bafe97a825..625516bc6e07e 100644 --- a/TOC.md +++ b/TOC.md @@ -140,6 +140,7 @@ + Tutorials + [Multiple Data Centers in One City Deployment](/multi-data-centers-in-one-city-deployment.md) + [Three Data Centers in Two Cities Deployment](/three-data-centers-in-two-cities-deployment.md) + + [Two Data Centers in One City Deployment](/two-data-centers-in-one-city-deployment.md) + Read Historical Data + Use Stale Read (Recommended) + [Usage Scenarios of Stale Read](/stale-read.md) diff --git a/media/two-dc-replication-1.png b/media/two-dc-replication-1.png new file mode 100644 index 0000000000000..e04dad91c9b34 Binary files /dev/null and b/media/two-dc-replication-1.png differ diff --git a/pd-configuration-file.md b/pd-configuration-file.md index c3ddade22d85f..4dd67c23dbddf 100644 --- a/pd-configuration-file.md +++ b/pd-configuration-file.md @@ -359,4 +359,4 @@ Configuration items related to the [TiDB Dashboard](/dashboard/dashboard-intro.m ## `replication-mode` -Configuration items related to the replication mode of all Regions. See [Enable synchronous replication in PD configuration file](/synchronous-replication.md#enable-synchronous-replication-in-the-pd-configuration-file) for details. +Configuration items related to the replication mode of all Regions. See [Enable the DR Auto-Sync mode](/two-data-centers-in-one-city-deployment.md#enable-the-dr-auto-sync-mode) for details. diff --git a/pd-control.md b/pd-control.md index fe003c7457ba6..806541d001f25 100644 --- a/pd-control.md +++ b/pd-control.md @@ -308,7 +308,7 @@ Usage: config set cluster-version 1.0.8 // Set the version of the cluster to 1.0.8 ``` -- `replication-mode` controls the replication mode of Regions in the dual data center scenario. See [Change replication mode manually](/synchronous-replication.md#change-the-replication-mode-manually) for details. +- `replication-mode` controls the replication mode of Regions in the dual data center scenario. See [Enable the DR Auto-Sync mode](/two-data-centers-in-one-city-deployment.md#enable-the-dr-auto-sync-mode) for details. - `leader-schedule-policy` is used to select the scheduling strategy for the leader. You can schedule the leader according to `size` or `count`. diff --git a/synchronous-replication.md b/synchronous-replication.md deleted file mode 100644 index 16b51c2a31b3b..0000000000000 --- a/synchronous-replication.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -title: Synchronous Replication for Dual Data Centers -summary: Learn how to configure synchronous replication for dual data centers. ---- - -# Synchronous Replication for Dual Data Centers - -This document introduces how to configure synchronous replication for dual data centers. - -> **Warning:** -> -> Synchronous replication is still an experimental feature. Do not use it in a production environment. - -In the scenario of dual data centers, one is the primary center and the other is the DR (data recovery) center. When a Region has an odd number of replicas, more replicas are placed in the primary center. When the DR center is down for more than a specified period of time, the asynchronous mode is used by default for the replication between two centers. - -To use the synchronous mode, you can configure it in the PD configuration file or change the replication mode manually using pd-ctl. - -## Enable synchronous replication in the PD configuration file - -The replication mode is controlled by PD. You can configure it in the PD configuration file when deploying a cluster. See the following example: - -{{< copyable "" >}} - -```toml -[replication-mode] -replication-mode = "dr-auto-sync" -[replication-mode.dr-auto-sync] -label-key = "zone" -primary = "z1" -dr = "z2" -primary-replicas = 2 -dr-replicas = 1 -wait-store-timeout = "1m" -wait-sync-timeout = "1m" -``` - -In the configuration above: - -+ `dr-auto-sync` is the mode to enable synchronous replication. -+ The label key `zone` is used to distinguish different data centers. -+ TiKV instances with the `"z1"` value are considered in the primary data center, and TiKV instances with `"z2"` are in the DR data center. -+ `primary-replicas` is the number of replicas that should be placed in the primary data center. -+ `dr-replicas` is the number of replicas that should be placed in the DR data center. -+ `wait-store-timeout` is the time to wait before falling back to asynchronous replication. - -To check the current replication state of the cluster, use the following URL: - -{{< copyable "shell-regular" >}} - -```bash -% curl http://pd_ip:pd_port/pd/api/v1/replication_mode/status -``` - -```bash -{ - "mode": "dr-auto-sync", - "dr-auto-sync": { - "label-key": "zone", - "state": "sync" - } -} -``` - -> **Note:** -> -> The replication state of the cluster indicates how all Regions are replicated, with the options of `async`, `sync-recover`, and `sync`. - -After the cluster state becomes `sync`, it will not become `async` unless the number of down instances is larger than the specified number of replicas in either data center. Once the cluster state becomes `async`, PD requests TiKV to change the replication mode to `asynchronous` and checks whether TiKV instances are recovered from time to time. When the number of down instances is smaller than the number of replicas in both data centers, the cluster enters the `sync-recover` state, and then requests TiKV to change the replication mode to `synchronous`. After all Regions become `synchronous`, the cluster becomes `sync` again. - -## Change the replication mode manually - -You can use [`pd-ctl`](/pd-control.md) to change a cluster from `asynchronous` to `synchronous`. - -{{< copyable "shell-regular" >}} - -```bash ->> config set replication-mode dr-auto-sync -``` - -Or change back to `asynchronous`: - -{{< copyable "shell-regular" >}} - -```bash ->> config set replication-mode majority -``` - -You can also update the label key: - -{{< copyable "shell-regular" >}} - -```bash ->> config set replication-mode dr-auto-sync label-key dc -``` diff --git a/three-data-centers-in-two-cities-deployment.md b/three-data-centers-in-two-cities-deployment.md index 620496c7491b9..1c08734dc5bd1 100644 --- a/three-data-centers-in-two-cities-deployment.md +++ b/three-data-centers-in-two-cities-deployment.md @@ -10,11 +10,11 @@ This document introduces the architecture and configuration of the three data ce ## Overview -The architecture of three DCs in two cities is a highly available and disaster tolerant deployment solution that provides a production data center, a disaster recovery center in the same city, and a disaster recovery centers in another city. In this mode, the three DCs in two cities are interconnected. If one DC fails or suffers from a disaster, other DCs can still operate well and take over the the key applications or all applications. Compared with the the multi-DC in one city deployment, this solution has the advantage of cross-city high availability and can survive city-level natural disasters. +The architecture of three DCs in two cities is a highly available and disaster tolerant deployment solution that provides a production data center, a disaster recovery center in the same city, and a disaster recovery center in another city. In this mode, the three DCs in two cities are interconnected. If one DC fails or suffers from a disaster, other DCs can still operate well and take over the the key applications or all applications. Compared with the the multi-DC in one city deployment, this solution has the advantage of cross-city high availability and can survive city-level natural disasters. The distributed database TiDB natively supports the three-DC-in-two-city architecture by using the Raft algorithm, and guarantees the consistency and high availability of data within a database cluster. Because the network latency across DCs in the same city is relatively low, the application traffic can be dispatched to two DCs in the same city, and the traffic load can be shared by these two DCs by controlling the distribution of TiKV Region leaders and PD leaders. -## Architecture +## Deployment architecture This section takes the example of Seattle and San Francisco to explain the deployment mode of three DCs in two cities for the distributed database of TiDB. @@ -72,7 +72,7 @@ server_configs: tikv: server.grpc-compression-type: gzip pd: - replication.location-labels: ["dc","rack","zone","host"] + replication.location-labels: ["dc","zone","rack","host"] schedule.tolerant-size-ratio: 20.0 pd_servers: diff --git a/two-data-centers-in-one-city-deployment.md b/two-data-centers-in-one-city-deployment.md new file mode 100644 index 0000000000000..93b182c647cc7 --- /dev/null +++ b/two-data-centers-in-one-city-deployment.md @@ -0,0 +1,243 @@ +--- +title: Two Data Centers in One City Deployment +summary: Learn the deployment solution of two data centers in one city. +aliases: ['/tidb/dev/synchronous-replication'] +--- + +# Two Data Centers in One City Deployment + +This document introduces the deployment mode of two data centers (DCs) in one city, including the architecture, configuration, how to enable this deployment mode, and how to use replicas in this mode. + +In an on-premises environment, TiDB usually adopts the multi-data-center deployment solution to ensure high availability and disaster recovery capability. The multi-data-center deployment solution includes multiple deployment modes, such as three data centers in two cities and three data centers in one city. This document introduces the deployment mode of two data centers in one city. Deployed in this mode, TiDB can also meet the requirements of high availability and disaster recovery, with a lower cost. This deployment solution adopts Data Replication Auto Synchronous mode, or the DR Auto-Sync mode. + +Under the mode of two data centers in one city, the two data centers are less than 50 kilometers apart. They are usually located in the same city or in two adjacent cities. The network latency between the two data centers is lower than 1.5 milliseconds and the bandwidth is higher than 10 Gbps. + +## Deployment architecture + +This section takes the example of a city where two data centers IDC1 and IDC2 are located respectively in the east and west. + +The architecture of the cluster deployment is as follows: + +- The TiDB cluster is deployed to two DCs in one city: the primary IDC1 in the east, and the disaster recovery (DR) IDC2 in the west. +- The cluster has 4 replicas: 2 Voter replicas in IDC1, 1 Voter replica and 1 Learner replica in IDC2. For the TiKV component, each rack has a proper label. +- The Raft protocol is adopted to ensure consistency and high availability of data, which is transparent to users. + +![2-DC-in-1-city architecture](/media/two-dc-replication-1.png) + +This deployment solution defines three statuses to control and identify the replication status of the cluster, which restricts the replication mode of TiKV. The replication mode of the cluster can automatically and adaptively switch between the three statuses. For details, see the [Status switch](#status-switch) section. + +- **sync**: Synchronous replication mode. In this mode, at least one replica in the disaster recovery (DR) data center synchronizes with the primary data center. The Raft algorithm ensures that each log is replicated to the DR based on the label. +- **async**: Asynchronous replication mode. In this mode, the DR data center is not fully synchronized with the primary data center. The Raft algorithm follows the majority protocol to replicate logs. +- **sync-recover**: Synchronous recovery mode. In this mode, the DR data center is not fully synchronized with the primary data center. Raft gradually switches to the label replication mode and then reports the label information to PD. + +## Configuration + +### Example + +The following `tiup topology.yaml` example file is a typical topology configuration for the two data centers in one city deployment mode: + +``` +# # Global variables are applied to all deployments and used as the default value of +# # the deployments if a specific deployment value is missing. +global: + user: "tidb" + ssh_port: 22 + deploy_dir: "/data/tidb_cluster/tidb-deploy" + data_dir: "/data/tidb_cluster/tidb-data" + +server_configs: + pd: + replication.location-labels: ["zone","rack","host"] + +pd_servers: + - host: 10.63.10.10 + name: "pd-10" + - host: 10.63.10.11 + name: "pd-11" + - host: 10.63.10.12 + name: "pd-12" + + +tidb_servers: + - host: 10.63.10.10 + - host: 10.63.10.11 + - host: 10.63.10.12 + + +tikv_servers: + - host: 10.63.10.30 + config: + server.labels: { zone: "east", rack: "east-1", host: "30" } + - host: 10.63.10.31 + config: + server.labels: { zone: "east", rack: "east-2", host: "31" } + - host: 10.63.10.32 + config: + server.labels: { zone: "west", rack: "west-1", host: "32" } + - host: 10.63.10.33 + config: + server.labels: { zone: "west", rack: "west-2", host: "33" } + + +monitoring_servers: + - host: 10.63.10.60 + +grafana_servers: + - host: 10.63.10.60 + +alertmanager_servers: + - host: 10.63.10.60 +``` + +### Placement Rules + +To deploy a cluster based on the planned topology, you need to use [placement rules](/configure-placement-rules.md) to determine the locations of the cluster replicas. If 4 replicas and 2 Voter replicas are at the primary center and 1 Voter replica and 1 Learner replica are at the DR center, you can use the placement rules to configure the replicas as follows: + +``` +cat rule.json +[ + { + "group_id": "pd", + "id": "zone-east", + "start_key": "", + "end_key": "", + "role": "voter", + "count": 2, + "label_constraints": [ + { + "key": "zone", + "op": "in", + "values": [ + "east" + ] + } + ], + "location_labels": [ + "zone", + "rack", + "host", + ] + }, + { + "group_id": "pd", + "id": "zone-west", + "start_key": "", + "end_key": "", + "role": "voter", + "count": 1, + "label_constraints": [ + { + "key": "zone", + "op": "in", + "values": [ + "west" + ] + } + ], + "location_labels": [ + "zone", + "rack", + "host" + ] + }, + { + "group_id": "pd", + "id": "zone-west", + "start_key": "", + "end_key": "", + "role": "learner", + "count": 1, + "label_constraints": [ + { + "key": "zone", + "op": "in", + "values": [ + "west" + ] + } + ], + "location_labels": [ + "zone", + "rack", + "host" + ] + } +] +``` + +### Enable the DR Auto-Sync mode + +The replication mode is controlled by PD. When deploying a cluster, you can configure the replication mode in the PD configuration file. For example: + +{{< copyable "" >}} + +```toml +[replication-mode] +replication-mode = "dr-auto-sync" +[replication-mode.dr-auto-sync] +label-key = "zone" +primary = "east" +dr = "west" +primary-replicas = 2 +dr-replicas = 1 +wait-store-timeout = "1m" +wait-sync-timeout = "1m" +``` + +In the configuration above: + ++ `replication-mode` is the replication mode to be enabled. In the above example, it is set to `dr-auto-sync`. By default, the majority protocol is used. ++ `label-key` is used to distinguish different data centers and needs to match placement rules. In this example, the primary data center is "east" and the DR data center is "west". ++ `primary-replicas` is the number of Voter replicas in the primary data center. ++ `dr-replicas` is the number of Voter replicas in the DR data center. ++ `wait-store-timeout` is the waiting time for switching to asynchronous replication mode when network isolation or failure occurs. If the time of network failure exceeds the waiting time, asynchronous replication mode is enabled. The default waiting time is 60 seconds. + +To check the current replication status of the cluster, use the following API: + +{{< copyable "shell-regular" >}} + +```bash +curl http://pd_ip:pd_port/pd/api/v1/replication_mode/status +``` + +{{< copyable "shell-regular" >}} + +```bash +{ + "mode": "dr-auto-sync", + "dr-auto-sync": { + "label-key": "zone", + "state": "sync" + } +} +``` + +#### Status switch + +The replication mode of a cluster can automatically and adaptively switch between three statuses: + +- When the cluster is normal, the synchronous replication mode is enabled to maximize the data integrity of the disaster recovery data center. +- When the network connection between the two data centers fails or the DR data center breaks down, after a pre-set protective interval, the cluster enables the asynchronous replication mode to ensure the availability of the application. +- When the network reconnects or the DR data center recovers, the TiKV node joins the cluster again and gradually replicates the data. Finally, the cluster switches to the synchronous replication mode. + +The details for the status switch are as follows: + +1. **Initialization**: At the initialization stage, the cluster is in the synchronous replication mode. PD sends the status information to TiKV, and all TiKV nodes strictly follow the synchronous replication mode to work. + +2. **Switch from sync to async**: PD regularly checks the heartbeat information of TiKV to judge whether the TiKV node fails or is disconnected. If the number of failed nodes exceeds the number of replicas of the primary data center (`primary-replicas`) and the DR data center (`dr-replicas`), the synchronous replication mode can no longer serve the data replication and it is necessary to switch the status. When the failure or disconnect time exceeds the time set by `wait-store-timeout`, PD switches the status of the cluster to the async mode. Then PD sends the status of async to all TiKV nodes, and the replication mode for TiKV switches from two-center replication to the native Raft majority. + +3. **Switch from async to sync**: PD regularly checks the heartbeat information of TiKV to judge whether the TiKV node is reconnected. If the number of failed nodes is less than the number of replicas of the primary data center (`primary-replicas`) and the DR data center (`dr-replicas`), the synchronous replication mode can be enabled again. PD first switches the status of the cluster to sync-recover and sends the status information to all TiKV nodes. All Regions of TiKV gradually switch to the two-data-center synchronous replication mode and then report the heartbeat information to PD. PD records the status of TiKV Regions and calculates the recovery progress. When all TiKV Regions finish the switching, PD switches the replication mode to sync. + +### Disaster recovery + +This section introduces the disaster recovery solution of the two data centers in one city deployment. + +When a disaster occurs to a cluster in the synchronous replication mode, you can perform data recovery with `RPO = 0`: + +- If the primary data center fails and most of the Voter replicas are lost, but complete data exists in the DR data center, the lost data can be recovered from the DR data center. At this time, manual intervention is required with professional tools. You can contact the TiDB team for a recovery solution. + +- If the DR center fails and a few Voter replicas are lost, the cluster automatically switches to the asynchronous replication mode. + +When a disaster occurs to a cluster that is not in the synchronous replication mode and you cannot perform data recovery with `RPO = 0`: + +- If most of the Voter replicas are lost, manual intervention is required with professional tools. You can contact the TiDB team for a recovery solution.