From bc5684ba1291c8cb0c4d9bf98e22009af1c1ad14 Mon Sep 17 00:00:00 2001 From: JaySon-Huang Date: Thu, 23 Sep 2021 18:23:32 +0800 Subject: [PATCH 1/5] Add desc of "Storage Write Stall" row in the TiFlash grafana Signed-off-by: JaySon-Huang --- tiflash/monitor-tiflash.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tiflash/monitor-tiflash.md b/tiflash/monitor-tiflash.md index f9641145a0789..bc071f79ab7f7 100644 --- a/tiflash/monitor-tiflash.md +++ b/tiflash/monitor-tiflash.md @@ -72,6 +72,13 @@ The following sections introduce the default monitoring information of **TiFlash > > These metrics only cover the monitoring information of the TiFlash storage layer and do not cover that in TiFlash-Proxy. +## Storage Write Stall + +- Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. `throughput_write` means the throughput of data sync through Raft. `throughput_delta-management` means the throughput of data compaction. `total_write` means the total bytes written since last starts. `total_delta-management` means the total bytes of data compaction since last starts. +- Write Stall Duration: The stall duration of write and delete range. +- Write Throughput By Instance: The throughput of write by instance. It includes the throughput by applying the Raft write commands and Raft snapshots. +- Write Command OPS By Instance: The total count of different kinds of commands received by instance. `write block` means some data sync through Raft. `delete_range` means some Regions are removed from or move to this instance. `ingest` means some Region snapshot are applied to this instance. + ## Raft - Read Index OPS: The number of times that each TiFlash instance triggers the `read_index` request per second, which equals to the number of Regions triggered. From 5eb5de300e226ea4532da4743c1229ddb30ceefe Mon Sep 17 00:00:00 2001 From: JaySon-Huang Date: Thu, 23 Sep 2021 18:40:03 +0800 Subject: [PATCH 2/5] Fix some wording Signed-off-by: JaySon-Huang --- tiflash/monitor-tiflash.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tiflash/monitor-tiflash.md b/tiflash/monitor-tiflash.md index bc071f79ab7f7..720f3b8f913ad 100644 --- a/tiflash/monitor-tiflash.md +++ b/tiflash/monitor-tiflash.md @@ -75,9 +75,9 @@ The following sections introduce the default monitoring information of **TiFlash ## Storage Write Stall - Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. `throughput_write` means the throughput of data sync through Raft. `throughput_delta-management` means the throughput of data compaction. `total_write` means the total bytes written since last starts. `total_delta-management` means the total bytes of data compaction since last starts. -- Write Stall Duration: The stall duration of write and delete range. +- Write Stall Duration: The stall duration of write and removing Region data (delete range) by instance. - Write Throughput By Instance: The throughput of write by instance. It includes the throughput by applying the Raft write commands and Raft snapshots. -- Write Command OPS By Instance: The total count of different kinds of commands received by instance. `write block` means some data sync through Raft. `delete_range` means some Regions are removed from or move to this instance. `ingest` means some Region snapshot are applied to this instance. +- Write Command OPS By Instance: The total count of different kinds of commands received by instance. `write block` means the data logs synced through Raft. `delete_range` means some Regions are removed from or move to this instance. `ingest` means some Region snapshot are applied to this instance. ## Raft From 24347f241deaae1b5daf9539df25558fd3351453 Mon Sep 17 00:00:00 2001 From: JaySon-Huang Date: Fri, 24 Sep 2021 16:57:32 +0800 Subject: [PATCH 3/5] Align with the Chinese version Signed-off-by: JaySon-Huang --- tiflash/monitor-tiflash.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/tiflash/monitor-tiflash.md b/tiflash/monitor-tiflash.md index 720f3b8f913ad..e4fc236d9607f 100644 --- a/tiflash/monitor-tiflash.md +++ b/tiflash/monitor-tiflash.md @@ -74,10 +74,17 @@ The following sections introduce the default monitoring information of **TiFlash ## Storage Write Stall -- Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. `throughput_write` means the throughput of data sync through Raft. `throughput_delta-management` means the throughput of data compaction. `total_write` means the total bytes written since last starts. `total_delta-management` means the total bytes of data compaction since last starts. +- Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. + - `throughput_write` means the throughput of data sync through Raft. + - `throughput_delta-management` means the throughput of data compaction. + - `total_write` means the total bytes written since last starts. + - `total_delta-management` means the total bytes of data compaction since last starts. - Write Stall Duration: The stall duration of write and removing Region data (delete range) by instance. - Write Throughput By Instance: The throughput of write by instance. It includes the throughput by applying the Raft write commands and Raft snapshots. -- Write Command OPS By Instance: The total count of different kinds of commands received by instance. `write block` means the data logs synced through Raft. `delete_range` means some Regions are removed from or move to this instance. `ingest` means some Region snapshot are applied to this instance. +- Write Command OPS By Instance: The total count of different kinds of commands received by instance. + - `write block` means the data logs synced through Raft. + - `delete_range` means some Regions are removed from or move to this instance. + - `ingest` means some Region snapshot are applied to this instance. ## Raft From c83620435030c4617b5dc404fddd486c5d63400c Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Sun, 26 Sep 2021 15:52:52 +0800 Subject: [PATCH 4/5] Apply suggestions from code review --- tiflash/monitor-tiflash.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/tiflash/monitor-tiflash.md b/tiflash/monitor-tiflash.md index e4fc236d9607f..58905e40a6967 100644 --- a/tiflash/monitor-tiflash.md +++ b/tiflash/monitor-tiflash.md @@ -75,16 +75,16 @@ The following sections introduce the default monitoring information of **TiFlash ## Storage Write Stall - Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. - - `throughput_write` means the throughput of data sync through Raft. + - `throughput_write` means the throughput of data synchronization through Raft. - `throughput_delta-management` means the throughput of data compaction. - - `total_write` means the total bytes written since last starts. - - `total_delta-management` means the total bytes of data compaction since last starts. + - `total_write` means the total bytes written since the last start. + - `total_delta-management` means the total bytes of data compaction since the last start. - Write Stall Duration: The stall duration of write and removing Region data (delete range) by instance. - Write Throughput By Instance: The throughput of write by instance. It includes the throughput by applying the Raft write commands and Raft snapshots. - Write Command OPS By Instance: The total count of different kinds of commands received by instance. - - `write block` means the data logs synced through Raft. - - `delete_range` means some Regions are removed from or move to this instance. - - `ingest` means some Region snapshot are applied to this instance. + - `write block` means the data logs synchronized through Raft. + - `delete_range` means some Regions are removed from or moved to this instance. + - `ingest` means some Region snapshots are applied to this instance. ## Raft From 3a8a5b324501ffe8eb15510e3e766e7b66ed7d46 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Sun, 26 Sep 2021 15:56:48 +0800 Subject: [PATCH 5/5] Update monitor-tiflash.md --- tiflash/monitor-tiflash.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tiflash/monitor-tiflash.md b/tiflash/monitor-tiflash.md index 58905e40a6967..10c3892325886 100644 --- a/tiflash/monitor-tiflash.md +++ b/tiflash/monitor-tiflash.md @@ -74,16 +74,16 @@ The following sections introduce the default monitoring information of **TiFlash ## Storage Write Stall -- Write & Delta Management Throughput: The throughput of write and delta's background management for all instances. +- Write & Delta Management Throughput: The throughput of write and data compaction for all instances. - `throughput_write` means the throughput of data synchronization through Raft. - `throughput_delta-management` means the throughput of data compaction. - `total_write` means the total bytes written since the last start. - - `total_delta-management` means the total bytes of data compaction since the last start. -- Write Stall Duration: The stall duration of write and removing Region data (delete range) by instance. + - `total_delta-management` means the total bytes of data compacted since the last start. +- Write Stall Duration: The stall duration of write and removing Region data (deleting ranges) by instance. - Write Throughput By Instance: The throughput of write by instance. It includes the throughput by applying the Raft write commands and Raft snapshots. - Write Command OPS By Instance: The total count of different kinds of commands received by instance. - `write block` means the data logs synchronized through Raft. - - `delete_range` means some Regions are removed from or moved to this instance. + - `delete_range` means that some Regions are removed from or moved to this instance. - `ingest` means some Region snapshots are applied to this instance. ## Raft