From 22491693820ba2c76b2bb1bdfc78628cd134f6eb Mon Sep 17 00:00:00 2001 From: Andreas Gkizas Date: Tue, 31 Oct 2023 14:52:02 +0200 Subject: [PATCH 1/8] Adding docs for leader election timings --- ...ubernetes_leaderelection-provider.asciidoc | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index c82619c8a..71b72bce7 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -18,6 +18,9 @@ providers.kubernetes_leaderelection: # qps: 5 # burst: 10 #leader_lease: agent-k8s-leader-lock + #leader_retryperiod: 2 + #leader_leaseduration: 15 + #leader_renewdeadline: 10 ---- `enabled`:: (Optional) Defaults to true. To explicitly disable the LeaderElection provider, @@ -30,6 +33,9 @@ Supported options are `qps` and `burst`. If not set, the Kubernetes client's default QPS and burst settings are used. `leader_lease`:: (Optional) Specify the name of the leader lease. This is set to `elastic-agent-cluster-leader` by default. +`leader_retryperiod`:: (Optional) Default value 2 (in sec). How long before {agent}s try to get leader role. +`leader_leaseduration`:: (Optional) Default value 15 (in sec). How long the Leader {agent} to hold the "leader" state +`leader_renewdeadline`:: (Optional) Default value 10 (in sec). How long for leaders to retry getting "leader" role The available key is: @@ -42,6 +48,24 @@ The available key is: |=== + +[discrete] += Understanding leader timings + +As described above, the LeaderElection configuration offers following parameters: Lease duration (`leader_leaseduration`), Renew deadline (`leader_renewdeadline`) and +Retry period (`leader_retryperiod`) . Based on the config provided, each agent will trigger {k8s} API requests and will try to check the status of the lease. + +NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that less API requests will be made towards {k8s} Control API, but will also increase the period where collection of metrics from Leader {agent} might be lost. + +The library applies https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L76[specific checks] for the timing parameters and if those are not verified the {agent} will exit with `panic` error. + +In general: +- Leaseduration must be greater than renewdeadline +- Renewdeadline must be greater than retryperiod*JitterFactor. + +NOTE: Constant JitterFactor=1.2 defined in https://pkg.go.dev/gopkg.in/kubernetes/client-go.v11/tools/leaderelection[leaderelection lib] + + [discrete] = Enabling configurations only when on leadership @@ -62,3 +86,5 @@ metricset only when the leadership lock is acquired: period: 10s condition: ${kubernetes_leaderelection.leader} == true ---- + + From 397067ad88e53468582a6f3ccefae12dee2c9d6a Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Tue, 31 Oct 2023 17:05:25 +0200 Subject: [PATCH 2/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- .../providers/kubernetes_leaderelection-provider.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index 71b72bce7..3c640946a 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -33,9 +33,9 @@ Supported options are `qps` and `burst`. If not set, the Kubernetes client's default QPS and burst settings are used. `leader_lease`:: (Optional) Specify the name of the leader lease. This is set to `elastic-agent-cluster-leader` by default. -`leader_retryperiod`:: (Optional) Default value 2 (in sec). How long before {agent}s try to get leader role. -`leader_leaseduration`:: (Optional) Default value 15 (in sec). How long the Leader {agent} to hold the "leader" state -`leader_renewdeadline`:: (Optional) Default value 10 (in sec). How long for leaders to retry getting "leader" role +`leader_retryperiod`:: (Optional) Default value 2 (in sec). How long before {agent}s try to get the `leader` role. +`leader_leaseduration`:: (Optional) Default value 15 (in sec). How long the Leader {agent} holds the `leader` state. +`leader_renewdeadline`:: (Optional) Default value 10 (in sec). How long leaders retry getting the `leader` role. The available key is: From 943f04ac888ca0567458478a4da08412fca4179e Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Tue, 31 Oct 2023 17:05:33 +0200 Subject: [PATCH 3/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- .../providers/kubernetes_leaderelection-provider.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index 3c640946a..a56ab5fc6 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -52,8 +52,8 @@ The available key is: [discrete] = Understanding leader timings -As described above, the LeaderElection configuration offers following parameters: Lease duration (`leader_leaseduration`), Renew deadline (`leader_renewdeadline`) and -Retry period (`leader_retryperiod`) . Based on the config provided, each agent will trigger {k8s} API requests and will try to check the status of the lease. +As described above, the LeaderElection configuration offers the following parameters: Lease duration (`leader_leaseduration`), Renew deadline (`leader_renewdeadline`), and +Retry period (`leader_retryperiod`). Based on the config provided, each agent will trigger {k8s} API requests and will try to check the status of the lease. NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that less API requests will be made towards {k8s} Control API, but will also increase the period where collection of metrics from Leader {agent} might be lost. From 601a80a506b09ada20fcbb74e2786f8e72877dc3 Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Tue, 31 Oct 2023 17:05:52 +0200 Subject: [PATCH 4/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- .../providers/kubernetes_leaderelection-provider.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index a56ab5fc6..4c7cfa35d 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -55,7 +55,7 @@ The available key is: As described above, the LeaderElection configuration offers the following parameters: Lease duration (`leader_leaseduration`), Renew deadline (`leader_renewdeadline`), and Retry period (`leader_retryperiod`). Based on the config provided, each agent will trigger {k8s} API requests and will try to check the status of the lease. -NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that less API requests will be made towards {k8s} Control API, but will also increase the period where collection of metrics from Leader {agent} might be lost. +NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that fewer API requests will be made towards the {k8s} Control API, but will also increase the period where collection of metrics from the leader {agent} might be lost. The library applies https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L76[specific checks] for the timing parameters and if those are not verified the {agent} will exit with `panic` error. From cc01e0c1f59c22710b871b0ee2d71ce58756a743 Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Tue, 31 Oct 2023 17:06:00 +0200 Subject: [PATCH 5/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- .../providers/kubernetes_leaderelection-provider.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index 4c7cfa35d..c36157a8e 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -57,7 +57,7 @@ Retry period (`leader_retryperiod`). Based on the config provided, each agent wi NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that fewer API requests will be made towards the {k8s} Control API, but will also increase the period where collection of metrics from the leader {agent} might be lost. -The library applies https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L76[specific checks] for the timing parameters and if those are not verified the {agent} will exit with `panic` error. +The library applies https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L76[specific checks] for the timing parameters and if those are not verified {agent} will exit with a `panic` error. In general: - Leaseduration must be greater than renewdeadline From 9be81c6c7be726b6e42b2ddcc956b2283b228cee Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Tue, 31 Oct 2023 17:06:14 +0200 Subject: [PATCH 6/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- .../providers/kubernetes_leaderelection-provider.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index c36157a8e..a313ada2b 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -63,7 +63,7 @@ In general: - Leaseduration must be greater than renewdeadline - Renewdeadline must be greater than retryperiod*JitterFactor. -NOTE: Constant JitterFactor=1.2 defined in https://pkg.go.dev/gopkg.in/kubernetes/client-go.v11/tools/leaderelection[leaderelection lib] +NOTE: Constant JitterFactor=1.2 is defined in https://pkg.go.dev/gopkg.in/kubernetes/client-go.v11/tools/leaderelection[leaderelection lib]. [discrete] From 2f9e997993149425bdbbe67cafc4c55a1c12289d Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Fri, 3 Nov 2023 11:35:41 +0200 Subject: [PATCH 7/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: Michael Katsoulis --- .../providers/kubernetes_leaderelection-provider.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index a313ada2b..8fe637663 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -34,7 +34,7 @@ default QPS and burst settings are used. `leader_lease`:: (Optional) Specify the name of the leader lease. This is set to `elastic-agent-cluster-leader` by default. `leader_retryperiod`:: (Optional) Default value 2 (in sec). How long before {agent}s try to get the `leader` role. -`leader_leaseduration`:: (Optional) Default value 15 (in sec). How long the Leader {agent} holds the `leader` state. +`leader_leaseduration`:: (Optional) Default value 15 (in sec). How long the leader {agent} holds the `leader` state. `leader_renewdeadline`:: (Optional) Default value 10 (in sec). How long leaders retry getting the `leader` role. The available key is: From f233d360c8d01f92d073e838725874e20b3accc3 Mon Sep 17 00:00:00 2001 From: Andrew Gizas Date: Fri, 3 Nov 2023 12:06:03 +0200 Subject: [PATCH 8/8] Update docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc Co-authored-by: Michael Katsoulis --- .../providers/kubernetes_leaderelection-provider.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc index 8fe637663..ec89d4ef9 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/providers/kubernetes_leaderelection-provider.asciidoc @@ -55,7 +55,7 @@ The available key is: As described above, the LeaderElection configuration offers the following parameters: Lease duration (`leader_leaseduration`), Renew deadline (`leader_renewdeadline`), and Retry period (`leader_retryperiod`). Based on the config provided, each agent will trigger {k8s} API requests and will try to check the status of the lease. -NOTE: The number of Leader API calls is proportional to the number of {agent}s installed. This means that Leader API requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that fewer API requests will be made towards the {k8s} Control API, but will also increase the period where collection of metrics from the leader {agent} might be lost. +NOTE: The number of leader calls to the K8s Control API is proportional to the number of {agent}s installed. This means that requests will come from all {agent}s per `leader_retryperiod`. Setting `leader_retryperiod` to a greater value than the default (2sec), means that fewer requests will be made towards the {k8s} Control API, but will also increase the period where collection of metrics from the leader {agent} might be lost. The library applies https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L76[specific checks] for the timing parameters and if those are not verified {agent} will exit with a `panic` error.