From 54a44810d605569b648e8850c63fe8261e5d10f3 Mon Sep 17 00:00:00 2001 From: Steffen Siering Date: Mon, 9 Oct 2017 21:09:10 +0200 Subject: [PATCH] Add queue settings to docs (#4884) * Move queue setting to TOC + address some review * Remove flush_interval and spooler from outputconfig * remove 'slot' * add [float] tags * Update queueconfig.asciidoc --- auditbeat/docs/configuring-howto.asciidoc | 3 + filebeat/docs/configuring-howto.asciidoc | 3 + heartbeat/docs/configuring-howto.asciidoc | 3 + libbeat/docs/outputconfig.asciidoc | 27 ++++---- libbeat/docs/queueconfig.asciidoc | 74 ++++++++++++++++++++++ metricbeat/docs/configuring-howto.asciidoc | 3 + packetbeat/docs/configuring-howto.asciidoc | 3 + winlogbeat/docs/configuring-howto.asciidoc | 3 + 8 files changed, 105 insertions(+), 14 deletions(-) create mode 100644 libbeat/docs/queueconfig.asciidoc diff --git a/auditbeat/docs/configuring-howto.asciidoc b/auditbeat/docs/configuring-howto.asciidoc index 54a63e86ce4..e16447de3cf 100644 --- a/auditbeat/docs/configuring-howto.asciidoc +++ b/auditbeat/docs/configuring-howto.asciidoc @@ -22,6 +22,7 @@ The following topics describe how to configure {beatname_uc}: * <> * <> * <<{beatname_lc}-configuration-reloading>> +* <> * <> * <> * <> @@ -46,6 +47,8 @@ include::./auditbeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc index 3a2d4d2f4a3..4f6e53fb691 100644 --- a/filebeat/docs/configuring-howto.asciidoc +++ b/filebeat/docs/configuring-howto.asciidoc @@ -24,6 +24,7 @@ The following topics describe how to configure Filebeat: * <> * <> * <> +* <> * <> * <> * <> @@ -53,6 +54,8 @@ include::./filebeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/heartbeat/docs/configuring-howto.asciidoc b/heartbeat/docs/configuring-howto.asciidoc index a46909a50f9..3774c62e983 100644 --- a/heartbeat/docs/configuring-howto.asciidoc +++ b/heartbeat/docs/configuring-howto.asciidoc @@ -23,6 +23,7 @@ The following topics describe how to configure Heartbeat: * <> * <> +* <> * <> * <> * <> @@ -43,6 +44,8 @@ include::./heartbeat-options.asciidoc[] include::./heartbeat-general-options.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index cd81954c5eb..06f42b2c12d 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -281,10 +281,9 @@ However big batch sizes can also increase processing times, which might result i API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled, -Beats that publish single events (such as Packetbeat) send each event directly to -Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the -spooler size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. ===== `timeout` @@ -526,10 +525,10 @@ However big batch sizes can also increase processing times, which might result i API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled, -Beats that publish single events (such as Packetbeat) send each event directly to -Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the -spooler size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. + ===== `slow_start` @@ -930,11 +929,9 @@ of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. -Setting `bulk_max_size` to values less than or equal to 0 disables buffering in -libbeat. When buffering is disabled, Beats that publish single events (such as -Packetbeat) send each event directly to Redis. Beats that publish -data in batches (such as Filebeat) send events in batches based on the spooler -size. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. ===== `ssl` @@ -1064,7 +1061,9 @@ The maximum number of events to buffer internally during publishing. The default Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this setting does not affect how events are published. -Setting `bulk_max_size` to 0 disables buffering in libbeat. +Setting `bulk_max_size` to values less than or equal to 0 disables the +splitting of batches. When splitting is disabled, the queue decides on the +number of events to be contained in a batch. [[configuration-output-codec]] === Configure the output codec diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc new file mode 100644 index 00000000000..3c4f77f4c74 --- /dev/null +++ b/libbeat/docs/queueconfig.asciidoc @@ -0,0 +1,74 @@ +[[configuring-internal-queue]] +== Configure the internal queue + +{beatname_uc} uses an internal queue to store events before publishing them. The +queue is responsible for buffering and combining events into batches that can +be consumed by the outputs. The outputs will use bulk operations to send a +batch of events in one transaction. + +You can configure the type and behavior of the internal queue by setting options in the `queue` section of the +{beatname_lc}.yml+ config file. + + +Example configuration: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 +------------------------------------------------------------------------------ + +[float] +[[configuration-internal-queue-memory]] +=== Configure the memory qeueue + +The memory queue keeps all events in memory. It is the only queue type +supported right now. By default no flush interval is configured. All events +published to this queue will be directly consumed by the outputs. +The output's `bulk_max_size` setting limits the number of events being processed at once. + +The memory queue waits for the output to acknowledge or drop events. If +the queue is full, no new events can be inserted into the memeory queue. Only +after the signal from the output will the queue free up space for more events to be accepted. + +To enforce spooling in the queue, set the `flush.min_events` and `flush.timeout` options. + +This sample configuration forwards events to the output if 512 events are +available or the oldest available event is already waiting for 5s in the queue: + +[source,yaml] +------------------------------------------------------------------------------ +queue.mem: + events: 4096 + flush.min_events: 512 + flush.timeout: 5s +------------------------------------------------------------------------------ + +[float] +==== Configuration options + +You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file: + +[float] +===== `events` + +Number of events the queue can store. + +The default value is 4096 events. + +[float] +===== `flush.min_events` + +Minimum number of events required for publishing. If this value is set to 0, the +output can start publishing events without additional waiting times. Otherwise +the output has to wait for more events to become available. + +The default value is 0. + +[float] +===== `flush.timeout` + +Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events +will be immediately available for consumption. + +The default values is 0s. + diff --git a/metricbeat/docs/configuring-howto.asciidoc b/metricbeat/docs/configuring-howto.asciidoc index 27fdf740058..2df113c9763 100644 --- a/metricbeat/docs/configuring-howto.asciidoc +++ b/metricbeat/docs/configuring-howto.asciidoc @@ -22,6 +22,7 @@ The following topics describe how to configure Metricbeat: * <> * <> * <> +* <> * <> * <> * <> @@ -44,6 +45,8 @@ include::./metricbeat-general-options.asciidoc[] include::./reload-configuration.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/packetbeat/docs/configuring-howto.asciidoc b/packetbeat/docs/configuring-howto.asciidoc index bab608cf2c2..018480bf84d 100644 --- a/packetbeat/docs/configuring-howto.asciidoc +++ b/packetbeat/docs/configuring-howto.asciidoc @@ -24,6 +24,7 @@ The following topics describe how to configure Packetbeat: * <> * <> * <> +* <> * <> * <> * <> @@ -44,6 +45,8 @@ include::./packetbeat-options.asciidoc[] include::./packetbeat-general-options.asciidoc[] :allplatforms: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[] diff --git a/winlogbeat/docs/configuring-howto.asciidoc b/winlogbeat/docs/configuring-howto.asciidoc index 8a6fa05ebce..dd1e94d2748 100644 --- a/winlogbeat/docs/configuring-howto.asciidoc +++ b/winlogbeat/docs/configuring-howto.asciidoc @@ -19,6 +19,7 @@ The following topics describe how to configure Winlogbeat: * <> * <> +* <> * <> * <> * <> @@ -38,6 +39,8 @@ include::./winlogbeat-options.asciidoc[] include::./winlogbeat-general-options.asciidoc[] :win: +include::../../libbeat/docs/queueconfig.asciidoc[] + include::../../libbeat/docs/outputconfig.asciidoc[] include::../../libbeat/docs/shared-ssl-config.asciidoc[]