Add queue settings to docs (elastic#4884)

* Move queue setting to TOC + address some review * Remove flush_interval and spooler from outputconfig * remove 'slot' * add [float] tags * Update queueconfig.asciidoc
ruflin · Oct 9, 2017 · 54a4481 · 54a4481
1 parent f9ac23f
commit 54a4481
Show file tree

Hide file tree

Showing 8 changed files with 105 additions and 14 deletions.
diff --git a/auditbeat/docs/configuring-howto.asciidoc b/auditbeat/docs/configuring-howto.asciidoc
@@ -22,6 +22,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-{beatname_lc}>>
 * <<configuration-general-options>>
 * <<{beatname_lc}-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -46,6 +47,8 @@ include::./auditbeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

diff --git a/filebeat/docs/configuring-howto.asciidoc b/filebeat/docs/configuring-howto.asciidoc
@@ -24,6 +24,7 @@ The following topics describe how to configure Filebeat:
 * <<multiline-examples>>
 * <<configuration-general-options>>
 * <<filebeat-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -53,6 +54,8 @@ include::./filebeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

diff --git a/heartbeat/docs/configuring-howto.asciidoc b/heartbeat/docs/configuring-howto.asciidoc
@@ -23,6 +23,7 @@ The following topics describe how to configure Heartbeat:
 
 * <<configuration-heartbeat-options>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -43,6 +44,8 @@ include::./heartbeat-options.asciidoc[]
 include::./heartbeat-general-options.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc
@@ -281,10 +281,9 @@ However big batch sizes can also increase processing times, which might result i
 API errors, killed connections, timed-out publishing requests, and, ultimately, lower
 throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
-Beats that publish single events (such as Packetbeat) send each event directly to
-Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
-spooler size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 ===== `timeout`
 
@@ -526,10 +525,10 @@ However big batch sizes can also increase processing times, which might result i
 API errors, killed connections, timed-out publishing requests, and, ultimately, lower
 throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
-Beats that publish single events (such as Packetbeat) send each event directly to
-Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
-spooler size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
+
 
 ===== `slow_start`
 
@@ -930,11 +929,9 @@ of sending events. However big batch sizes can also increase processing times,
 which might result in API errors, killed connections, timed-out publishing
 requests, and, ultimately, lower throughput.
 
-Setting `bulk_max_size` to values less than or equal to 0 disables buffering in
-libbeat. When buffering is disabled, Beats that publish single events (such as
-Packetbeat) send each event directly to Redis. Beats that publish
-data in batches (such as Filebeat) send events in batches based on the spooler
-size.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 ===== `ssl`
 
@@ -1064,7 +1061,9 @@ The maximum number of events to buffer internally during publishing. The default
 Specifying a larger batch size may add some latency and buffering during publishing. However, for Console output, this
 setting does not affect how events are published.
 
-Setting `bulk_max_size` to 0 disables buffering in libbeat.
+Setting `bulk_max_size` to values less than or equal to 0 disables the
+splitting of batches. When splitting is disabled, the queue decides on the
+number of events to be contained in a batch.
 
 [[configuration-output-codec]]
 === Configure the output codec

diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc
@@ -0,0 +1,74 @@
+[[configuring-internal-queue]]
+== Configure the internal queue
+
+{beatname_uc} uses an internal queue to store events before publishing them. The
+queue is responsible for buffering and combining events into batches that can
+be consumed by the outputs. The outputs will use bulk operations to send a
+batch of events in one transaction.
+
+You can configure the type and behavior of the internal queue by setting options in the `queue` section of the +{beatname_lc}.yml+ config file.
+
+
+Example configuration:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+------------------------------------------------------------------------------
+
+[float]
+[[configuration-internal-queue-memory]]
+=== Configure the memory qeueue
+
+The memory queue keeps all events in memory. It is the only queue type
+supported right now.  By default no flush interval is configured. All events
+published to this queue will be directly consumed by the outputs.
+The output's `bulk_max_size` setting limits the number of events being processed at once.
+
+The memory queue waits for the output to acknowledge or drop events. If
+the queue is full, no new events can be inserted into the memeory queue. Only
+after the signal from the output will the queue free up space for more events to be accepted.
+
+To enforce spooling in the queue, set the `flush.min_events` and `flush.timeout` options.
+
+This sample configuration forwards events to the output if 512 events are
+available or the oldest available event is already waiting for 5s in the queue:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+  flush.min_events: 512
+  flush.timeout: 5s
+------------------------------------------------------------------------------
+
+[float]
+==== Configuration options
+
+You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file:
+
+[float]
+===== `events`
+
+Number of events the queue can store. 
+
+The default value is 4096 events.
+
+[float]
+===== `flush.min_events`
+
+Minimum number of events required for publishing. If this value is set to 0, the
+output can start publishing events without additional waiting times. Otherwise
+the output has to wait for more events to become available.
+
+The default value is 0.
+
+[float]
+===== `flush.timeout`
+
+Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events
+will be immediately available for consumption.
+
+The default values is 0s.
+
diff --git a/metricbeat/docs/configuring-howto.asciidoc b/metricbeat/docs/configuring-howto.asciidoc
@@ -22,6 +22,7 @@ The following topics describe how to configure Metricbeat:
 * <<configuration-metricbeat>>
 * <<configuration-general-options>>
 * <<metricbeat-configuration-reloading>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -44,6 +45,8 @@ include::./metricbeat-general-options.asciidoc[]
 include::./reload-configuration.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

diff --git a/packetbeat/docs/configuring-howto.asciidoc b/packetbeat/docs/configuring-howto.asciidoc
@@ -24,6 +24,7 @@ The following topics describe how to configure Packetbeat:
 * <<configuration-protocols>>
 * <<configuration-processes>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -44,6 +45,8 @@ include::./packetbeat-options.asciidoc[]
 include::./packetbeat-general-options.asciidoc[]
 
 :allplatforms:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]

diff --git a/winlogbeat/docs/configuring-howto.asciidoc b/winlogbeat/docs/configuring-howto.asciidoc
@@ -19,6 +19,7 @@ The following topics describe how to configure Winlogbeat:
 
 * <<configuration-winlogbeat-options>>
 * <<configuration-general-options>>
+* <<configuring-internal-queue>>
 * <<configuring-output>>
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
@@ -38,6 +39,8 @@ include::./winlogbeat-options.asciidoc[]
 include::./winlogbeat-general-options.asciidoc[]
 
 :win:
+include::../../libbeat/docs/queueconfig.asciidoc[]
+
 include::../../libbeat/docs/outputconfig.asciidoc[]
 
 include::../../libbeat/docs/shared-ssl-config.asciidoc[]