Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Update documentation copied from beats #2131

Merged
merged 2 commits into from
Apr 23, 2019
Merged

[docs] Update documentation copied from beats #2131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3bb3480
docs: update beats documenation
bmorelli25 Apr 22, 2019
92a5fab
update yml files with new monitoring
bmorelli25 Apr 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _beats/script/build_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ do
params="$params --resource=${resource_dir}"
fi

$docs_dir/build_docs.pl $params --doc "$index" -out "$dest_dir"
$docs_dir/build_docs $params --doc "$index" --out "$dest_dir"
done
17 changes: 9 additions & 8 deletions _meta/beat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -742,14 +742,15 @@ output.elasticsearch:
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:
#monitoring.enabled: false

# Most settings from the Elasticsearch output are accepted here as well.
# Note that these settings should be configured to point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration. This means that if you have the Elasticsearch output configured,
# you can simply uncomment the following line.
#monitoring.elasticsearch:

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "apm_system"
Expand Down
17 changes: 9 additions & 8 deletions apm-server.docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -742,14 +742,15 @@ output.elasticsearch:
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:
#monitoring.enabled: false

# Most settings from the Elasticsearch output are accepted here as well.
# Note that these settings should be configured to point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration. This means that if you have the Elasticsearch output configured,
# you can simply uncomment the following line.
#monitoring.elasticsearch:

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "apm_system"
Expand Down
17 changes: 9 additions & 8 deletions apm-server.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -742,14 +742,15 @@ output.elasticsearch:
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:
#monitoring.enabled: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late response here. With the latest changes from libbeat, both options - xpack.monitoring and monitoring are supported, but xpack.monitoring will be deprecated from 7.1.0 on. Afaik we usually still document deprecated settings, so I suggest to document them both, but clearly mark the deprecated setting.
Also, when describing the new setting, could you please ensure to fully remove xpack also from the header and the description.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do typically document deprecated settings, but I chose to follow Beats lead by removing it from yml files and docs: elastic/beats/pull/11677+ elastic/beats/pull/11678. We can document the deprecated settings, but it will mean adding more complexities to the docs as this is beats documentation - which isn't necessarily a bad thing - just want you to be aware.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm asking to see if there's any rhyme or reason as to why xpack was left in the yml files, or if that was an oversight.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking @bmorelli25.

Up to you if you want to leave the yml+docs as they are, just wanted to ensure it wasn't removed by mistake and is streamlined with documenting other/future deprecated features and settings.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool beans - I appreciate you bringing this up! I'm very curious to hear the thought process for why beats did this the way they did. Based on their answers, I'll figure out the path forward and determine what I need to or don't need to do.


# Most settings from the Elasticsearch output are accepted here as well.
# Note that these settings should be configured to point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration. This means that if you have the Elasticsearch output configured,
# you can simply uncomment the following line.
#monitoring.elasticsearch:

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "apm_system"
Expand Down
11 changes: 11 additions & 0 deletions docs/copied-from-beats/command-reference.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,12 +658,23 @@ modules to load pipelines for.
endif::[]

ifeval::["{beatname_lc}"!="filebeat"]

ifndef::no_dashboards[]
["source","sh",subs="attributes"]
-----
{beatname_lc} setup --dashboards
{beatname_lc} setup --machine-learning
{beatname_lc} setup --template
-----
endif::no_dashboards[]
ifdef::no_dashboards[]
["source","sh",subs="attributes"]
-----
{beatname_lc} setup --machine-learning
{beatname_lc} setup --template
-----
endif::no_dashboards[]

endif::[]

[[test-command]]
Expand Down
33 changes: 16 additions & 17 deletions docs/copied-from-beats/monitoring/monitoring-beats.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@
[partintro]
--

NOTE: {monitoring} for {beatname_uc} requires {es} {beat_monitoring_version} or later.
NOTE: The {monitor-features} for {beatname_uc} require {es} {beat_monitoring_version} or later.

{monitoring} enables you to easily monitor {beatname_uc} from {kib}. For more
The {stack} {monitor-features} enable you to easily monitor {beatname_uc} from {kib}. For more
information, see
{xpack-ref}/xpack-monitoring.html[Monitoring the Elastic Stack] and
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}] and
{kibana-ref}/beats-page.html[Beats Monitoring Metrics].

To configure {beatname_uc} to collect and send monitoring metrics:
Expand All @@ -30,35 +30,34 @@ To configure {beatname_uc} to collect and send monitoring metrics:
data to {es}. For example, you can use the built-in +{beat_monitoring_user}+ user or
assign the built-in +{beat_monitoring_user}+ role to another user. For more
information, see
{xpack-ref}/setting-up-authentication.html[Setting Up User Authentication] and
{xpack-ref}/built-in-roles.html[Built-in Roles].
{stack-ov}/setting-up-authentication.html[Setting Up User Authentication] and
{stack-ov}/built-in-roles.html[Built-in Roles].

. Add the `xpack.monitoring` settings in the {beatname_uc} configuration file. If you
configured {es} output, specify the following minimal configuration:
. Add the `monitoring` settings in the {beatname_uc} configuration file. If you
configured the {es} output and want to send {beatname_uc} monitoring events to
the same {es} cluster, specify the following minimal configuration:
+
--
[source, yml]
--------------------
xpack.monitoring.enabled: true
monitoring.enabled: true
--------------------

If you configured a different output, such as {ls}, you must specify additional
configuration options. For example:
If you configured a different output, such as {ls} or you want to send {beatname_uc}
monitoring events to a separate {es} cluster (referred to as the _monitoring cluster_),
you must specify additional configuration options. For example:

["source","yml",subs="attributes"]
--------------------
xpack.monitoring:
monitoring:
enabled: true
elasticsearch:
hosts: ["https://example.com:9200", "https://example2.com:9200"]
hosts: ["https://example.com:9200", "https://example2.com:9200"] <1>
username: {beat_monitoring_user}
password: somepassword
--------------------

NOTE: Currently you must send monitoring data to the same cluster as all other events.
If you configured {es} output, do not specify additional hosts in the monitoring
configuration.

<1> This setting identifies the hosts and port numbers of {es} nodes
that are part of the monitoring cluster.
--

. {kibana-ref}/monitoring-xpack-kibana.html[Configure monitoring in {kib}].
Expand Down
30 changes: 17 additions & 13 deletions docs/copied-from-beats/outputconfig.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ output.elasticsearch:

If the Elasticsearch nodes are defined by `IP:PORT`, then add `protocol: https` to the yaml file.

["source","yaml",subs="attributes,callouts"]
[source,yaml]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["localhost"]
Expand Down Expand Up @@ -249,7 +249,7 @@ endif::no_dashboards[]

You can set the index dynamically by using a format string to access any event
field. For example, this configuration uses a custom field, `fields.log_type`,
to set the index:
to set the index:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
Expand All @@ -261,13 +261,13 @@ output.elasticsearch:
<1> We recommend including +{beat_version_key}+ in the name to avoid mapping issues
when you upgrade.

With this configuration, all events with `log_type: normal` are sent to an
With this configuration, all events with `log_type: normal` are sent to an
index named +normal-{version}-{localdate}+, and all events with
`log_type: critical` are sent to an index named
+critical-{version}-{localdate}+.

TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
<<libbeat-configuration-fields,`fields`>> option.

See the <<indices-option-es,`indices`>> setting for other ways to set the index
dynamically.
Expand All @@ -285,7 +285,7 @@ matches, the <<index-option-es,`index`>> setting is used.
Rule settings:

*`index`*:: The index format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule fails.
references, such as `%{[fields.name]}`, the fields must exist, or the rule fails.

*`mappings`*:: A dictionary that takes the value returned by `index` and maps it
to a new name.
Expand Down Expand Up @@ -347,7 +347,7 @@ ifndef::no_ilm[]
[[ilm-es]]
===== `ilm`

Configuration options for index lifecycle management.
Configuration options for index lifecycle management.

See <<ilm>> for more information.
endif::no_ilm[]
Expand All @@ -369,7 +369,7 @@ For more information, see <<configuring-ingest-node>>.

You can set the ingest node pipeline dynamically by using a format string to
access any event field. For example, this configuration uses a custom field,
`fields.log_type`, to set the pipeline for each event:
`fields.log_type`, to set the pipeline for each event:

["source","yaml",subs="attributes"]
------------------------------------------------------------------------------
Expand All @@ -384,7 +384,7 @@ named `normal_pipeline`, and all events with `log_type: critical` are sent to a
pipeline named `critical_pipeline`.

TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
<<libbeat-configuration-fields,`fields`>> option.

See the <<pipelines-option-es,`pipelines`>> setting for other ways to set the
ingest node pipeline dynamically.
Expand All @@ -403,7 +403,7 @@ Rule settings:

*`pipeline`*:: The pipeline format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.
fails.

*`mappings`*:: A dictionary that takes the value returned by `pipeline` and maps
it to a new name.
Expand Down Expand Up @@ -870,7 +870,7 @@ topic: '%{[fields.log_topic]}'
-----

TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
<<libbeat-configuration-fields,`fields`>> option.

See the <<topics-option-kafka,`topics`>> setting for other ways to set the
topic dynamically.
Expand All @@ -889,7 +889,7 @@ Rule settings:

*`topic`*:: The topic format string to use. If this string contains field
references, such as `%{[fields.name]}`, the fields must exist, or the rule
fails.
fails.

*`mappings`*:: A dictionary that takes the value returned by `topic` and maps it
to a new name.
Expand All @@ -901,7 +901,7 @@ match.
ifndef::no-processors[]
All the <<conditions,conditions>> supported by processors are also supported
here.
endif::no-processors[]
endif::no-processors[]


===== `key`
Expand Down Expand Up @@ -955,6 +955,10 @@ brokers, topics, partition, and active leaders to use for publishing.

*`refresh_frequency`*:: Metadata refresh interval. Defaults to 10 minutes.

*`full`*:: Strategy to use when fetching metadata, when this option is `true`, the client will maintain
a full set of metadata for all the available topics, if the this option is set to `false` it will only refresh the
metadata for the configured topics. The default is true.

*`retry.max`*:: Total number of metadata update retries when cluster is in middle of leader election. The default is 3.

*`retry.backoff`*:: Waiting time between retries during leader elections. Default is 250ms.
Expand Down Expand Up @@ -1099,7 +1103,7 @@ output.redis:


TIP: To learn how to add custom fields to events, see the
<<libbeat-configuration-fields,`fields`>> option.
<<libbeat-configuration-fields,`fields`>> option.

See the <<keys-option-redis,`keys`>> setting for other ways to set the key
dynamically.
Expand Down
56 changes: 56 additions & 0 deletions docs/copied-from-beats/repositories.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ ifeval::["{release-state}"=="prerelease"]
--------------------------------------------------
echo "deb https://artifacts.elastic.co/packages/{major-version}-prerelease/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}-prerelease.list
--------------------------------------------------
+
endif::[]
ifeval::["{release-state}"=="released"]
. Save the repository definition to +/etc/apt/sources.list.d/elastic-{major-version}.list+:
Expand All @@ -64,7 +65,34 @@ ifeval::["{release-state}"=="released"]
--------------------------------------------------
echo "deb https://artifacts.elastic.co/packages/{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list
--------------------------------------------------
+
endif::[]
[NOTE]
==================================================

The package is free to use under the Elastic license. An alternative package
which contains only features that are available under the Apache 2.0 license is
also available. To install it, use the following sources list:

ifeval::["{release-state}"=="prerelease"]

["source","sh",subs="attributes"]
--------------------------------------------------
echo "deb https://artifacts.elastic.co/packages/oss-{major-version}-prerelease/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}-prerelease.list
--------------------------------------------------

endif::[]

ifeval::["{release-state}"!="prerelease"]

["source","sh",subs="attributes"]
--------------------------------------------------
echo "deb https://artifacts.elastic.co/packages/oss-{major-version}/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-{major-version}.list
--------------------------------------------------

endif::[]

==================================================
+
[WARNING]
==================================================
Expand Down Expand Up @@ -147,6 +175,34 @@ type=rpm-md
--------------------------------------------------
endif::[]
+
[NOTE]
==================================================

The package is free to use under the Elastic license. An alternative package
which contains only features that are available under the Apache 2.0 license is
also available. To install it, use the following `baseurl` in your
`.repo` file:

ifeval::["{release-state}"=="prerelease"]

["source","sh",subs="attributes"]
--------------------------------------------------
baseurl=https://artifacts.elastic.co/packages/oss-{major-version}-prerelease/yum
--------------------------------------------------

endif::[]

ifeval::["{release-state}"!="prerelease"]

["source","sh",subs="attributes"]
--------------------------------------------------
baseurl=https://artifacts.elastic.co/packages/oss-{major-version}/yum
--------------------------------------------------

endif::[]

==================================================
+
Your repository is ready to use. For example, you can install {beatname_uc} by
running:
+
Expand Down
6 changes: 5 additions & 1 deletion docs/copied-from-beats/shared-docker.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,11 @@ ifndef::apm-server[]
==== Run the {beatname_uc} setup

Running {beatname_uc} with the setup command will create the index pattern and
load visualizations, dashboards, and machine learning jobs. Run this command:
load visualizations
ifndef::no_dashboards[]
, dashboards,
endif::no_dashboards[]
and machine learning jobs. Run this command:

ifeval::[("{beatname_lc}"=="filebeat") or ("{beatname_lc}"=="metricbeat") or ("{beatname_lc}"=="heartbeat") or ("{beatname_lc}"=="journalbeat")]
["source", "sh", subs="attributes"]
Expand Down
2 changes: 1 addition & 1 deletion docs/copied-from-beats/shared-path-config.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
//////////////////////////////////////////////////////////////////////////

[[configuration-path]]
== Set up project paths
== Configure project paths

The `path` section of the +{beatname_lc}.yml+ config file contains configuration
options that define where {beatname_uc} looks for its files. For example, {beatname_uc}
Expand Down
Loading