Log by default to journald on systems with systemd

CHANGELOG-developer.asciidoc

@@ -31,6 +31,7 @@ The list below covers the major changes between 6.3.0 and master only.
   used because the value is specified in magefile.go. {pull}7670[7670]
 - Outputs must implement String. {pull}6404[6404]
 - Renamed `-beat-name` CLI option used in `kibana_index_pattern.go` to `-beat` for consistency with other scripts in `dev-tools/cmd`. {pull}8615[8615]
+- Systemd unit file template used on Linux packaging now includes environment variables to ease flag overriding. One of them includes the `-e` flag, making beats log to stderr by default on systemd uses. {pull}8942[8942]
 - Removed dashboards and index patterns generation for Kibana 5. {pull}8927[8927]
 - Move generator packages of Filebeat from `scripts/generator` to `generator`. {pull}9147[9147]
 

CHANGELOG.asciidoc

@@ -128,6 +128,7 @@ https://github.com/elastic/beats/compare/v6.5.0...v7.0.0-alpha1[View commits]
 *Affecting all Beats*
 
 - Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054]
+- On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. {pull}8942[8942].
 - Removed dashboards and index patterns generation for Kibana 5. {pull}8927[8927]
 
 *Auditbeat*

auditbeat/docs/setting-up-running.asciidoc

@@ -22,6 +22,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-kubernetes>>
 
+* <<running-with-systemd>>
+
 
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
 
@@ -35,4 +37,6 @@ include::./running-on-docker.asciidoc[]
 
 include::./running-on-kubernetes.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]

dev-tools/packaging/templates/linux/systemd.unit.tmpl

@@ -5,7 +5,10 @@ Wants=network-online.target
 After=network-online.target
 
 [Service]
-ExecStart=/usr/share/{{.BeatName}}/bin/{{.BeatName}} -c /etc/{{.BeatName}}/{{.BeatName}}.yml -path.home /usr/share/{{.BeatName}} -path.config /etc/{{.BeatName}} -path.data /var/lib/{{.BeatName}} -path.logs /var/log/{{.BeatName}}
+Environment=BEAT_LOG_OPTS="-e"
+Environment=BEAT_CONFIG_OPTS="-c /etc/{{.BeatName}}/{{.BeatName}}.yml"
+Environment=BEAT_PATH_OPTS="-path.home /usr/share/{{.BeatName}} -path.config /etc/{{.BeatName}} -path.data /var/lib/{{.BeatName}} -path.logs /var/log/{{.BeatName}}"
+ExecStart=/usr/share/{{.BeatName}}/bin/{{.BeatName}} $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
 Restart=always
 
 [Install]

filebeat/docs/setting-up-running.asciidoc

@@ -22,6 +22,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-kubernetes>>
 
+* <<running-with-systemd>>
+
 
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
 
@@ -35,4 +37,6 @@ include::./running-on-docker.asciidoc[]
 
 include::./running-on-kubernetes.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]

heartbeat/docs/setting-up-running.asciidoc

@@ -20,6 +20,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-docker>>
 
+* <<running-with-systemd>>
+
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
 
 include::../../libbeat/docs/shared-directory-layout.asciidoc[]
@@ -30,4 +32,6 @@ include::../../libbeat/docs/command-reference.asciidoc[]
 
 include::./running-on-docker.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]

journalbeat/docs/setting-up-running.asciidoc

@@ -17,6 +17,7 @@ This section includes additional information on how to set up and run
 * <<directory-layout>>
 * <<keystore>>
 * <<command-line-options>>
+* <<running-with-systemd>>
 * <<shutdown>>
 
 
@@ -28,4 +29,6 @@ include::../../libbeat/docs/keystore.asciidoc[]
 
 include::../../libbeat/docs/command-reference.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]

libbeat/docs/shared-systemd.asciidoc

@@ -0,0 +1,103 @@
+[[running-with-systemd]]
+=== {beatname_uc} and systemd
+
+The DEB and RPM packages include a service unit for Linux systems with
+systemd. On these systems, you can manage {beatname_uc} by using the usual
+systemd commands.
+
+==== Start and stop {beatname_uc}
+
+Use `systemctl` to start or stop {beatname_uc}:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl start {beatname_lc}
+------------------------------------------------
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl stop {beatname_lc}
+------------------------------------------------
+
+By default, the {beatname_uc} service starts automatically when the system
+boots. To enable or disable auto start use:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl enable {beatname_lc}
+------------------------------------------------
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl disable {beatname_lc}
+------------------------------------------------
+
+
+==== {beatname_uc} status and logs
+
+To get the service status, use `systemctl`:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl status {beatname_lc}
+------------------------------------------------
+
+Logs are stored by default in journald. To view the Logs, use `journalctl`:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+journalctl -u {beatname_lc}.service
+------------------------------------------------
+
+NOTE: The unit file included in the packages sets the `-e` flag by default.
+This flag makes {beatname_uc} log to stderr and disables other log outputs.
+Systemd stores all output sent to stderr in journald.
+
+[float]
+=== Customize systemd unit for {beatname_uc}
+
+The systemd service unit file includes environment variables that you can
+override to change the default options.
+
+[cols="<h,<,<m",options="header",]
+|=======================================
+| Variable | Description | Default value
+| BEAT_LOG_OPTS | Log options | `-e`
+| BEAT_CONFIG_OPTS | Flags for configuration file path | +-c /etc/{beatname_lc}/{beatname_lc}.yml+
+| BEAT_PATH_OPTS | Other paths | +-path.home /usr/share/{beatname_lc} -path.config /etc/{beatname_lc} -path.data /var/lib/{beatname_lc} -path.logs /var/log/{beatname_lc}+
+|=======================================
+
+To override these variables, create a drop-in unit file in the
++/etc/systemd/system/{beatname_lc}.service.d+ directory.
+
+For example a file with the following content placed in
++/etc/systemd/system/{beatname_lc}.service.d/debug.conf+
+would override `BEAT_LOG_OPTS` to enable debug for Elasticsearch output.
+
+["source", "systemd", subs="attributes"]
+------------------------------------------------
+[Service]
+Environment=BEAT_LOG_OPTS="-e -d elasticsearch"
+------------------------------------------------
+
+To use settings from the {beatname_uc} file, empty the environment
+variable. For example:
+
+["source", "systemd", subs="attributes"]
+------------------------------------------------
+[Service]
+Environment=BEAT_LOG_OPTS=""
+------------------------------------------------
+
+To apply your changes, reload the systemd configuration and restart
+the service:
+
+["source", "sh", subs="attributes"]
+------------------------------------------------
+systemctl daemon-reload
+systemctl restart {beatname_lc}
+------------------------------------------------
+
+NOTE: It is recommended that you use a configuration management tool to
+include drop-in unit files. If you need to add a drop-in manually, use
++systemctl edit {beatname_lc}.service+.

metricbeat/docs/setting-up-running.asciidoc

@@ -22,6 +22,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-kubernetes>>
 
+* <<running-with-systemd>>
+
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
 
 include::../../libbeat/docs/shared-directory-layout.asciidoc[]
@@ -34,4 +36,6 @@ include::./running-on-docker.asciidoc[]
 
 include::./running-on-kubernetes.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]

packetbeat/docs/setting-up-running.asciidoc

@@ -20,6 +20,8 @@ This section includes additional information on how to set up and run
 
 * <<running-on-docker>>
 
+* <<running-with-systemd>>
+
 //MAINTAINERS: If you add a new file to this section, make sure you update the bulleted list ^^ too.
 
 include::../../libbeat/docs/shared-directory-layout.asciidoc[]
@@ -30,4 +32,6 @@ include::../../libbeat/docs/command-reference.asciidoc[]
 
 include::./running-on-docker.asciidoc[]
 
+include::../../libbeat/docs/shared-systemd.asciidoc[]
+
 include::../../libbeat/docs/shared-shutdown.asciidoc[]