Add documentation for the elastic-agent diagnostics pprof commands (#…

…1284) (#1373)

* Add documentation for the elastic-agent diagnostics pprof commands

Add command reference docs for gathering pprof data with the
elastic-agent diagnostics collect and diagnostics pprof commands. Add
secrtion to troubleshooting guide for gathering pprof data into archive.

* Apply suggestions from code review

Co-authored-by: DeDe Morton <[email protected]>

* rename monitoring.pprof to monitoring.pprof.enabled

* set example to false

Co-authored-by: DeDe Morton <[email protected]>
(cherry picked from commit baea492)

Co-authored-by: Michel Laterman <[email protected]>

docs/en/ingest-management/commands.asciidoc

@@ -39,6 +39,9 @@ Note that *credentials are not redacted* in the archive; they may appear in plai
 
 This command is intended for debugging purposes only. The output format and structure of the archive produced by `collect` may change between releases.
 
+If `pprof` is specified, it gathers specific `pprof` data from the {agent} or one of the specified underlying {beats}, and displays to `stdout`.
+By default it gathers a 30s profile of the {agent}.
+
 [discrete]
 === Synopsis
 
@@ -47,8 +50,19 @@ This command is intended for debugging purposes only. The output format and stru
 elastic-agent diagnostics [--help] [--output <string>] [global-flags]
 elastic-agent diagnostics collect [--output <string>]
                                   [--file <string>]
+                                  [--pprof]
+                                  [--pprof-duration <string>]
+                                  [--timeout <string>]
                                   [--help]
                                   [global-flags]
+elastic-agent diagnostics pprof [--file <string>]
+                                [--pprof-type <string]
+                                [--pprof-duration <string>]
+                                [--timeout <string>]
+                                [--pprof-application <string>]
+                                [--pprof-route-key <string>]
+                                [--help]
+                                [global-flags]
 ----
 
 [discrete]
@@ -59,8 +73,31 @@ Output format. If using `collect`, specify `json` or `yaml` (the default).
 Otherwise specify `json`, `yaml`, or `human` (the default).
 
 `--file`::
-Output archive name for the `collect` option.
-Defaults to `elastic-agent-diagnostics-<timestamp>.zip` where the timestamp is the current time in UTC.
+When running `collect`, specifies the output archive name. Defaults to `elastic-agent-diagnostics-<timestamp>.zip`, where the timestamp is the current time in UTC.
++
+When running `pprof`, specifies the output file name. Defaults to `stdout`.
+
+`--pprof`::
+Gather `pprof` data when running `collect` (default `false`).
+
+`--pprof-duration <string>`::
+The length for the `trace` and `profile` data that's collected when gathering `pprof` data.
+Must be a string that can be parsed as a `time.Duration` value (default: `30s`).
+
+`--timeout <string>`::
+The timeout value for the `diagnostics collect` or `diagnostics pprof` command.
+Should be longer then `pprof-duration` as the command needs to gather and process the data.
+Must be a string that can be parsed as a `time.Duration` value (default: `30s + pprof-duration`).
+
+`--pprof-type <string>`::
+The pprof data to gather, one of: `allocs`, `block`, `cmdline`, `goroutine`, `heap`, `mutex`, `profile`, `threadcreate`, or `trace` (default: `profile`).
+
+`--pprof-application <string>`::
+The application name to gather `pprof` data from (default `elastic-agent`).
+
+`--pprof-route-key <string>`::
+The route key (output destination) associated with the application to gather `pprof` data from (default `default`).
+Note that this setting is ignored in the case where `pprof-application=elastic-agent`.
 
 `--help`::
 Show help for the `diagnostics` command.

docs/en/ingest-management/elastic-agent/configuration/elastic-agent-monitoring.asciidoc

@@ -23,7 +23,7 @@ agent.monitoring:
   metrics: true
   # exposes /debug/pprof/ endpoints for Elastic Agent and Beats
   # enable these endpoints if the monitoring endpoint is set to localhost
-  pprof: false
+  pprof.enabled: false
   # specifies output to be used
   use_output: monitoring
 ----
@@ -37,7 +37,7 @@ To enable monitoring, set `agent.monitoring.enabled` to `true`. Also set the
 collected. If neither setting is specified, monitoring is turned off. Set
 `use_output` to specify the output to which monitoring events are sent.
 
-The `agent.monitoring.pprof` option controls whether the {agent} and {beats} expose the
+The `agent.monitoring.pprof.enabled` option controls whether the {agent} and {beats} expose the
 `/debug/pprof/` endpoints with the monitoring endpoints. It is set to `false`
 by default. Data produced by these endpoints can be useful for debugging but present a
 security risk. It is recommended that this option remains `false` if the monitoring endpoint

docs/en/ingest-management/troubleshooting/troubleshooting.asciidoc

@@ -253,9 +253,22 @@ This archive collects the following information:
 
 Note that the diagnostics bundle is intended for debugging purposes only, its structure may change between releases.
 
-IMPORTANT: The configration files in the archive may have credentials in *plain text*.
+IMPORTANT: The configuration files in the archive may have credentials in *plain text*.
 These will need to be manually redacted before the archive is shared.
 
+The diagnostics command is also able to collect `pprof` data in the archive if the `--pprof` flag is passed:
+
+[source,shell]
+----
+elastic-agent diagnostics collect --pprof
+----
+
+By default this command takes a little more than 30s to run because it collects 30s trace and profile information from the {agent} and all underlying {beats}.
+To change the duration of the trace and profile data, set the `--pprof-duration` flag.
+
+To set the timeout value for the collection command, use the `--timeout` flag. The timeout defaults to 30s + `pprof-duration`.
+Use this flag to increase the timeout if the diagnostics command times out before creating the archive.
+
 [discrete]
 [[not-installing-no-logs-in-terminal]]
 == Some problems occur so early that insufficient logging is available