Add documentation for the elastic-agent diagnostics pprof commands

[michel-laterman]

Add command reference docs for gathering pprof data with the
elastic-agent diagnostics collect and diagnostics pprof commands. Add
secrtion to troubleshooting guide for gathering pprof data into archive.

[apmmachine]

A documentation preview will be available soon:

• 📚 HTML diff
• 📘 Fleet and Elastic Agent Guide
• 📙 Observability Guide

[dedemorton]

A few comment. Otherwise, looks great! Thanks for adding this.

[EricDavisX]

@dikshachauhan-qasource pinging you in the Docs PR - we should review docs as well as test out the diagnostics as part of it on the 3 OSes, with Endpoint and OSQuery enabled as test subjects.

@michel-laterman it will collect Endpoint, yes? And logs from any other sub-process (like OSQuery) or just Beats (like Filebeat/Metricbeat)?

Is there any environment where it is known not to work, like when deployed as a Docker container?

So very cool - so glad we have docs, thanks for working it.

[michel-laterman]

@EricDavisX I know it will collect all *beats. I don't think it will collect from endpoint as it logs under a different path (and i don't know if the agent is aware of endpoint's install path). I have not tested if it interacts with other processes like OSQuery

[amolnater-qasource]

Hi @EricDavisX
We have revalidated elastic-agent diagnostics pprof command on below agents:

• Windows
• Linux
• MAC

We added below integrations to these agents:

• System
• Endpoint Security
• OS Query

Output for Windows:

Output for MAC:

Output for Linux:

Observations on running elastic-agent diagnostics pprof:

• Data is available for filebeat
• Data is available for metricbeat
• Data is available for OS-Query module.
• Name: Endpoint Security is there for all agents but it shows error: Get "http://npipe/": open \\.\pipe\default-endpoint-security: The system cannot find the file specified..

We have also tested this with sudo elastic-agent diagnostics collect command, referred from guide:
We get the desired zip file for logs on all 3 OS's:
MAC:
elastic-agent-diagnostics-2021-12-07T08-53-43Z-00-MAC.zip

Windows:
elastic-agent-diagnostics-2021-12-07T08-07-46Z-00-Windows.zip

Linux:
elastic-agent-diagnostics-2021-12-07T08-46-44Z-00-Linux.zip

I don't think it will collect from endpoint as it logs under a different path

As also confirmed by @michel-laterman we got same output:
These zip files do contains logs for filebeat, metricbeat and os-querybeat but no logs for Endpoint Security.

Further, we will be testing docs whenever this ticket will be merged.

Please let us know if we are missing anything.
Thanks

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[dedemorton]

LGTM.

[dedemorton]

@michel-laterman Not sure if you need to make any more content changes, but what's here looks good to me. I've added the backport labels, but can't remember if this is valid for 7.16. Just remove that backport label if it's not. Thanks!

[mergify]

This pull request is now in conflict. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-diagnostics-pprof upstream/agent-diagnostics-pprof
git merge upstream/main
git push upstream agent-diagnostics-pprof