[Elastic Agent] Improve Elastic Agent debuggability

[ruflin]

The Elastic Agent is a supervisor which runs sub processes. This combination of multiple process and dependency on Fleet makes debugging the Elastic Agent sometimes tricky. One part is for users to provide the necessary and the other is for engineers to reproduce and understand the issue. Just on example can be found here: #26213

Improving this consists of many different parts. This issue is to link together this overall effort.

Related issues (incomplete list):

High Prio

• [Elastic Agent] Improve error logging (reduce to Info Level where possible) wrt changing logging level to debug elastic-agent#154
• [Elastic Agent] How to monitor inputs #21413
• Agent should log the exit code of processes and write logs stdout and stderr of each process to file before exiting elastic-agent#101
• Display which integration is causing the agent to become unhealthy elastic-agent#100
• "elastic fail 503::" - Improve log message for better debug fleet-server#1128
• https://github.com/elastic/enhancements/issues/14587
• Integrate Elastic Endpoint's log output and available stats into Elastic Agent 'diagnostic' collection  elastic-agent#105
• added more detailed logs around ES communication failure fleet-server#992
• [ ]
• https://github.com/elastic/sdh-beats/issues/1569 (modifications to the download timeout and extra logging for progression of download)
• elastic-agent diagnostics: filebeat and metricbeat configuration #29898
• [Agent] Improve error log messages when agent binary cannot be downloaded elastic-agent#104
• Elastic-agent should not swallows stderr and panics elastic-agent#88
• Correctly aggregates the health status of the running process, a problem with a sub process should impact the agent status. [Fleet] Improve agent observability kibana#78188
  [DESIGN][Enhancement][Elastic Agent] Write down Elastic Agent status report structure elastic-agent#79

Medium Prio

• [Elastic Agent] Debug fleet-server connectivity elastic-agent#115
• Display the amount of memory/cpu in use by the agent, including cgroup metrics(a snapshot. are there other useful metrics?)
• Make enrollment errors more accurate #27764
• Do not log ERROR for watchers in not upgradable scenarios  #28334
• Fleet installation script fails to detect error in service start  elastic-agent#117
• Elastic Agent inspect to include windows registry key elastic-agent#99
• Implement a sub-command to show or follow the Elastic Agent logs elastic-agent#114
• Allow elastic-agent diagnostics collect to ignore failures elastic-agent#92

Low Prio

• Display how much data is pooled to disk
• [Elastic Agent] Improve error logging (reduce to Info Level where possible) wrt changing logging level to debug elastic-agent#154
• https://github.com/elastic/enhancements/issues/13780
• Inspect command does not show fleet server config elastic-agent#103

---

• [elastic-agent] Improve status output for different applications #25449
• * [Elastic Agent] Improve logging #25230
• * [Elastic Agent] Expose httpprof for Elastic Agent #27059
• * Elastic Agent Diagnostics support #27138
• elastic-agent diagnostics pprof #28798
  https://github.com/elastic/kibana/issues/78188√https://github.com/elastic/kibana/issues/78188

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[jlind23]

@nimarezainia this issue: elastic/elastic-agent#104
Is becoming pretty urgent as a lot of users were experiencing issue while downloading our binary. I would be happy to treat it as an high priority.

[jlind23]

Transferring here Marcin comment from another issue related to this one - elastic/kibana#78188

We had an offline conversation with dover around improvements.

There is a noticeable amount of SDH issues coming, which end up with a root cause, or one of the possible causes, like proxy connectivity issues. The customer has to dive into logs to figure out if the used proxy operates properly (whether connections are established, no 503s, etc.).

I believe we could more proactive and verify the connectivity between Agent and Elasticsearch, Agent and Fleet Server. I was thinking about a special technical policy first to verify all connections and settings, but maybe we can start with picking up the elastic-agent install feedback.

It would definitely help with researching customer problems ("Has your proxy ever worked?" vs "Is there an proxy outage now?").

[mtojek]

Another lost customer: https://discuss.elastic.co/t/cant-enroll-an-agent-to-create-fleet-server-no-error-shown-in-log/295375