Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elastic-agent diagnostics pprof #28798

Merged
merged 20 commits into from
Dec 14, 2021
Merged

elastic-agent diagnostics pprof #28798

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
dd06f1f
Allow -httpprof to bind to sockets/pipes
michel-laterman Nov 3, 2021
b035fe8
Enable pprof debug endpoint on socket for agent and beats
michel-laterman Nov 4, 2021
78069ae
Add new Pprof command to control.proto
michel-laterman Nov 6, 2021
131bf1f
Add pprof option to diagnostics collect
michel-laterman Nov 6, 2021
c61c010
Fix linting issues
michel-laterman Nov 8, 2021
e3fba76
Add diagonstics pprof command allow pprof to collect from agent
michel-laterman Nov 8, 2021
f750e5b
Merge branch 'master' into elastic-agent-profile
michel-laterman Nov 17, 2021
dc66dbd
Revert debug socket changes
michel-laterman Nov 17, 2021
bbaf43a
Cleanup timeout handling
michel-laterman Nov 17, 2021
f29e207
Merge remote-tracking branch 'origin/master' into elastic-agent-profile
michel-laterman Nov 22, 2021
c82f697
Fix linting issue add timeout flag
michel-laterman Nov 23, 2021
a9b3693
Add more command help text.
michel-laterman Nov 23, 2021
b4ffdf0
Add CHANGELOG
michel-laterman Nov 23, 2021
8f145a6
move spec collection for routes to fn
michel-laterman Nov 25, 2021
1b2a9ff
add monitoringCfg reference to control server
michel-laterman Nov 25, 2021
639707c
Merge remote-tracking branch 'origin/master' into elastic-agent-profile
michel-laterman Dec 2, 2021
e6707c4
elastic-agent server only processes pprof requests when enabled
michel-laterman Dec 2, 2021
1bdc078
Fix error message fix commands only on elastic-agent
michel-laterman Dec 2, 2021
91140b7
Add pprof fleet.yml, fix nil reference
michel-laterman Dec 6, 2021
4e6ecbb
Change pprof setting name to monitoring.pprof.enabled
michel-laterman Dec 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions x-pack/elastic-agent/CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,4 +149,6 @@
- Add diagnostics command to gather beat metadata. {pull}28265[28265]
- Add diagnostics collect command to gather beat metadata, config, policy, and logs and bundle it into an archive. {pull}28461[28461]
- Add `KIBANA_FLEET_SERVICE_TOKEN` to Elastic Agent container. {pull}28096[28096]
- Enable pprof endpoints for beats processes. Allow pprof endpoints for elastic-agent if enabled. {pull}28983[28983]
- Add `--pprof` flag to `elastic-agent diagnostics` and an `elastic-agent pprof` command to allow operators to gather pprof data from the agent and beats running under it. {pull}28798[28798]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets make sure we follow up with proper docs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already in a pr

- Allow pprof endpoints for elastic-agent or beats if enabled. {pull}28983[28983] {pull}29155[29155]
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/_meta/config/common.p2.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ inputs:
# metrics: true
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/_meta/config/common.reference.p2.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ inputs:
# metrics: false
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/_meta/config/elastic-agent.docker.yml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ inputs:
# metrics: false
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
42 changes: 42 additions & 0 deletions x-pack/elastic-agent/control.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@ enum ActionStatus {
FAILURE = 1;
}

// pprof endpoint that can be requested.
enum PprofOption {
ALLOCS = 0;
BLOCK = 1;
CMDLINE = 2;
GOROUTINE = 3;
HEAP = 4;
MUTEX = 5;
PROFILE = 6;
THREADCREATE = 7;
TRACE = 8;
}

// Empty message.
message Empty {
}
Expand Down Expand Up @@ -128,6 +141,32 @@ message ProcMetaResponse {
repeated ProcMeta procs = 1;
}

// PprofRequest is a request for pprof data from and http/pprof endpoint.
message PprofRequest {
// The profiles that are requested
repeated PprofOption pprofType = 1;
// A string representing a time.Duration to apply to trace, and profile options.
string traceDuration = 2;
// The application that will be profiled, if empty all applications are profiled.
string appName = 3;
// The route key to match for profiling, if empty all are profiled.
string routeKey = 4;
}

// PprofResult is the result of a pprof request for a given application/route key.
message PprofResult {
string appName = 1;
string routeKey = 2;
PprofOption pprofType = 3;
bytes result = 4;
string error = 5;
}

// PprofResponse is a wrapper to return all pprof responses.
message PprofResponse {
repeated PprofResult results = 1;
}

service ElasticAgentControl {
// Fetches the currently running version of the Elastic Agent.
rpc Version(Empty) returns (VersionResponse);
Expand All @@ -143,4 +182,7 @@ service ElasticAgentControl {

// Gather all running process metadata.
rpc ProcMeta(Empty) returns (ProcMetaResponse);

// Gather requested pprof data from specified applications.
rpc Pprof(PprofRequest) returns (PprofResponse);
}
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/elastic-agent.docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ inputs:
# metrics: false
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/elastic-agent.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ inputs:
# metrics: false
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/elastic-agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ inputs:
# metrics: true
# # exposes /debug/pprof/ endpoints
# # recommended that these endpoints are only enabled if the monitoring endpoint is set to localhost
# pprof: false
# pprof.enabled: false
# # exposes agent metrics using http, by default sockets and named pipes are used
# http:
# # enables http endpoint
Expand Down
7 changes: 4 additions & 3 deletions ...tic-agent/pkg/agent/application/pipeline/actions/handlers/handler_action_policy_change.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,10 @@ func fleetToReader(agentInfo *info.AgentInfo, cfg *configuration.Configuration)
configToStore := map[string]interface{}{
"fleet": cfg.Fleet,
"agent": map[string]interface{}{
"id": agentInfo.AgentID(),
"logging.level": cfg.Settings.LoggingConfig.Level,
"monitoring.http": cfg.Settings.MonitoringConfig.HTTP,
"id": agentInfo.AgentID(),
"logging.level": cfg.Settings.LoggingConfig.Level,
"monitoring.http": cfg.Settings.MonitoringConfig.HTTP,
"monitoring.pprof": cfg.Settings.MonitoringConfig.Pprof,
},
}

Expand Down
Loading