[Elastic-Agent] Enable introspecting configuration #18124

michalpristas · 2020-04-30T08:05:14Z

What does this PR do?

This PR adds 3 options to our agents CLI

Option 1 - Introspecting current config

Usage ./elastic-agent inspect
This will print out current configuration either from fleet or from yml.
Sample output:

datasources:
- inputs:
  - streams:
    - dataset: system.cpu
      metricset: cpu
    - dataset: system.memory
      metricset: memory
    - dataset: system.network
      metricset: network
    - dataset: system.filesystem
      metricset: filesystem
    type: system/metrics
  namespace: default
  use_output: default
download:
  install_path: /Users/michalpristas/agent/data/install
  pgpfile: /Users/michalpristas/agent/data/elastic.pgp
  sourceURI: https://artifacts.elastic.co/downloads/beats/
  target_directory: /Users/michalpristas/agent/data/downloads
  timeout: 30s
...

Option 2 - Introspecting used outputs

Usage ./elastic-agent inspect output
This will print out outputs which are actually used . If config contains definition for output but output is not used in any datastream it will not be present in the printout
Sample output:

default
monitoring

Option 3: Introspecting specific output

Usage: ./elastic-agent inspect output -o default
This will print out actual configuration of the beats, this helps you understand how configuration retrieved from fleet translates to configuration which is sent out to the process itself.
Sample output:

[default] metricbeat:
metricbeat:
  modules:
  - index: metrics-system.filesystem-default
    metricsets:
    - filesystem
    module: system
    processors:
    - add_fields:
        fields:
          dataset: system.filesystem
          namespace: default
          type: metrics
        target: stream
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    username: elastic

Why is it important?

To improve debugging experience

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2020-04-30T08:05:16Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-04-30T08:23:28Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request [Elastic-Agent] Enable introspecting configuration #18124 updated]
Start Time: 2020-05-04T16:03:53.535+0000
Duration: 35 min 43 sec (2083435)
Commit: b84c919

ruflin · 2020-04-30T12:50:52Z

I'm used to docker image inspect ...

ph · 2020-04-30T12:51:50Z

x-pack/elastic-agent/pkg/agent/application/introspect_config_cmd.go

+	}, nil
+}
+
+// Execute tries to enroll the agent into Fleet.


You are sure the comment matches the behavior?

oh my copy cat

ruflin · 2020-04-30T12:52:57Z

I'm confused by your Option 3: You select an output and get the metricbeat config for it

michalpristas · 2020-04-30T12:53:45Z

you select the output and you get configuration for all beats writing to that specific output

ruflin · 2020-04-30T12:59:26Z

@michalpristas I wonder if that would be the way someone thinks of trying to find what goes wrong during debugging. But lets iterate later on naming and order of params and get this in.

michalpristas · 2020-04-30T13:00:24Z

i'm open to suggestions,i'm biased by how agent deal with streams

ph

I think this would effectively help integrations team to move forward with debugging, let's fix the issue concerning the -c we can do a follow for my suggestion of been able to export them.

ph · 2020-04-30T17:06:33Z

x-pack/elastic-agent/pkg/agent/cmd/introspect.go

+		Args:  cobra.ExactArgs(0),
+		Run: func(c *cobra.Command, args []string) {
+			command, err := application.NewIntrospectConfigCmd("elastic-agent.yml")
+			if err != nil {


There is a problem here, if you use -c to specify another configuration we will still load the default and not the one specified.

ph · 2020-04-30T17:18:47Z

x-pack/elastic-agent/pkg/agent/cmd/introspect.go

+	cmd.Flags().StringP("program", "p", "", "type of program to introspect, needs to be combined with output. e.g filebeat")
+
+	return cmd
+}


Two things, I've played with the command locally. When you run the "output" subcommand you have only the list of the outputs. Maybe we could have a better default like displaying all the configuration that we have for all the defined outputs?

Maybe we could add a flag that will output to disk all the configurations and grouped output, Maybe -disk, its hard to find a good name because that behavior is often associated to -o (output).

/default/ filebeat.yml metricbeat.yml

The above would allow anyone to test the configuration outside of the agent.

can we do this in a follwup? let's get this in for 7.8 and we can work on naming and use cases in a separate issue. nicolas had also some comments about the form of the command

ph

LGTM get some debug in place.

…o agent-introspect

* introspection * changelog * mage fmt * introspect -> inspect * follow -c argument * conflicts

michalpristas added 3 commits April 28, 2020 14:16

introspection

3dfb1f7

Merge branch 'master' into agent-introspect

007aaf4

Merge branch 'master' into agent-introspect

33dd3ac

michalpristas added enhancement Project:fleet Ingest Management:beta1 Group issues for ingest management beta1 labels Apr 30, 2020

michalpristas self-assigned this Apr 30, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2020

changelog

b03a49d

michalpristas added needs_backport PR is waiting to be backported to other branches. and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 30, 2020

mage fmt

846f0de

michalpristas added Ingest Management:alpha1 Group issues for ingest management alpha1 and removed Ingest Management:beta1 Group issues for ingest management beta1 labels Apr 30, 2020

ph reviewed Apr 30, 2020

View reviewed changes

introspect -> inspect

fef4aeb

ph suggested changes Apr 30, 2020

View reviewed changes

michalpristas added 2 commits May 4, 2020 08:35

Merge branch 'master' into agent-introspect

9502075

follow -c argument

1e97f92

michalpristas added the review label May 4, 2020

ph approved these changes May 4, 2020

View reviewed changes

michalpristas added 4 commits May 4, 2020 17:37

Merge branch 'master' into agent-introspect

cdcabbf

changelog

47204df

conflicts

833662a

Merge branch 'agent-introspect' of github.com:michalpristas/beats int…

b84c919

…o agent-introspect

michalpristas merged commit 1c2e4b9 into elastic:master May 4, 2020

michalpristas added a commit to michalpristas/beats that referenced this pull request May 4, 2020

[Elastic-Agent] Enable introspecting configuration (elastic#18124)

f227b3c

* introspection * changelog * mage fmt * introspect -> inspect * follow -c argument * conflicts

michalpristas mentioned this pull request May 4, 2020

Cherry-pick #18124 to 7.x: Enable introspecting configuration #18186

Closed

6 tasks

michalpristas added a commit to michalpristas/beats that referenced this pull request May 4, 2020

[Elastic-Agent] Enable introspecting configuration (elastic#18124)

adc089c

* introspection * changelog * mage fmt * introspect -> inspect * follow -c argument * conflicts

michalpristas mentioned this pull request May 4, 2020

Cherry-pick #18124 to 7.x: Enable introspecting configuration #18194

Merged

6 tasks

michalpristas added a commit that referenced this pull request May 4, 2020

[Elastic-Agent] Enable introspecting configuration (#18124) (#18194)

a45dd08

* introspection * changelog * mage fmt * introspect -> inspect * follow -c argument * conflicts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Elastic-Agent] Enable introspecting configuration #18124

[Elastic-Agent] Enable introspecting configuration #18124

michalpristas commented Apr 30, 2020 •

edited

Loading

elasticmachine commented Apr 30, 2020

elasticmachine commented Apr 30, 2020 •

edited

Loading

Build stats

ruflin commented Apr 30, 2020

ph Apr 30, 2020

michalpristas Apr 30, 2020

ruflin commented Apr 30, 2020

michalpristas commented Apr 30, 2020

ruflin commented Apr 30, 2020

michalpristas commented Apr 30, 2020

ph left a comment

ph Apr 30, 2020

michalpristas May 4, 2020

ph Apr 30, 2020

michalpristas May 4, 2020

ph left a comment

[Elastic-Agent] Enable introspecting configuration #18124

[Elastic-Agent] Enable introspecting configuration #18124

Conversation

michalpristas commented Apr 30, 2020 • edited Loading

What does this PR do?

Option 1 - Introspecting current config

Option 2 - Introspecting used outputs

Option 3: Introspecting specific output

Why is it important?

Checklist

elasticmachine commented Apr 30, 2020

elasticmachine commented Apr 30, 2020 • edited Loading

💚 Build Succeeded

Build stats

ruflin commented Apr 30, 2020

ph Apr 30, 2020

Choose a reason for hiding this comment

michalpristas Apr 30, 2020

Choose a reason for hiding this comment

ruflin commented Apr 30, 2020

michalpristas commented Apr 30, 2020

ruflin commented Apr 30, 2020

michalpristas commented Apr 30, 2020

ph left a comment

Choose a reason for hiding this comment

ph Apr 30, 2020

Choose a reason for hiding this comment

michalpristas May 4, 2020

Choose a reason for hiding this comment

ph Apr 30, 2020

Choose a reason for hiding this comment

michalpristas May 4, 2020

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

michalpristas commented Apr 30, 2020 •

edited

Loading

elasticmachine commented Apr 30, 2020 •

edited

Loading