Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[metricbeat] [auditbeat] Add formatted index option to metricbeat / auditbeat modules #15100

Merged
merged 6 commits into from
Dec 16, 2019

Conversation

faec
Copy link
Contributor

@faec faec commented Dec 13, 2019

Resolves #14923 and #15065.

Adds an index formatted string option to all metricbeat modules that, when present, specifies the (elasticsearch) output index for events from that module.

@faec faec added Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. labels Dec 13, 2019

// KeepNull determines whether published events will keep null values or omit them.
KeepNull bool `config:"keep_null"`

common.EventMetadata `config:",inline"` // Fields and tags to add to events.
}

func NewConnector(pipeline beat.Pipeline, c *common.Config, dynFields *common.MapStrPointer) (*Connector, error) {
func NewConnector(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function NewConnector should have comment or be unexported


// KeepNull determines whether published events will keep null values or omit them.
KeepNull bool `config:"keep_null"`

common.EventMetadata `config:",inline"` // Fields and tags to add to events.
}

func NewConnector(pipeline beat.Pipeline, c *common.Config, dynFields *common.MapStrPointer) (*Connector, error) {
func NewConnector(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exported function NewConnector should have comment or be unexported

@faec faec requested a review from ycombinator December 13, 2019 18:15
}
procs.AddProcessors(*userProcs)

return procs, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I quite liked the comments you had in the Filebeat implementation, numbering the processor additions and also noting that ordering was significant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I removed it because metricbeat processor config is simpler (only one configurable processor source), but you're right, an indication of intent here is good so I re-added a comment :-)


// KeepNull determines whether published events will keep null values or omit them.
KeepNull bool `config:"keep_null"`

common.EventMetadata `config:",inline"` // Fields and tags to add to events.
}

func NewConnector(pipeline beat.Pipeline, c *common.Config, dynFields *common.MapStrPointer) (*Connector, error) {
func NewConnector(
beatInfo beat.Info, pipeline beat.Pipeline,
Copy link
Contributor

@ycombinator ycombinator Dec 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been torn about this sort of change in one of my related PRs as well. Basically, we're now passing an extra beat.Info parameter to this function. Looking at the calls to this constructor, they now look like this:

module.NewConnector(b.Info, b.Publisher, ...)

where b is of type *beat.Beat.

I wonder if should just make this constructor take b *beat.Beat instead of beatInfo beat.Info, pipeline beat.Pipeline. I could go either way on this. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My usual preference is to pass the simpler information, both to reduce contextual load (a BeatInfo parameter implies a lot more about the function's behavior than a broadly-scoped container that could be affecting many things in non-obvious ways) and because it's easy to refactor to beat.Beat in the future if we ever really need to, but hard to take it back once it's in the API and people start depending on that field in unexpected ways :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, that's fair. I think my concern of adding more arguments is not as strong in comparison, at least not at this point, where the function would be taking in only 4 arguments. If that number ever grows to be really unwieldy, we can revist.

+1 for keeping this change as-is.

Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@faec faec added the Auditbeat label Dec 16, 2019
@faec faec changed the title [metricbeat] Add formatted index option to metricbeat modules [metricbeat] [auditbeat] Add formatted index option to metricbeat / auditbeat modules Dec 16, 2019
@faec
Copy link
Contributor Author

faec commented Dec 16, 2019

All remaining test failures are Jenkins timeouts that are also happening at head; Travis is green, so going ahead with merge.

@faec faec merged commit ac9434b into elastic:master Dec 16, 2019
@faec faec deleted the metricbeat-input-index branch December 16, 2019 18:33
faec added a commit to faec/beats that referenced this pull request Dec 16, 2019
@faec faec added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 16, 2019
faec added a commit that referenced this pull request Dec 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[metricbeat] Allow inputs to specify output index
3 participants