Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Cloud Foundry Processor #16621

Merged
merged 8 commits into from
Feb 27, 2020
Merged

Add Cloud Foundry Processor #16621

merged 8 commits into from
Feb 27, 2020

Conversation

blakerouse
Copy link
Contributor

@blakerouse blakerouse commented Feb 26, 2020 •
edited by andresrc
Loading

What does this PR do?

Adds a add_cloudfoundry_metadata processor that adds application metadata to an event with a Cloud Foundry application ID.

Why is it important?

To provide more context on the application for the event. Processor allows this to easily be shared between both metricbeat and filebeat.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • [ ]

How to test this PR locally

With a running cloud foundry us the following filebeat configuration:

filebeat.inputs:
  - &cloudfoundry
    type: cloudfoundry
    api_address: https://api.dev.cfdev.sh
    skip_verify: true
    client_id: firehose-to-filebeat
    client_secret: verysecret

processors:
  - add_cloudfoundry_metadata:
      <<: *cloudfoundry

output.console:
  pretty: true

Related issues

Use cases

Screenshots

Logs

@blakerouse blakerouse requested a review from a team February 26, 2020 17:35
@dedemorton
Copy link
Contributor

@elasticmachine run elasticsearch-ci/docs

2 similar comments
@dedemorton
Copy link
Contributor

@elasticmachine run elasticsearch-ci/docs

@dedemorton
Copy link
Contributor

@elasticmachine run elasticsearch-ci/docs

@exekias exekias added enhancement Team:Platforms Label for the Integrations - Platforms team containers Related to containers use case labels Feb 27, 2020
exekias
exekias suggested changes Feb 27, 2020
View reviewed changes
Copy link
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great, made a few comments

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
x-pack/libbeat/processors/add_cloudfoundry_metadata/_meta/fields.yml Outdated Show resolved Hide resolved
x-pack/libbeat/processors/add_cloudfoundry_metadata/add_cloudfoundry_metadata.go Outdated Show resolved Hide resolved
x-pack/libbeat/processors/add_cloudfoundry_metadata/docs/add_cloudfoundry_metadata.asciidoc Outdated
The `add_cloudfoundry_metadata` processor annotates each event with relevant metadata
from Cloud Foundry applications. The events are annotated with Cloud Foundry metadata,
only if the event contains a reference to a Cloud Foundry application and the configured
Cloud Foundry client is able to retrieve information for the application.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to explicitly explain what field (cf.app.id) is used for correlation. So people wanting to do their own enrichment have everything they need.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay will add, it will become cloudfoundry.app.id. Should a field be added to the config to adjust this field path? Don't think its a requirement, but easy to add, might be have a use case?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have done this in the past, ie:

beats/libbeat/processors/add_process_metadata/config.go

Line 39 in 3a764e6

MatchPIDs []string `config:"match_pids" validate:"required"`
. That said, I'm fine with having a more opinionated behavior here. Nowadays people can use the rename processor to adjust the doc before enriching (this wasn't possible in the past).

@blakerouse blakerouse force-pushed the cloudfoundry-processor branch from b28cb48 to 18bcca6 Compare February 27, 2020 13:20
@blakerouse
Copy link
Contributor Author

@exekias It's ready for another look!

exekias
exekias approved these changes Feb 27, 2020
View reviewed changes
x-pack/libbeat/processors/add_cloudfoundry_metadata/docs/add_cloudfoundry_metadata.asciidoc Outdated
The `add_cloudfoundry_metadata` processor annotates each event with relevant metadata
from Cloud Foundry applications. The events are annotated with Cloud Foundry metadata,
only if the event contains a reference to a Cloud Foundry application and the configured
Cloud Foundry client is able to retrieve information for the application.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have done this in the past, ie:

beats/libbeat/processors/add_process_metadata/config.go

Line 39 in 3a764e6

MatchPIDs []string `config:"match_pids" validate:"required"`
. That said, I'm fine with having a more opinionated behavior here. Nowadays people can use the rename processor to adjust the doc before enriching (this wasn't possible in the past).

@blakerouse blakerouse merged commit 1e0c818 into elastic:master Feb 27, 2020
@blakerouse blakerouse deleted the cloudfoundry-processor branch February 27, 2020 15:38
blakerouse added a commit to blakerouse/beats that referenced this pull request Feb 27, 2020
@blakerouse
Add Cloud Foundry Processor (elastic#16621)
8261800 
* Add the cloudfoundry metadata processor.

* Fix testing and documentation.

* Add changelog entry.

* Run mage fmt.

* Fix docs.

* Update asciidoc to set role=xpack and experimental.

* Switch from cf to cloudfoundry for event key. Improve some event fields. Fix changelog, and update asciidoc to reference the field that is read for application ID.

* Update fields.yml from cf to cloudfoundry.

(cherry picked from commit 1e0c818)
@blakerouse blakerouse mentioned this pull request Feb 27, 2020
Merged
4 tasks
blakerouse added a commit that referenced this pull request Feb 27, 2020
@blakerouse
Cherry-pick #16621 to 7.x: Add Cloud Foundry Processor (#16666)
a621de1 
* Add Cloud Foundry Processor (#16621)

* Add the cloudfoundry metadata processor.

* Fix testing and documentation.

* Add changelog entry.

* Run mage fmt.

* Fix docs.

* Update asciidoc to set role=xpack and experimental.

* Switch from cf to cloudfoundry for event key. Improve some event fields. Fix changelog, and update asciidoc to reference the field that is read for application ID.

* Update fields.yml from cf to cloudfoundry.

(cherry picked from commit 1e0c818)

* Fix changelog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@exekias exekias exekias approved these changes

Assignees
No one assigned
Labels
containers Related to containers use case enhancement Team:Platforms Label for the Integrations - Platforms team v7.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@blakerouse @dedemorton @exekias @andresrc