-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support shipping directly to monitoring cluster #9260
Conversation
@urso @ruflin This PR is very much a WIP but I'd like some early feedback from you to know if I'm on the right track (since this is my first non-trivial PR to libbeat code). Specifically at this point I'm wondering what you think of how/when I'm getting the
|
4eb0c4e
to
52ac31d
Compare
170e5fe
to
76013d0
Compare
Is this already ready to go? Asking because I see elastic/kibana#27595 is still open? If someone configures What is the expected behaviour if someone connects to an OSS cluster? Will the data be shipped or not? |
This is ready for review. But, as the note at the top of this PR says, it should not be merged until elastic/kibana#27595 is merged first. Basically I would like to have this PR reviewed and ready to merge so we can just push the button as soon as elastic/kibana#27595 gets merged.
Yes, the end result will be the same: exactly the same documents will end up in [EDIT] This means that we expect — and want — users to use the
The data should not be shipped. I would expect the checks in this method to fail and corresponding errors to be logged: beats/libbeat/monitoring/report/elasticsearch/client.go Lines 48 to 81 in 4f826ed
[EDIT] I just confirmed this by actually running this PR with an OSS ES distribution. The same
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a few minor comments. I think this definitively deserves a changelog entry.
For the testing: It would be great to have system tests for this that ingest some data with both options and then check if the data is there.
76013d0
to
7bdb46c
Compare
888e4e3
to
40d09bb
Compare
0dfb412
to
e980dc9
Compare
e7f57fa
to
c64a59b
Compare
🎉 🍾 |
) Beats now have the ability to send their monitoring data directly to the **monitoring** Elasticsearch cluster (#9260). This PR updates the default and reference configuration files for all Beats with the new monitoring settings. It also updates the configuration template used by Central Management.
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege. [Going forward](elastic/beats#9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist. This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege. [Going forward](elastic/beats#9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist. This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege. [Going forward](elastic/beats#9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist. This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege. [Going forward](elastic/beats#9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist. This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
…stic#11677) Beats now have the ability to send their monitoring data directly to the **monitoring** Elasticsearch cluster (elastic#9260). This PR updates the default and reference configuration files for all Beats with the new monitoring settings. It also updates the configuration template used by Central Management.
Currently all beats ship their x-pack monitoring data to the production Elasticsearch cluster by
POST
ing it to the custom_xpack/monitoring/_bulk
Elasticsearch API endpoint.In the future we want to support beats shipping their x-pack monitoring data directly to a monitoring Elasticsearch cluster (which could be the same as their production cluster but doesn't necessarily have to be) by
POST
ing it to the regular_bulk
Elasticsearch API endpoint.This PR introduces this "direct shipping" feature. Concretely:
It introduces a new class of settings named
monitoring.*
that will parallel the currentxpack.monitoring.*
settings. If the user has setxpack.monitoring.*
the beat will continue to ship to the production cluster as it does today (but also emit a deprecation warning about these settings). However, if the user has set themonitoring.*
, the beat will instead ship directly to the monitoring cluster.It modifies the Elastisearch reporter to decide where to ship the data (production cluster,
_xpack/monitoring/_bulk
API endpoint -or- monitoring cluster,_bulk
API endpoint).If data is being shipped directly to the monitoring cluster AND the beat is configured to use an
elasticsearch
output, the Elasticsearch reporter will inject thiscluster_uuid
of theelasticsearch
output cluster into the monitoring data event before publishing it.