Support shipping directly to monitoring cluster

[ycombinator]

Currently all beats ship their x-pack monitoring data to the production Elasticsearch cluster by POSTing it to the custom _xpack/monitoring/_bulk Elasticsearch API endpoint.

In the future we want to support beats shipping their x-pack monitoring data directly to a monitoring Elasticsearch cluster (which could be the same as their production cluster but doesn't necessarily have to be) by POSTing it to the regular _bulk Elasticsearch API endpoint.

This PR introduces this "direct shipping" feature. Concretely:

• It introduces a new class of settings named monitoring.* that will parallel the current xpack.monitoring.* settings. If the user has set xpack.monitoring.* the beat will continue to ship to the production cluster as it does today (but also emit a deprecation warning about these settings). However, if the user has set the monitoring.*, the beat will instead ship directly to the monitoring cluster.

• It modifies the Elastisearch reporter to decide where to ship the data (production cluster, _xpack/monitoring/_bulk API endpoint -or- monitoring cluster, _bulk API endpoint).

• If data is being shipped directly to the monitoring cluster AND the beat is configured to use an elasticsearch output, the Elasticsearch reporter will inject this cluster_uuid of the elasticsearch output cluster into the monitoring data event before publishing it.

[ycombinator]

@urso @ruflin This PR is very much a WIP but I'd like some early feedback from you to know if I'm on the right track (since this is my first non-trivial PR to libbeat code).

Specifically at this point I'm wondering what you think of how/when I'm getting the cluster_uuid from the ES output cluster and then how I can properly use it in the ES reporter to add it to the monitoring event.

Note that I haven't yet added the logic to ship to the regular _bulk API on the monitoring cluster instead of the custom _xpack/monitoring/_bulk API on the production cluster. I've started implementing this but it's not quite working yet.

[ycombinator]

@urso @ruflin I've just finished the pending work on this PR and it is ready for your review again. Thanks!

[ruflin]

Is this already ready to go? Asking because I see elastic/kibana#27595 is still open?

If someone configures monitoring.* with the same values as in x-pack.monitoring the outcome will be identical?

What is the expected behaviour if someone connects to an OSS cluster? Will the data be shipped or not?

[ycombinator]

Is this already ready to go? Asking because I see elastic/kibana#27595 is still open?

This is ready for review. But, as the note at the top of this PR says, it should not be merged until elastic/kibana#27595 is merged first. Basically I would like to have this PR reviewed and ready to merge so we can just push the button as soon as elastic/kibana#27595 gets merged.

If someone configures monitoring.* with the same values as in x-pack.monitoring the outcome will be identical?

Yes, the end result will be the same: exactly the same documents will end up in .monitoring-beats-* indices. The difference will be in how they get there. If the user configures xpack.monitoring.*, they will get there via the POST _xpack/monitoring/bulk API on the production ES cluster. However, if the user configures monitoring.*, they will get there via the POST _bulk API on the monitoring cluster.

[EDIT] This means that we expect — and want — users to use the monitoring.* settings and pointing to the Monitoring cluster, instead of using the xpack.monitoring.* settings and pointing to the Production cluster. This "want" is reflected in the deprecation log messages implemented by this PR.

What is the expected behaviour if someone connects to an OSS cluster? Will the data be shipped or not?

The data should not be shipped. I would expect the checks in this method to fail and corresponding errors to be logged:

beats/libbeat/monitoring/report/elasticsearch/client.go

Lines 48 to 81 in 4f826ed

	func (c *publishClient) Connect() error {
	debugf("Monitoring client: connect.")
	params := map[string]string{
	"filter_path": "features.monitoring.enabled",
	}
	status, body, err := c.es.Request("GET", "/_xpack", "", params, nil)
	if err != nil {
	return fmt.Errorf("X-Pack capabilities query failed with: %v", err)
	}
	if status != 200 {
	return fmt.Errorf("X-Pack capabilities query failed with status code: %v", status)
	}
	resp := struct {
	Features struct {
	Monitoring struct {
	Enabled bool
	}
	}
	}{}
	if err := json.Unmarshal(body, &resp); err != nil {
	return err
	}
	if !resp.Features.Monitoring.Enabled {
	debugf("XPack monitoring is disabled.")
	return errNoMonitoring
	}
	debugf("XPack monitoring is enabled")
	return nil
	}

[EDIT] I just confirmed this by actually running this PR with an OSS ES distribution. The same INFO messages that are emitted by master in this scenario when the user uses xpack.monitoring.* are emitted by this PR when the user uses either xpack.monitoring.* or monitoring.*:

2019-01-22T05:49:11.839-0800    INFO    [monitoring]    elasticsearch/elasticsearch.go:234      Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying.

[ruflin]

Left a few minor comments. I think this definitively deserves a changelog entry.

For the testing: It would be great to have system tests for this that ingest some data with both options and then check if the data is there.

[ycombinator]

I started writing a system test for this "direct monitoring" feature per @ruflin's recommendation. While doing so I realized we don't have a system test for the current "indirect monitoring" feature either. So I'm going to first write that in #10645.

[cachedout]

🎉 🍾