Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the cluster_uuid field to Enterprise search stats metricset #28287

Merged
merged 7 commits into from
Oct 18, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 54 additions & 9 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
@@ -31086,7 +31086,7 @@ type: keyword
--

[[exported-fields-enterprisesearch]]
== enterprisesearch fields
== Enterprise Search fields
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed we can control this via our module metadata and decided to make it prettier


Enterprise Search module

@@ -31099,21 +31099,21 @@ Enterprise Search module



[float]
=== health

Enterprise Search health


*`enterprisesearch.health.cluster_uuid`*::
*`enterprisesearch.cluster_uuid`*::
+
--
Elasticsearch Cluster UUID used by Enterprise Search
Cluster UUID for the Elasticsearch cluster used as the data store for Enterprise Search.

type: keyword

--

[float]
=== health

Enterprise Search health


*`enterprisesearch.health.name`*::
+
--
@@ -31885,6 +31885,51 @@ type: long

--

[float]
=== product_usage

Aggregate product usage statistics for the Enterprise Search deployment.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a few more fields from our stats endpoint to improve the Stack Monitoring experience



[float]
=== app_search

App Search product usage statistics.


*`enterprisesearch.stats.product_usage.app_search.total_engines`*::
+
--
Current number of App Search engines within the deployment.

type: long

--

[float]
=== workplace_search

Workplace Search product usage statistics.


*`enterprisesearch.stats.product_usage.workplace_search.total_org_sources`*::
+
--
Current number of Workplace Search org-wide content sources within the deployment.

type: long

--

*`enterprisesearch.stats.product_usage.workplace_search.total_private_sources`*::
+
--
Current number of Workplace Search private content sources within the deployment.

type: long

--

[[exported-fields-envoyproxy]]
== Envoyproxy fields

4 changes: 2 additions & 2 deletions metricbeat/docs/modules/enterprisesearch.asciidoc
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ This file is generated! See scripts/mage/docs_collector.go

[[metricbeat-module-enterprisesearch]]
[role="xpack"]
== enterprisesearch module
== Enterprise Search module

beta[]

@@ -24,7 +24,7 @@ The Enterprise Search module requires a set of credentials (a username and a pas
[float]
=== Example configuration

The enterprisesearch module supports the standard configuration options that are described
The Enterprise Search module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
2 changes: 1 addition & 1 deletion metricbeat/docs/modules/enterprisesearch/health.asciidoc
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ This file is generated! See scripts/mage/docs_collector.go

[[metricbeat-metricset-enterprisesearch-health]]
[role="xpack"]
=== enterprisesearch health metricset
=== Enterprise Search health metricset

beta[]

2 changes: 1 addition & 1 deletion metricbeat/docs/modules/enterprisesearch/stats.asciidoc
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ This file is generated! See scripts/mage/docs_collector.go

[[metricbeat-metricset-enterprisesearch-stats]]
[role="xpack"]
=== enterprisesearch stats metricset
=== Enterprise Search stats metricset

beta[]

2 changes: 1 addition & 1 deletion metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
@@ -107,7 +107,7 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-metricset-elasticsearch-node_stats,node_stats>>
|<<metricbeat-metricset-elasticsearch-pending_tasks,pending_tasks>>
|<<metricbeat-metricset-elasticsearch-shard,shard>>
|<<metricbeat-module-enterprisesearch,enterprisesearch>> beta[] |image:./images/icon-yes.png[Prebuilt dashboards are available] |
|<<metricbeat-module-enterprisesearch,Enterprise Search>> beta[] |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.2+| .2+| |<<metricbeat-metricset-enterprisesearch-health,health>> beta[]
|<<metricbeat-metricset-enterprisesearch-stats,stats>> beta[]
|<<metricbeat-module-envoyproxy,Envoyproxy>> |image:./images/icon-no.png[No prebuilt dashboards] |
2 changes: 1 addition & 1 deletion x-pack/metricbeat/metricbeat.reference.yml
Original file line number Diff line number Diff line change
@@ -513,7 +513,7 @@ metricbeat.modules:
#xpack.enabled: false
#scope: node

#--------------------------- Enterprisesearch Module ---------------------------
#-------------------------- Enterprise Search Module --------------------------
- module: enterprisesearch
metricsets: ["health", "stats"]
enabled: true
5 changes: 4 additions & 1 deletion x-pack/metricbeat/module/enterprisesearch/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
- key: enterprisesearch
title: "enterprisesearch"
title: "Enterprise Search"
release: beta
settings: ["ssl", "http"]
description: >
@@ -10,3 +10,6 @@
description: >
`enterprisesearch` contains metrics and health information for Enterprise Search
fields:
- name: cluster_uuid
type: keyword
description: Cluster UUID for the Elasticsearch cluster used as the data store for Enterprise Search.
2 changes: 1 addition & 1 deletion x-pack/metricbeat/module/enterprisesearch/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
This is the health metricset of the module enterprisesearch.
This is the `health` metricset of the Enterprise Search module.

This metricset allows users to fetch and ingest Enterprise Search solution health information from the https://www.elastic.co/guide/en/enterprise-search/current/monitoring-apis.html#health-api[Health API].
Original file line number Diff line number Diff line change
@@ -3,10 +3,6 @@
release: beta
description: Enterprise Search health
fields:
- name: cluster_uuid
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this to the module level

type: keyword
description: Elasticsearch Cluster UUID used by Enterprise Search

- name: name
type: keyword
description: Host name for the Enterprise Search node
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[
{
"enterprisesearch": {
"cluster_uuid": "MTD6fXYGTZylTMoW5AG5JA",
"health": {
"cluster_uuid": "MTD6fXYGTZylTMoW5AG5JA",
"crawler": {
"workers": {
"active": 0,
21 changes: 15 additions & 6 deletions x-pack/metricbeat/module/enterprisesearch/health/data.go
Original file line number Diff line number Diff line change
@@ -13,12 +13,12 @@ import (
"github.com/elastic/beats/v7/libbeat/common"
s "github.com/elastic/beats/v7/libbeat/common/schema"
c "github.com/elastic/beats/v7/libbeat/common/schema/mapstriface"
"github.com/elastic/beats/v7/metricbeat/mb"
)

var (
schema = s.Schema{
"cluster_uuid": c.Str("cluster_uuid"), // This is going to be included in 7.16+
"name": c.Str("name"),
"name": c.Str("name"),

"version": c.Dict("version", s.Schema{
"number": c.Str("number"),
@@ -73,14 +73,21 @@ var (
}
)

func eventMapping(input []byte) (common.MapStr, error) {
func eventMapping(report mb.ReporterV2, input []byte) error {
var data map[string]interface{}
err := json.Unmarshal(input, &data)
if err != nil {
return nil, err
return err
}
var errs multierror.Errors

// All events need to have a cluster_uuid to work with Stack Monitoring
event := mb.Event{
ModuleFields: common.MapStr{},
MetricSetFields: common.MapStr{},
}
event.ModuleFields.Put("cluster_uuid", data["cluster_uuid"])

// Collect process info in a form ready for mapping
process := make(map[string]interface{})
process["filebeat"] = data["filebeat"]
@@ -108,10 +115,12 @@ func eventMapping(input []byte) (common.MapStr, error) {
// Set the process info we have collected
data["process"] = process

dataFields, err := schema.Apply(data)
event.MetricSetFields, err = schema.Apply(data)
if err != nil {
errs = append(errs, errors.Wrap(err, "failure to apply health schema"))
} else {
report.Event(event)
}

return dataFields, errs.Err()
return errs.Err()
}
4 changes: 1 addition & 3 deletions x-pack/metricbeat/module/enterprisesearch/health/health.go
Original file line number Diff line number Diff line change
@@ -63,12 +63,10 @@ func (m *MetricSet) Fetch(report mb.ReporterV2) error {
return errors.Wrap(err, "error in fetch")
}

event, err := eventMapping(content)
err = eventMapping(report, content)
if err != nil {
return errors.Wrap(err, "error converting event")
}

report.Event(mb.Event{MetricSetFields: event})

return nil
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
This is the stats metricset of the module enterprisesearch.
This is the `stats` metricset of the Enterprise Search module.

This metricset allows users to fetch and ingest Enterprise Search solution statistics information from the https://www.elastic.co/guide/en/enterprise-search/current/monitoring-apis.html#stats-api[Stats API].
30 changes: 27 additions & 3 deletions x-pack/metricbeat/module/enterprisesearch/stats/_meta/fields.yml
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@
- name: working
type: long
description: Number of connectors jobs currently being processed.

- name: job_types
type: group
description: Breakdown of connectors jobs by types.
@@ -218,15 +218,39 @@
- name: engine_destroyer.count
type: long
description: Total number of jobs processed via the engine_destroyer queue since the start of the instance.

- name: mailer.count
type: long
description: Total number of jobs processed via the mailer queue since the start of the instance.

- name: process_crawl.count
type: long
description: Total number of jobs processed via the process_crawl queue since the start of the instance.

- name: failed.count
type: long
description: Total number of jobs waiting in the failed queue.

- name: product_usage
type: group
description: Aggregate product usage statistics for the Enterprise Search deployment.
fields:
- name: app_search
type: group
description: App Search product usage statistics.
fields:
- name: total_engines
type: long
description: Current number of App Search engines within the deployment.

- name: workplace_search
type: group
description: Workplace Search product usage statistics.
fields:
- name: total_org_sources
type: long
description: Current number of Workplace Search org-wide content sources within the deployment.

- name: total_private_sources
type: long
description: Current number of Workplace Search private content sources within the deployment.
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{
"cluster_uuid": "s7QHFAuAQmqeqnGZmAqXIA",
"http": {
"connections": {
"current": 1,
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[
{
"enterprisesearch": {
"cluster_uuid": "s7QHFAuAQmqeqnGZmAqXIA",
"stats": {
"connectors": {
"job_store": {
@@ -75,6 +76,15 @@
"5xx": 5
}
},
"product_usage": {
"app_search": {
"total_engines": 1
},
"workplace_search": {
"total_org_sources": 2,
"total_private_sources": 3
}
},
"queues": {
"engine_destroyer": {
"count": 1
Loading