[Metricbeat] Add PubSub metricset to Google Cloud Platform module (#1…

…5536) (#17264) (cherry picked from commit 246225b) Co-authored-by: Mario Castro <[email protected]>
elastic · Mar 26, 2020 · 6d6264b · 6d6264b
1 parent 19c141a
commit 6d6264b
Show file tree

Hide file tree

Showing 15 changed files with 269 additions and 4 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -350,6 +350,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Release Oracle module as GA. {issue}14279[14279] {pull}16833[16833]
 - Add Storage metricsets to GCP module {pull}15598[15598]
 - Release vsphere module as GA. {issue}15798[15798] {pull}17119[17119]
+- Add PubSub metricset to Google Cloud Platform module {pull}15536[15536]
 
 *Packetbeat*
 

diff --git a/metricbeat/docs/modules/googlecloud.asciidoc b/metricbeat/docs/modules/googlecloud.asciidoc
@@ -44,7 +44,7 @@ Authentication and authorization in Google Cloud Platform can be achieved in man
 }
 ----
 
-Generally, you have to create a Service Account and assign it the following roles or the permissions described on each role:
+Generally, you have to create a Service Account and assign it the following roles or the permissions described on each role (applies to all metricsets):
 
 * `Compute Viewer` (1 permission in use of 138):
 - `monitoring.metricDescriptors.list`
@@ -104,6 +104,7 @@ metricbeat.modules:
     - pubsub
     - loadbalancing
   zone: "us-central1-a"
+  region: "us-central1"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
@@ -128,11 +129,15 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>>
 
+* <<metricbeat-metricset-googlecloud-pubsub,pubsub>>
+
 * <<metricbeat-metricset-googlecloud-storage,storage>>
 
 include::googlecloud/compute.asciidoc[]
 
 include::googlecloud/loadbalancing.asciidoc[]
 
+include::googlecloud/pubsub.asciidoc[]
+
 include::googlecloud/storage.asciidoc[]
 
diff --git a/metricbeat/docs/modules/googlecloud/pubsub.asciidoc b/metricbeat/docs/modules/googlecloud/pubsub.asciidoc
@@ -0,0 +1,18 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-googlecloud-pubsub]]
+=== Google Cloud Platform pubsub metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-googlecloud,exported fields>> section.
+
diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc
@@ -109,8 +109,9 @@ This file is generated! See scripts/mage/docs_collector.go
 .2+| .2+|  |<<metricbeat-metricset-golang-expvar,expvar>>   
 |<<metricbeat-metricset-golang-heap,heap>>   
 |<<metricbeat-module-googlecloud,Google Cloud Platform>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.3+| .3+|  |<<metricbeat-metricset-googlecloud-compute,compute>> beta[]  
+.4+| .4+|  |<<metricbeat-metricset-googlecloud-compute,compute>> beta[]  
 |<<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>> beta[]  
+|<<metricbeat-metricset-googlecloud-pubsub,pubsub>> beta[]  
 |<<metricbeat-metricset-googlecloud-storage,storage>> beta[]  
 |<<metricbeat-module-graphite,Graphite>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-graphite-server,server>>   

diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml
@@ -487,6 +487,7 @@ metricbeat.modules:
     - pubsub
     - loadbalancing
   zone: "us-central1-a"
+  region: "us-central1"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false

diff --git a/x-pack/metricbeat/module/googlecloud/_meta/config.yml b/x-pack/metricbeat/module/googlecloud/_meta/config.yml
@@ -4,6 +4,7 @@
     - pubsub
     - loadbalancing
   zone: "us-central1-a"
+  region: "us-central1"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false

diff --git a/x-pack/metricbeat/module/googlecloud/_meta/docs.asciidoc b/x-pack/metricbeat/module/googlecloud/_meta/docs.asciidoc
@@ -34,7 +34,7 @@ Authentication and authorization in Google Cloud Platform can be achieved in man
 }
 ----
 
-Generally, you have to create a Service Account and assign it the following roles or the permissions described on each role:
+Generally, you have to create a Service Account and assign it the following roles or the permissions described on each role (applies to all metricsets):
 
 * `Compute Viewer` (1 permission in use of 138):
 - `monitoring.metricDescriptors.list`

diff --git a/x-pack/metricbeat/module/googlecloud/fields.go b/x-pack/metricbeat/module/googlecloud/fields.go
diff --git a/x-pack/metricbeat/module/googlecloud/module.yml b/x-pack/metricbeat/module/googlecloud/module.yml
@@ -1,5 +1,6 @@
 name: googlecloud
 metricsets:
 - compute
+- pubsub
 - loadbalancing
 - storage
diff --git a/x-pack/metricbeat/module/googlecloud/pubsub/_meta/data_subscription.json b/x-pack/metricbeat/module/googlecloud/pubsub/_meta/data_subscription.json
@@ -0,0 +1,53 @@
+{
+    "@timestamp": "2020-01-14T11:46:00.000Z",
+    "@metadata": {
+        "beat": "metricbeat",
+        "type": "_doc",
+        "version": "8.0.0"
+    },
+    "googlecloud": {
+        "labels": {
+            "resource": {
+                "subscription_id": "test-subscription"
+            }
+        },
+        "pubsub": {
+            "subscription": {
+                "num_undelivered_messages": 0,
+                "oldest_unacked_message_age": 0,
+                "backlog_bytes": 0
+            }
+        }
+    },
+    "service": {
+        "type": "googlecloud"
+    },
+    "cloud": {
+        "account": {
+            "id": "elastic-metricbeat"
+        },
+        "provider": "googlecloud"
+    },
+    "host": {
+        "name": "mcastro"
+    },
+    "agent": {
+        "version": "8.0.0",
+        "type": "metricbeat",
+        "ephemeral_id": "b3b8d9da-0094-4d54-b1dc-4d5e103d38aa",
+        "hostname": "mcastro",
+        "id": "7e36a073-1a32-4a94-b65b-4c7f971fb228"
+    },
+    "ecs": {
+        "version": "1.4.0"
+    },
+    "event": {
+        "dataset": "googlecloud.pubsub",
+        "module": "googlecloud",
+        "duration": 933264720
+    },
+    "metricset": {
+        "name": "pubsub",
+        "period": 300000
+    }
+}
diff --git a/x-pack/metricbeat/module/googlecloud/pubsub/_meta/data_topic.json b/x-pack/metricbeat/module/googlecloud/pubsub/_meta/data_topic.json
@@ -0,0 +1,61 @@
+{
+    "@timestamp": "2020-01-14T11:42:00.000Z",
+    "@metadata": {
+        "beat": "metricbeat",
+        "type": "_doc",
+        "version": "8.0.0"
+    },
+    "ecs": {
+        "version": "1.4.0"
+    },
+    "cloud": {
+        "provider": "googlecloud",
+        "account": {
+            "id": "elastic-metricbeat"
+        }
+    },
+    "metricset": {
+        "name": "pubsub",
+        "period": 300000
+    },
+    "event": {
+        "duration": 932060852,
+        "dataset": "googlecloud.pubsub",
+        "module": "googlecloud"
+    },
+    "googlecloud": {
+        "labels": {
+            "resource": {
+                "subscription_id": "test-subscription"
+            }
+        },
+        "pubsub": {
+            "topic": {
+                "message_sizes": {
+                    "bucket_options": {
+                        "Options": {
+                            "ExponentialBuckets": {
+                                "growth_factor": 4,
+                                "scale": 1,
+                                "num_finite_buckets": 16
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "service": {
+        "type": "googlecloud"
+    },
+    "host": {
+        "name": "mcastro"
+    },
+    "agent": {
+        "ephemeral_id": "b3b8d9da-0094-4d54-b1dc-4d5e103d38aa",
+        "hostname": "mcastro",
+        "id": "7e36a073-1a32-4a94-b65b-4c7f971fb228",
+        "version": "8.0.0",
+        "type": "metricbeat"
+    }
+}
diff --git a/x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc b/x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc
@@ -0,0 +1,25 @@
+PubSub Metricset to fetch metrics from https://cloud.google.com/pubsub/[Pub/Sub] topics and subscriptions in Google Cloud Platform.
+
+The `pubsub` Metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[Stackdriver API]. The field names have been left untouched for people already familiar with them.
+
+No special permissions are needed apart from the ones detailed in the module section of the docs.
+
+- `pubsub.subscription.ack_message_count`: Cumulative count of messages acknowledged by Acknowledge requests, grouped by delivery type.
+- `pubsub.subscription.backlog_bytes`: Total byte size of the unacknowledged messages (a.k.a. backlog messages) in a subscription.
+- `pubsub.subscription.num_outstanding_messages`: Number of messages delivered to a subscription's push endpoint, but not yet acknowledged.
+- `pubsub.subscription.num_undelivered_messages`: Number of unacknowledged messages (a.k.a. backlog messages) in a subscription.
+- `pubsub.subscription.oldest_unacked_message_age`: Age (in seconds) of the oldest unacknowledged message (a.k.a. backlog message) in a subscription.
+- `pubsub.subscription.pull_ack_message_operation_count`: Cumulative count of acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
+- `pubsub.subscription.pull_ack_request_count`: Cumulative count of acknowledge requests, grouped by result.
+- `pubsub.subscription.pull_message_operation_count`: Cumulative count of pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
+- `pubsub.subscription.pull_request_count`: Cumulative count of pull requests, grouped by result.
+- `pubsub.subscription.push_request_count`: Cumulative count of push attempts, grouped by result. Unlike pulls, the push server implementation does not batch user messages. So each request only contains one user message. The push server retries on errors, so a given user message can appear multiple times.
+- `pubsub.subscription.push_request_latencies`: Distribution of push request latencies (in microseconds), grouped by result.
+- `pubsub.subscription.sent_message_count`: Cumulative count of messages sent by Cloud Pub/Sub to subscriber clients, grouped by delivery type.
+- `pubsub.subscription.streaming_pull_ack_message_operation_count`: Cumulative count of StreamingPull acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
+- `pubsub.subscription.streaming_pull_ack_request_count`: Cumulative count of streaming pull requests with non-empty acknowledge ids, grouped by result.
+- `pubsub.subscription.streaming_pull_message_operation_count`: Cumulative count of streaming pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric <code>subscription/mod_ack_deadline_message_operation_count
+- `pubsub.subscription.streaming_pull_response_count`: Cumulative count of streaming pull responses, grouped by result.
+- `pubsub.topic.streaming_pull_response_count`: Cumulative count of streaming pull responses, grouped by result.
+- `pubsub.topic.send_message_operation_count`: Cumulative count of publish message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
+- `pubsub.topic.send_request_count`: Cumulative count of publish requests, grouped by result.