Skip to content

Commit

Permalink
[Metricbeat] remove distribution type metrics from googlecloud (elast…
Browse files Browse the repository at this point in the history
…ic#17946)

* remove distribution type metrics from googlecloud
  • Loading branch information
kaiyan-sheng authored Apr 24, 2020
1 parent 98f02e1 commit 5e18850
Show file tree
Hide file tree
Showing 6 changed files with 43 additions and 28 deletions.
23 changes: 20 additions & 3 deletions x-pack/metricbeat/module/googlecloud/compute/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -1,26 +1,37 @@
Compute Metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine] Virtual Machines in Google Cloud Platform. No Monitoring or Logging agent is required in your instances to use this Metricset.
Compute metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine] Virtual Machines in Google Cloud Platform. No Monitoring or Logging agent is required in your instances to use this metricset.

The `compute` Metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API]. The field names have been left untouched for people already familiar with them.
The `compute` metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API]. The field names have been left untouched for people already familiar with them.

Extra labels and metadata are also extracted using the https://cloud.google.com/compute/docs/reference/rest/v1/instances/get[Compute API]. This is enough to get most of the info associated with a metric like Compute labels and metadata and metric specific Labels.

[float]
=== Fields and labels
=== Metrics and labels
Here is a list of metrics collected by `compute` metricset:

[float]
==== firewall

* `instance.firewall.dropped_bytes_count`: Incoming bytes dropped by the firewall.
- `instance_name`: The name of the VM instance.

* `instance.firewall.dropped_packets_count`: Incoming packets dropped by the firewall.
- `instance_name`: The name of the VM instance.

[float]
==== cpu

* `instance.cpu.reserved_cores`: Number of cores reserved on the host of the `instance`.
- `instance_name`: The name of the VM instance.

* `instance.cpu.utilization`: The fraction of the allocated CPU that is currently in use on the `instance`.
- `instance_name`: The name of the VM instance.

* `instance.cpu.usage_time`: Usage for all cores in seconds.
- `instance_name`: The name of the VM instance.

[float]
==== disk

* `instance.disk.read_bytes_count`: Count of bytes read from disk.
- `instance_name`: The name of the VM instance.
- `device_name`: The name of the disk device.
Expand All @@ -45,9 +56,15 @@ Extra labels and metadata are also extracted using the https://cloud.google.com/
- `storage_type`: The storage type: `pd-standard`, `pd-ssd`, or `local-ssd`.
- `device_type`: The disk type: `ephemeral` or `permanent`.

[float]
==== uptime

* `instance.uptime`: How long the VM has been running, in seconds
- `instance_name`: The name of the VM instance.

[float]
==== network

* `instance.network.received_bytes_count`: Count of bytes received from the network
- `instance_name`: The name of the VM instance.
- `loadBalanced`: Whether traffic was sent from an L3 loadbalanced IP address assigned to the VM. Traffic that is externally routed from the VM's standard internal or external IP address, such as L7 loadbalanced traffic, is not considered to be loadbalanced in this metric.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,32 @@ Load Balancing metricset to fetch metrics from https://cloud.google.com/load-bal
The `loadbalancing` metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing[Stackdriver API]. The field names have been left untouched for people already familiar with them.

[float]
=== Fields
=== Metrics
Here is a list of metrics collected by `loadbalancing` metricset:

[float]
==== https

- `loadbalancing.https.backend_latencies`: A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.
- `loadbalancing.https.backend_request_bytes_count`: The number of bytes sent as requests from HTTP/S load balancer to backends.
- `loadbalancing.https.backend_request_count`: The number of requests served by backends of HTTP/S load balancer.
- `loadbalancing.https.backend_response_bytes_count`: The number of bytes sent as responses from backends (or cache) to HTTP/S load balancer.
- `loadbalancing.https.frontend_tcp_rtt`: A distribution of the RTT measured for each connection between client and proxy.
- `loadbalancing.https.request_bytes_count`: The number of bytes sent as requests from clients to HTTP/S load balancer.
- `loadbalancing.https.request_count`: The number of requests served by HTTP/S load balancer.
- `loadbalancing.https.response_bytes_count`: The number of bytes sent as responses from HTTP/S load balancer to clients.
- `loadbalancing.https.total_latencies`: A distribution of the latency calculated from when the request was received by the proxy until the proxy got ACK from client on last response byte.

[float]
==== l3

- `loadbalancing.l3.internal.egress_bytes_count`: The number of bytes sent from ILB backend to client (for TCP flows it's counting bytes on application stream only).
- `loadbalancing.l3.internal.egress_packets_count`: The number of packets sent from ILB backend to client of the flow.
- `loadbalancing.l3.internal.ingress_bytes_count`: The number of bytes sent from client to ILB backend (for TCP flows it's counting bytes on application stream only).
- `loadbalancing.l3.internal.ingress_packets_count`: The number of packets sent from client to ILB backend.
- `loadbalancing.l3.internal.rtt_latencies`: A distribution of RTT measured over TCP connections for ILB flows.

[float]
==== tcp_ssl_proxy

- `loadbalancing.tcp_ssl_proxy.closed_connections`: Number of connections that were terminated over TCP/SSL proxy.
- `loadbalancing.tcp_ssl_proxy.egress_bytes_count`: Number of bytes sent from VM to client using proxy.
- `loadbalancing.tcp_ssl_proxy.frontend_tcp_rtt`: A distribution of the smoothed RTT (in ms) measured by the proxy's TCP stack, each minute application layer bytes pass from proxy to client.
- `loadbalancing.tcp_ssl_proxy.ingress_bytes_count`: Number of bytes sent from client to VM using proxy.
- `loadbalancing.tcp_ssl_proxy.new_connections`: Number of connections that were created over TCP/SSL proxy.
- `loadbalancing.tcp_ssl_proxy.open_connections`: Current number of outstanding connections through the TCP/SSL proxy.
10 changes: 0 additions & 10 deletions x-pack/metricbeat/module/googlecloud/loadbalancing/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,18 @@ input:
service: loadbalancing
metrics:
- metric_types:
- "loadbalancing.googleapis.com/https/backend_latencies"
- "loadbalancing.googleapis.com/https/backend_latencies"
- "loadbalancing.googleapis.com/https/backend_request_bytes_count"
- "loadbalancing.googleapis.com/https/backend_request_count"
- "loadbalancing.googleapis.com/https/backend_response_bytes_count"
- "loadbalancing.googleapis.com/https/frontend_tcp_rtt"
- "loadbalancing.googleapis.com/https/request_bytes_count"
- "loadbalancing.googleapis.com/https/request_bytes_count"
- "loadbalancing.googleapis.com/https/request_count"
- "loadbalancing.googleapis.com/https/request_count"
- "loadbalancing.googleapis.com/https/response_bytes_count"
- "loadbalancing.googleapis.com/https/response_bytes_count"
- "loadbalancing.googleapis.com/https/total_latencies"
- "loadbalancing.googleapis.com/https/total_latencies"
- "loadbalancing.googleapis.com/l3/internal/egress_bytes_count"
- "loadbalancing.googleapis.com/l3/internal/egress_packets_count"
- "loadbalancing.googleapis.com/l3/internal/ingress_bytes_count"
- "loadbalancing.googleapis.com/l3/internal/ingress_packets_count"
- "loadbalancing.googleapis.com/l3/internal/rtt_latencies"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/closed_connections"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/egress_bytes_count"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/frontend_tcp_rtt"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/ingress_bytes_count"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/new_connections"
- "loadbalancing.googleapis.com/tcp_ssl_proxy/open_connections"
10 changes: 6 additions & 4 deletions x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
PubSub Metricset to fetch metrics from https://cloud.google.com/pubsub/[Pub/Sub] topics and subscriptions in Google Cloud Platform.
PubSub metricsetf to fetch metrics from https://cloud.google.com/pubsub/[Pub/Sub] topics and subscriptions in Google Cloud Platform.

The `pubsub` Metricset contains all GA stage metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[Stackdriver API]. The field names have been left untouched for people already familiar with them.
The `pubsub` metricset contains all GA stage metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[Stackdriver API]. The field names have been left untouched for people already familiar with them.

No special permissions are needed apart from the ones detailed in the module section of the docs.

[float]
=== Metrics
Here is a list of metrics collected by `pubsub` metricset:

[float]
==== Snapshot Metrics
- `pubsub.snapshot.backlog_bytes`: Total byte size of the messages retained in a snapshot.
Expand Down Expand Up @@ -35,7 +39,6 @@ No special permissions are needed apart from the ones detailed in the module sec
- `pubsub.subscription.pull_message_operation_count`: Cumulative count of pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
- `pubsub.subscription.pull_request_count`: Cumulative count of pull requests, grouped by result.
- `pubsub.subscription.push_request_count`: Cumulative count of push attempts, grouped by result. Unlike pulls, the push server implementation does not batch user messages. So each request only contains one user message. The push server retries on errors, so a given user message can appear multiple times.
- `pubsub.subscription.push_request_latencies`: Distribution of push request latencies (in microseconds), grouped by result.
- `pubsub.subscription.retained_acked_bytes`: otal byte size of the acknowledged messages retained in a subscription.
- `pubsub.subscription.retained_acked_bytes_by_region`: Total byte size of the acknowledged messages retained in a subscription, broken down by Cloud region.
- `pubsub.subscription.seek_request_count`: Cumulative count of seek attempts, grouped by result.
Expand All @@ -52,7 +55,6 @@ No special permissions are needed apart from the ones detailed in the module sec
==== Topic Metrics
- `pubsub.topic.byte_cost`: Cost of operations, measured in bytes. This is used to measure utilization for quotas.
- `pubsub.topic.config_updates_count`: Cumulative count of configuration changes, grouped by operation type and result.
- `pubsub.topic.message_sizes`: Distribution of publish message sizes (in bytes).
- `pubsub.topic.oldest_retained_acked_message_age_by_region`: Age (in seconds) of the oldest acknowledged message retained in a topic, broken down by Cloud region.
- `pubsub.topic.oldest_unacked_message_age_by_region`: Age (in seconds) of the oldest unacknowledged message in a topic, broken down by Cloud region.
- `pubsub.topic.retained_acked_bytes_by_region`: Total byte size of the acknowledged messages retained in a topic, broken down by Cloud region.
Expand Down
2 changes: 0 additions & 2 deletions x-pack/metricbeat/module/googlecloud/pubsub/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ input:
- "pubsub.googleapis.com/subscription/pull_message_operation_count"
- "pubsub.googleapis.com/subscription/pull_request_count"
- "pubsub.googleapis.com/subscription/push_request_count"
- "pubsub.googleapis.com/subscription/push_request_latencies"
- "pubsub.googleapis.com/subscription/retained_acked_bytes"
- "pubsub.googleapis.com/subscription/retained_acked_bytes_by_region"
- "pubsub.googleapis.com/subscription/seek_request_count"
Expand All @@ -47,7 +46,6 @@ input:
- "pubsub.googleapis.com/subscription/unacked_bytes_by_region"
- "pubsub.googleapis.com/topic/byte_cost"
- "pubsub.googleapis.com/topic/config_updates_count"
- "pubsub.googleapis.com/topic/message_sizes"
- "pubsub.googleapis.com/topic/oldest_retained_acked_message_age_by_region"
- "pubsub.googleapis.com/topic/oldest_unacked_message_age_by_region"
- "pubsub.googleapis.com/topic/retained_acked_bytes_by_region"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
Storage Metricset to fetch metrics from https://cloud.google.com/storage/[Storage] in Google Cloud Platform.
Storage metricset to fetch metrics from https://cloud.google.com/storage/[Storage] in Google Cloud Platform.

The `storage` Metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-storage[Stackdriver API]. The field names have been left untouched for people already familiar with them.
The `storage` metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-storage[Stackdriver API]. The field names have been left untouched for people already familiar with them.

You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Storage does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets.

[float]
=== Fields
=== Metrics
Here is a list of metrics collected by `storage` metricset:

- `storage.api.request_count`: Delta count of API calls, grouped by the API method name and response code.
- `storage.authz.acl_based_object_access_count`: Delta count of requests that result in an object being granted access solely due to object ACLs.
Expand Down

0 comments on commit 5e18850

Please sign in to comment.