Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Add Google Cloud Platform module #14829

Merged
merged 59 commits into from
Jan 8, 2020
Merged

[Metricbeat] Add Google Cloud Platform module #14829

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
8d9ed54
Atomic commit
sayden Oct 17, 2019
9a5e4e3
add vendor
sayden Nov 13, 2019
a4969cd
Atomic commit
sayden Nov 27, 2019
1c787b1
Add google.golang.org/api/googleapi to vendoring
sayden Nov 28, 2019
410b901
Added google.golang.org/api/internal/third_party/uritemplates library
sayden Nov 28, 2019
b562f27
Added field.yml for compute and update docs
sayden Nov 28, 2019
57f956a
Add google.golang.org/genproto/googleapis/api to vendor
sayden Nov 29, 2019
e01a700
Add google.golang.org/genproto/googleapis/api/distribution to vendor
sayden Nov 29, 2019
39f9ed9
Add google.golang.org/genproto/googleapis/api/label to vendor
sayden Nov 29, 2019
af4dd18
Add google.golang.org/genproto/googleapis/type/calendarperiod to vendor
sayden Nov 29, 2019
f17f4c6
Run go vet
sayden Nov 29, 2019
3f33340
Fix unnamed variable in test
sayden Nov 29, 2019
13a00ed
Update notice
sayden Nov 29, 2019
94c2b95
Add context to the Metrics Requester
sayden Dec 11, 2019
87435f5
Separate groups of ECS fields
sayden Dec 11, 2019
c84bc76
Add machine type to the event output
sayden Dec 11, 2019
e12112f
Update vendoring with fixed version of cloud monitoring api (no changes)
sayden Dec 11, 2019
8573777
Fix version of compute google.golang.org/api/compute/v1 and update ve…
sayden Dec 11, 2019
25d1e14
Fixed version for google.golang.org/api/googleapi and update vendor f…
sayden Dec 11, 2019
906ceb4
Fix vendoring for package google.golang.org/api/internal/gensupport t…
sayden Dec 11, 2019
915bf18
Fix package google.golang.org/api/internal/third_party/uritemplates t…
sayden Dec 11, 2019
48f7ba2
Revert google.golang.org/api/googleapi to version 0.7.0 and update ve…
sayden Dec 11, 2019
154108e
Fix typo
sayden Dec 12, 2019
79d4955
Change contants to lowercase
sayden Dec 13, 2019
3191352
Changes after reviews
sayden Dec 16, 2019
33894e8
Rename some constants
sayden Dec 17, 2019
d5f93c4
Fix a couple of comments
sayden Dec 17, 2019
8d93a3d
One last comment
sayden Dec 17, 2019
24ce9c8
Address comments
sayden Dec 18, 2019
55fdb51
Merge branch 'master' into feature/xp/mb/gcp/stackdriver-compute
sayden Dec 18, 2019
d57231a
Fix behavior of exclude labels
sayden Dec 19, 2019
03d9c28
Remove services of 2nd phase
sayden Dec 19, 2019
6ad3133
Add missing comment
sayden Dec 19, 2019
8a079a0
Update notice file
sayden Dec 19, 2019
44cf435
Remove future metricsets
sayden Dec 19, 2019
6a3accf
Remove unused metricsets
sayden Dec 19, 2019
7530216
WIP add docs
sayden Dec 20, 2019
d12ae42
Fix use of HTTP client instead of gRPC
sayden Dec 31, 2019
c8a974e
Update docs and metadata
sayden Dec 31, 2019
a3e6f6a
Merge branch 'master' into feature/xp/mb/gcp/stackdriver-compute
sayden Dec 31, 2019
5fdf780
Remove storage metricset and stackdriver docs
sayden Dec 31, 2019
68b906c
Add dynamic mapping fields?
sayden Dec 31, 2019
242262c
Potential docs fix
sayden Dec 31, 2019
0c685b7
Fix incorrect vendor merge
sayden Dec 31, 2019
f61f051
Update some docs (without collect?)
sayden Jan 6, 2020
fd455c9
Fix indentation in vendor file
sayden Jan 8, 2020
bf5f06e
Store only one option and save it in New
sayden Jan 8, 2020
9edadd7
Fix and rephrasing things in docs
sayden Jan 8, 2020
f12fdc7
Regenerate docs and pass fmt
sayden Jan 8, 2020
45f33e6
Add period recommendations
sayden Jan 8, 2020
c96397a
Removed spaces from indendation
sayden Jan 8, 2020
b83ff57
Updated docs.
sayden Jan 8, 2020
5ed657f
Remove data.json from stackdriver. Update the one from compute
sayden Jan 8, 2020
242ace3
Split data.json
sayden Jan 8, 2020
71d9cff
Remove markdown from asciidocs
sayden Jan 8, 2020
07fda7c
Beautify machine type data
sayden Jan 8, 2020
5a3cac1
Merge branch 'master' into feature/xp/mb/gcp/stackdriver-compute
sayden Jan 8, 2020
edff0e9
Set the reference region to a known one by default
sayden Jan 8, 2020
6e69878
Update docs
sayden Jan 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions NOTICE.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5629,8 +5629,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------
Dependency: google.golang.org/api
Version: v0.7.0
jsoriano marked this conversation as resolved.
Show resolved Hide resolved
Revision: 02490b97dff7cfde1995bd77de808fd27053bc87
Revision: 006ab4afe25e899e4ec545ce1cbfdaed675d15d8
License type (autodetected): BSD-3-Clause
./vendor/google.golang.org/api/LICENSE:
--------------------------------------------------------------------
Expand Down Expand Up @@ -5662,6 +5661,40 @@ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------
Dependency: google.golang.org/api/internal/third_party/uritemplates
Revision: 006ab4afe25e899e4ec545ce1cbfdaed675d15d8
License type (autodetected): BSD-3-Clause
./vendor/google.golang.org/api/internal/third_party/uritemplates/LICENSE:
--------------------------------------------------------------------
Copyright (c) 2013 Joshua Tacoma. All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------------------
Dependency: google.golang.org/appengine
Version: v1.6.1
Expand All @@ -5683,7 +5716,7 @@ Apache License 2.0

--------------------------------------------------------------------
Dependency: google.golang.org/genproto
Revision: a7e196e89fd3a3c4d103ca540bd5dac3a736e375
Revision: 83cc0476cb11ea0da33dacd4c6354ab192de6fe6
License type (autodetected): Apache-2.0
./vendor/google.golang.org/genproto/LICENSE:
--------------------------------------------------------------------
Expand Down
75 changes: 75 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ grouped in the following categories:
* <<exported-fields-elasticsearch>>
* <<exported-fields-envoyproxy>>
* <<exported-fields-etcd>>
* <<exported-fields-gcp>>
* <<exported-fields-golang>>
* <<exported-fields-graphite>>
* <<exported-fields-haproxy>>
Expand Down Expand Up @@ -13613,6 +13614,80 @@ type: integer

--

[[exported-fields-gcp]]
== Google Cloud Platform fields

GCP module
- name: compute
release: beta
type: group
description: Google Cloud Compute metrics
fields:
- name: instance
type: group
fields:
- name: firewall
type: group
fields:
- name: dropped_bytes_count
type: long
description: Incoming bytes dropped by the firewall
- name: dropped_packets_count
type: long
description: Incoming packets dropped by the firewall
- name: cpu
type: group
fields:
- name: reserved_cores
type: double
description: Number of cores reserved on the host of the instance
- name: utilization
type: double
description: The fraction of the allocated CPU that is currently in use on the instance
- name: usage_time
type: double
description: Usage for all cores in seconds
- name: disk
type: group
fields:
- name: read_bytes_count
type: long
description: Count of bytes read from disk
- name: read_ops_count
type: long
description: Count of disk read IO operations
- name: write_bytes_count
type: long
description: Count of bytes written to disk
- name: write_ops_count
type: long
description: Count of disk write IO operations
- name: uptime
type: long
description: How long the VM has been running, in seconds
- name: network
type: group
fields:
- name: received_bytes_count
type: long
description: Count of bytes received from the network
- name: received_packets_count
type: long
description: Count of packets received from the network
- name: sent_bytes_count
type: long
description: Count of bytes sent over the network
- name: sent_packets_count
type: long
description: Count of packets sent over the network
- release: beta
- release: beta
- release: beta
- release: beta
- release: beta
- release: beta


[[exported-fields-golang]]
== Golang fields

Expand Down
72 changes: 72 additions & 0 deletions metricbeat/docs/modules/googlecloud.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-module-googlecloud]]
[role="xpack"]
== Google Cloud Platform module

beta[]

This module periodically fetches monitoring metrics from AWS Stackdriver using
https://cloud.google.com/monitoring/api/metrics_gcp[Stackdriver Monitoring API] for Google Cloud Platform services.
Note: extra GCP charges on Stackdriver Monitoring API requests will be generated by this module.


[float]
=== Example configuration

The Google Cloud Platform module supports the standard configuration options that are described
in <<configuration-metricbeat>>. Here is an example configuration:

[source,yaml]
----
metricbeat.modules:
- module: googlecloud
metricsets:
- compute
- pubsub
- firestore
- firebase
- storage
- loadbalancing
zone: "your zone"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we specify more than one zone here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. We can't specify it right now. The idea was to maintain first version as simple as possible. It's actually possible to request all metrics for a project without zone filter or even request various zones but we are moving slow yet and see how it goes because the code to request metrics and convert them using lightweight modules is pretty complex already.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps it would be interesting to put a real zone here so things won't fail if they start the module out of the box?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer that things do fail explicitly so that a user with machines in Europe that runs Metricbeat will have an specific error saying zone "your zone" not found instead of silent errors of simply not sending any event because there are no machines in that zone/region which may lead to think that Metricbeat is not working properly (it's your fault because you didn't set the correct zone, but that's implicit)

project_id: "your project id"
credentials_file_path: "your JSON credentials file path"
jsoriano marked this conversation as resolved.
Show resolved Hide resolved
include_labels: true
period: 300s
----

[float]
=== Metricsets

The following metricsets are available:

* <<metricbeat-metricset-googlecloud-compute,compute>>

* <<metricbeat-metricset-googlecloud-firebase,firebase>>

* <<metricbeat-metricset-googlecloud-firestore,firestore>>

* <<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>>

* <<metricbeat-metricset-googlecloud-pubsub,pubsub>>

* <<metricbeat-metricset-googlecloud-stackdriver,stackdriver>>

* <<metricbeat-metricset-googlecloud-storage,storage>>

include::googlecloud/compute.asciidoc[]

include::googlecloud/firebase.asciidoc[]

include::googlecloud/firestore.asciidoc[]

include::googlecloud/loadbalancing.asciidoc[]

include::googlecloud/pubsub.asciidoc[]

include::googlecloud/stackdriver.asciidoc[]

include::googlecloud/storage.asciidoc[]

24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/compute.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-compute]]
=== Google Cloud Platform compute metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/compute/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/compute/_meta/data.json[]
----
24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/firebase.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-firebase]]
=== Google Cloud Platform firebase metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/firebase/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/firebase/_meta/data.json[]
----
24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/firestore.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-firestore]]
=== Google Cloud Platform firestore metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/firestore/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/firestore/_meta/data.json[]
----
24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/loadbalancing.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-loadbalancing]]
=== Google Cloud Platform loadbalancing metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/loadbalancing/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/loadbalancing/_meta/data.json[]
----
24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/pubsub.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-pubsub]]
=== Google Cloud Platform pubsub metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/pubsub/_meta/data.json[]
----
23 changes: 23 additions & 0 deletions metricbeat/docs/modules/googlecloud/stackdriver.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-stackdriver]]
=== Google Cloud Platform stackdriver metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/stackdriver/_meta/docs.asciidoc[]


==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/stackdriver/_meta/data.json[]
----
24 changes: 24 additions & 0 deletions metricbeat/docs/modules/googlecloud/storage.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
////
This file is generated! See scripts/mage/docs_collector.go
////

[[metricbeat-metricset-googlecloud-storage]]
=== Google Cloud Platform storage metricset

beta[]

include::../../../../x-pack/metricbeat/module/googlecloud/storage/_meta/docs.asciidoc[]

This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.

==== Fields

For a description of each field in the metricset, see the
<<exported-fields-googlecloud,exported fields>> section.

Here is an example document generated by this metricset:

[source,json]
----
include::../../../../x-pack/metricbeat/module/googlecloud/storage/_meta/data.json[]
----
Loading