diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 21e53cc0e256..371646aae625 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -263,6 +263,7 @@ automatic splitting at root level, if root level element is an array. {pull}3415 - Changed cloudwatch module to call ListMetrics API only once per region, instead of per AWS namespace {pull}34055[34055] - Add beta ingest_pipeline metricset to Elasticsearch module for ingest pipeline monitoring {pull}34012[34012] - Handle duplicated TYPE line for prometheus metrics {issue}18813[18813] {pull}33865[33865] +- Add GCP Carbon Footprint metricbeat data {pull}34820[34820] *Packetbeat* diff --git a/NOTICE.txt b/NOTICE.txt index 39faebdcc344..27d17a64e347 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -9,6 +9,218 @@ Third party libraries used by the Elastic Beats project: ================================================================================ +-------------------------------------------------------------------------------- +Dependency : cloud.google.com/go +Version: v0.105.0 +Licence type (autodetected): Apache-2.0 +-------------------------------------------------------------------------------- + +Contents of probable licence file $GOMODCACHE/cloud.google.com/go@v0.105.0/LICENSE: + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + -------------------------------------------------------------------------------- Dependency : cloud.google.com/go/bigquery Version: v1.42.0 @@ -24729,218 +24941,6 @@ Public License instead of this License. Indirect dependencies --------------------------------------------------------------------------------- -Dependency : cloud.google.com/go -Version: v0.105.0 -Licence type (autodetected): Apache-2.0 --------------------------------------------------------------------------------- - -Contents of probable licence file $GOMODCACHE/cloud.google.com/go@v0.105.0/LICENSE: - - - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - - -------------------------------------------------------------------------------- Dependency : cloud.google.com/go/compute Version: v1.10.0 diff --git a/go.mod b/go.mod index 0606ac6fc52f..085e643f03bf 100644 --- a/go.mod +++ b/go.mod @@ -184,6 +184,7 @@ require ( ) require ( + cloud.google.com/go v0.105.0 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1 github.com/Azure/go-autorest/autorest/adal v0.9.14 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.17 @@ -213,7 +214,6 @@ require ( ) require ( - cloud.google.com/go v0.105.0 // indirect cloud.google.com/go/compute v1.10.0 // indirect cloud.google.com/go/iam v0.6.0 // indirect code.cloudfoundry.org/gofileutils v0.0.0-20170111115228-4d0c80011a0f // indirect diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc index 48bc0c369545..0b57f00cb42c 100644 --- a/metricbeat/docs/fields.asciidoc +++ b/metricbeat/docs/fields.asciidoc @@ -35358,6 +35358,102 @@ type: float -- +[float] +=== carbon + +Google Cloud Carbon Footprint metrics + + +*`gcp.carbon.project_id`*:: ++ +-- +Project ID the carbon footprint report belongs to. + +type: keyword + +-- + +*`gcp.carbon.project_name`*:: ++ +-- +Project name the carbon footprint report belongs to. + +type: keyword + +-- + +*`gcp.carbon.service_id`*:: ++ +-- +Service ID for the carbon footprint usage. + +type: keyword + +-- + +*`gcp.carbon.service_description`*:: ++ +-- +Service description for the carbon footprint usage. + +type: keyword + +-- + +*`gcp.carbon.region`*:: ++ +-- +Region for the carbon fooprint usage. + +type: keyword + +-- + +*`gcp.carbon.footprint.scope1`*:: ++ +-- +Scope 1 carbon footprint. + +type: float + +-- + +*`gcp.carbon.footprint.scope2.location`*:: ++ +-- +Scope 2 carbon footprint using location-based methodology. + +type: float + +-- + +*`gcp.carbon.footprint.scope2.market`*:: ++ +-- +Scope 2 carbon footprint using market-based methodology. + +type: float + +-- + +*`gcp.carbon.footprint.scope3`*:: ++ +-- +Scope 3 carbon footprint. + +type: float + +-- + +*`gcp.carbon.footprint.offsets`*:: ++ +-- +Total carbon offsets. + +type: float + +-- + [float] === compute diff --git a/metricbeat/docs/modules/gcp.asciidoc b/metricbeat/docs/modules/gcp.asciidoc index 463d18cbf86c..ad2ec8b01a77 100644 --- a/metricbeat/docs/modules/gcp.asciidoc +++ b/metricbeat/docs/modules/gcp.asciidoc @@ -329,6 +329,15 @@ metricbeat.modules: dataset_id: "dataset id" table_pattern: "table pattern" cost_type: "regular" + +- module: gcp + metricsets: + - carbon + period: 24h + project_id: "your project id" + credentials_file_path: "your JSON credentials file path" + dataset_id: "dataset id" + table_pattern: "table pattern" ---- [float] @@ -338,6 +347,8 @@ The following metricsets are available: * <> +* <> + * <> * <> @@ -356,6 +367,8 @@ The following metricsets are available: include::gcp/billing.asciidoc[] +include::gcp/carbon.asciidoc[] + include::gcp/compute.asciidoc[] include::gcp/dataproc.asciidoc[] diff --git a/metricbeat/docs/modules/gcp/carbon.asciidoc b/metricbeat/docs/modules/gcp/carbon.asciidoc new file mode 100644 index 000000000000..1f926fa96ecd --- /dev/null +++ b/metricbeat/docs/modules/gcp/carbon.asciidoc @@ -0,0 +1,29 @@ +//// +This file is generated! See scripts/mage/docs_collector.go +//// +:edit_url: https://github.com/elastic/beats/edit/main/x-pack/metricbeat/module/gcp/carbon/_meta/docs.asciidoc + + +[[metricbeat-metricset-gcp-carbon]] +[role="xpack"] +=== Google Cloud Platform carbon metricset + +beta[] + +include::../../../../x-pack/metricbeat/module/gcp/carbon/_meta/docs.asciidoc[] + + +:edit_url: + +==== Fields + +For a description of each field in the metricset, see the +<> section. + +Here is an example document generated by this metricset: + +[source,json] +---- +include::../../../../x-pack/metricbeat/module/gcp/carbon/_meta/data.json[] +---- +:edit_url!: \ No newline at end of file diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc index 615f056dd37d..42394e4b0641 100644 --- a/metricbeat/docs/modules_list.asciidoc +++ b/metricbeat/docs/modules_list.asciidoc @@ -121,7 +121,8 @@ This file is generated! See scripts/mage/docs_collector.go |<> |<> |<> beta[] |image:./images/icon-yes.png[Prebuilt dashboards are available] | -.9+| .9+| |<> beta[] +.10+| .10+| |<> beta[] +|<> beta[] |<> beta[] |<> beta[] |<> beta[] diff --git a/x-pack/metricbeat/include/list.go b/x-pack/metricbeat/include/list.go index bbd45ada6d3b..0cbedc06dd4a 100644 --- a/x-pack/metricbeat/include/list.go +++ b/x-pack/metricbeat/include/list.go @@ -36,6 +36,7 @@ import ( _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/enterprisesearch/stats" _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp" _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp/billing" + _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp/carbon" _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp/metrics" _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/ibmmq" _ "github.com/elastic/beats/v7/x-pack/metricbeat/module/iis" diff --git a/x-pack/metricbeat/metricbeat.reference.yml b/x-pack/metricbeat/metricbeat.reference.yml index f94dad7e8d27..1c7478a919c1 100644 --- a/x-pack/metricbeat/metricbeat.reference.yml +++ b/x-pack/metricbeat/metricbeat.reference.yml @@ -611,6 +611,15 @@ metricbeat.modules: table_pattern: "table pattern" cost_type: "regular" +- module: gcp + metricsets: + - carbon + period: 24h + project_id: "your project id" + credentials_file_path: "your JSON credentials file path" + dataset_id: "dataset id" + table_pattern: "table pattern" + #-------------------------------- Golang Module -------------------------------- - module: golang #metricsets: diff --git a/x-pack/metricbeat/module/gcp/_meta/config.yml b/x-pack/metricbeat/module/gcp/_meta/config.yml index f81e6d38707f..f4e245080aeb 100644 --- a/x-pack/metricbeat/module/gcp/_meta/config.yml +++ b/x-pack/metricbeat/module/gcp/_meta/config.yml @@ -60,3 +60,12 @@ dataset_id: "dataset id" table_pattern: "table pattern" cost_type: "regular" + +- module: gcp + metricsets: + - carbon + period: 24h + project_id: "your project id" + credentials_file_path: "your JSON credentials file path" + dataset_id: "dataset id" + table_pattern: "table pattern" diff --git a/x-pack/metricbeat/module/gcp/carbon/_meta/data.json b/x-pack/metricbeat/module/gcp/carbon/_meta/data.json new file mode 100644 index 000000000000..c1486b099d21 --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/_meta/data.json @@ -0,0 +1,32 @@ +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud.account.id": "01475F-5B1080-1137E7", + "cloud.project.id": "elastic-bi", + "cloud.project.name": "elastic-containerlib-prod", + "cloud.provider": "gcp", + "event": { + "dataset": "gcp.carbon", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "carbon": { + "project_id": "containerlib-prod-12763", + "project_name": "elastic-containerlib-prod", + "service_id": "24E6-581D-38E5", + "service_description": "BigQuery", + "footprint.scope1": 4.044, + "footprint.scope2.location": 1.797, + "footprint.scope2.market": null, + "footprint.scope3": 2.337, + "footprint.offsets": null + } + }, + "metricset": { + "name": "carbon", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} diff --git a/x-pack/metricbeat/module/gcp/carbon/_meta/docs.asciidoc b/x-pack/metricbeat/module/gcp/carbon/_meta/docs.asciidoc new file mode 100644 index 000000000000..1208cc250845 --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/_meta/docs.asciidoc @@ -0,0 +1,31 @@ +The `carbon` metricset is designed to collect Carbon Footprint data from GCP +BigQuery monthly cost detail table. BigQuery is a fully-managed, serverless +data warehouse. + +Cloud Carbon export to BigQuery enables you to export detailed Google Cloud +carbon footprint data (such as carbon produced by tier and service) automatically +throughout the month to a BigQuery dataset that you specify. Then you can access +your Cloud Carbon data from BigQuery for detailed analysis using Metricbeat. +Please see https://cloud.google.com/carbon-footprint/docs/export[export carbon +footprint data to BigQuery] for more details on how to export carbon footprint data. + +[float] +=== Metricset-specific configuration notes +* *dataset_id*: (Required) Dataset ID that points to the top-level container which contains +the actual carbon footprint tables. +* *table_pattern*: (Optional) The name of the table where carbon footprint data +is stored. Default to `carbon_footprint`. + +[float] +=== Configuration example +[source,yaml] +---- +- module: gcp + metricsets: + - carbon + period: 24h + project_id: "your project id" + credentials_file_path: "your JSON credentials file path" + dataset_id: "dataset id" + table_name: "table name" +---- diff --git a/x-pack/metricbeat/module/gcp/carbon/_meta/fields.yml b/x-pack/metricbeat/module/gcp/carbon/_meta/fields.yml new file mode 100644 index 000000000000..42a8a8dfd475 --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/_meta/fields.yml @@ -0,0 +1,35 @@ +- name: carbon + description: Google Cloud Carbon Footprint metrics + release: beta + type: group + fields: + - name: project_id + type: keyword + description: Project ID the carbon footprint report belongs to. + - name: project_name + type: keyword + description: Project name the carbon footprint report belongs to. + - name: service_id + type: keyword + description: Service ID for the carbon footprint usage. + - name: service_description + type: keyword + description: Service description for the carbon footprint usage. + - name: region + type: keyword + description: Region for the carbon fooprint usage. + - name: footprint.scope1 + type: float + description: Scope 1 carbon footprint. + - name: footprint.scope2.location + type: float + description: Scope 2 carbon footprint using location-based methodology. + - name: footprint.scope2.market + type: float + description: Scope 2 carbon footprint using market-based methodology. + - name: footprint.scope3 + type: float + description: Scope 3 carbon footprint. + - name: footprint.offsets + type: float + description: Total carbon offsets. diff --git a/x-pack/metricbeat/module/gcp/carbon/carbon.go b/x-pack/metricbeat/module/gcp/carbon/carbon.go new file mode 100644 index 000000000000..6da5dc86281a --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/carbon.go @@ -0,0 +1,323 @@ +// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +// or more contributor license agreements. Licensed under the Elastic License; +// you may not use this file except in compliance with the Elastic License. + +package carbon + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "time" + + "cloud.google.com/go/bigquery" + "cloud.google.com/go/civil" //nolint:typecheck // civil is used for type casting + "google.golang.org/api/iterator" + "google.golang.org/api/option" + + "github.com/elastic/beats/v7/libbeat/common/cfgwarn" + "github.com/elastic/beats/v7/metricbeat/mb" + "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp" + "github.com/elastic/elastic-agent-libs/logp" + "github.com/elastic/elastic-agent-libs/mapstr" +) + +const ( + // metricsetName is the name of this metricset + metricsetName = "carbon" +) + +// init registers the MetricSet with the central registry as soon as the program +// starts. The New function will be called later to instantiate an instance of +// the MetricSet for each host defined in the module's configuration. After the +// MetricSet has been created then Fetch will begin to be called periodically. +func init() { + mb.Registry.MustAddMetricSet(gcp.ModuleName, metricsetName, New) +} + +// MetricSet holds any configuration or state information. It must implement +// the mb.MetricSet interface. And this is best achieved by embedding +// mb.BaseMetricSet because it implements all of the required mb.MetricSet +// interface methods except for Fetch. +type MetricSet struct { + mb.BaseMetricSet + config config + logger *logp.Logger +} + +type config struct { + Period time.Duration `config:"period" validate:"required"` + ProjectID string `config:"project_id" validate:"required"` + CredentialsFilePath string `config:"credentials_file_path"` + CredentialsJSON string `config:"credentials_json"` + DatasetID string `config:"dataset_id" validate:"required"` + TableName string `config:"table_name"` +} + +// Validate checks for deprecated config options +func (c config) Validate() error { + if c.CredentialsFilePath == "" && c.CredentialsJSON == "" { + return errors.New("no credentials_file_path or credentials_json specified") + } + + if c.Period.Hours() < 24 { + return fmt.Errorf("collection period for carbon footprint metricset %s cannot be less than 24 hours", c.Period) + } + return nil +} + +// New creates a new instance of the MetricSet. New is responsible for unpacking +// any MetricSet specific configuration options if there are any. +func New(base mb.BaseMetricSet) (mb.MetricSet, error) { + cfgwarn.Beta("The gcp '%s' metricset is beta.", metricsetName) + + m := &MetricSet{ + BaseMetricSet: base, + logger: logp.NewLogger(metricsetName), + } + + if err := base.Module().UnpackConfig(&m.config); err != nil { + return nil, fmt.Errorf("unpack carbon footprint config failed: %w", err) + } + + m.Logger().Debugf("metricset config: %v", m.config) + return m, nil +} + +// Fetch methods implements the data gathering and data conversion to the right +// format. It publishes the event which is then forwarded to the output. In case +// of an error set the Error field of mb.Event or simply call report.Error(). +func (m *MetricSet) Fetch(ctx context.Context, reporter mb.ReporterV2) (err error) { + // find current month + month := getReportMonth(time.Now()) + + var opt []option.ClientOption + if m.config.CredentialsFilePath != "" && m.config.CredentialsJSON != "" { + return errors.New("both credentials_file_path and credentials_json specified, you must use only one of them") + } else if m.config.CredentialsFilePath != "" { + opt = []option.ClientOption{option.WithCredentialsFile(m.config.CredentialsFilePath)} + } else if m.config.CredentialsJSON != "" { + opt = []option.ClientOption{option.WithCredentialsJSON([]byte(m.config.CredentialsJSON))} + } else { + return errors.New("no credentials_file_path or credentials_json specified") + } + + client, err := bigquery.NewClient(ctx, m.config.ProjectID, opt...) + if err != nil { + return fmt.Errorf("gerror creating bigquery client: %w", err) + } + + defer client.Close() + + if m.config.TableName == "" { + m.logger.Warn("table_name is not set in config, \"carbon_footprint\" will be used by default.") + m.config.TableName = "carbon_footprint" + } + + tableMeta, err := getTable(ctx, client, m.config.DatasetID, m.config.TableName) + if err != nil { + return fmt.Errorf("getTables failed: %w", err) + } + + var events []mb.Event + eventsPerQuery, err := m.queryBigQuery(ctx, client, tableMeta, month) + if err != nil { + return fmt.Errorf("queryBigQuery failed: %w", err) + } + + events = append(events, eventsPerQuery...) + + m.Logger().Debugf("Total %d of events are created for carbon footprint", len(events)) + for _, event := range events { + reporter.Event(event) + } + return nil +} + +// getReportMonth gets the year-month of the latest expected report. +// GCP creates new reports on the 15 of each month. So if the date is below +// that, we fetch the previous month. +func getReportMonth(now time.Time) string { + if now.Day() < 15 { + now = now.AddDate(0, -1, 0) + } + return fmt.Sprintf("%04d-%02d-01", now.Year(), int(now.Month())) +} + +type tableMeta struct { + tableFullID string + location string +} + +func getTable(ctx context.Context, client *bigquery.Client, datasetID string, tableName string) (*tableMeta, error) { + dit := client.Datasets(ctx) + + for { + dataset, err := dit.Next() + if errors.Is(err, iterator.Done) { + break + } + if err != nil { + return nil, err + } + + meta, err := client.Dataset(dataset.DatasetID).Metadata(ctx) + if err != nil { + return nil, err + } + + // compare with given dataset_id + if dataset.DatasetID != datasetID { + continue + } + + tit := dataset.Tables(ctx) + for { + table, err := tit.Next() + if errors.Is(err, iterator.Done) { + break + } + if err != nil { + return nil, err + } + + if table.TableID == tableName { + return &tableMeta{ + tableFullID: table.ProjectID + "." + table.DatasetID + "." + table.TableID, + location: meta.Location, + }, nil + } + } + } + return nil, fmt.Errorf("could not find table '%s'", tableName) +} + +func (m *MetricSet) queryBigQuery(ctx context.Context, client *bigquery.Client, tableMeta *tableMeta, month string) ([]mb.Event, error) { + var events []mb.Event + + query := generateQuery(tableMeta.tableFullID, month) + m.logger.Debug("bigquery query = ", query) + + q := client.Query(query) + + // Location must match that of the dataset(s) referenced in the query. + q.Location = tableMeta.location + + // Run the query and print results when the query job is completed. + job, err := q.Run(ctx) + if err != nil { + err = fmt.Errorf("bigquery Run failed: %w", err) + m.logger.Error(err) + return events, err + } + + status, err := job.Wait(ctx) + if err != nil { + err = fmt.Errorf("bigquery Wait failed: %w", err) + m.logger.Error(err) + return events, err + } + + if err := status.Err(); err != nil { + err = fmt.Errorf("bigquery status error: %w", err) + m.logger.Error(err) + return events, err + } + + it, err := job.Read(ctx) + if err != nil { + return events, err + } + for { + var row []bigquery.Value + err := it.Next(&row) + if errors.Is(err, iterator.Done) { + break + } + + if err != nil { + err = fmt.Errorf("bigquery RowIterator Next failed: %w", err) + m.logger.Error(err) + return events, err + } + + if len(row) == 12 { + events = append(events, createEvents(row, m.config.ProjectID)) + } + } + return events, nil +} + +func createEvents(rowItems []bigquery.Value, projectID string) mb.Event { + event := mb.Event{} + event.MetricSetFields = mapstr.M{ + "project_id": rowItems[1], + "project_name": rowItems[2], + "service_id": rowItems[4], + "service_description": rowItems[5], + "region": rowItems[6], + + "footprint.scope1": rowItems[7], + "footprint.scope2.location": rowItems[8], + "footprint.scope2.market": rowItems[9], + "footprint.scope3": rowItems[10], + "footprint.offsets": rowItems[11], + } + + event.RootFields = mapstr.M{ + "cloud.provider": "gcp", + "cloud.project.id": projectID, + "cloud.project.name": rowItems[2], + "cloud.account.id": rowItems[3], + } + + event.ID = generateEventID(rowItems) + return event +} + +func generateEventID(rowItems []bigquery.Value) string { + // create eventID using hash of usage_month + project.id + project.name + service.description + region + // This will prevent more than one carbon metric getting collected for the same month. + eventID := rowItems[0].(civil.Date).String() + + rowItems[1].(string) + + rowItems[2].(string) + + rowItems[3].(string) + + rowItems[5].(string) + + h := sha256.New() + h.Write([]byte(eventID)) + prefix := hex.EncodeToString(h.Sum(nil)) + return prefix[:20] +} + +// generateQuery returns the query to be used by the BigQuery client to retrieve monthly +// cost types breakdown. +func generateQuery(tableName, month string) string { + // The table name is user provided, so it may contains special characters. + // In order to allow any character in the table identifier, use the Quoted identifier format. + // See https://github.com/elastic/beats/issues/26855 + // NOTE: is not possible to escape backtics (`) in a multiline string + escapedTableName := fmt.Sprintf("`%s`", tableName) + query := fmt.Sprintf(` +SELECT + usage_month, + project.number, + project.id, + billing_account_id, + service.id, + service.description, + location.region, + + carbon_footprint_kgCO2e.scope1, + carbon_footprint_kgCO2e.scope2.location_based, + carbon_footprint_kgCO2e.scope2.market_based, + carbon_footprint_kgCO2e.scope3, + carbon_offsets_kgCO2e +FROM %s +WHERE project.id IS NOT NULL +AND usage_month = '%s' +ORDER BY usage_month ASC;`, escapedTableName, month) + return query +} diff --git a/x-pack/metricbeat/module/gcp/carbon/carbon_integration_test.go b/x-pack/metricbeat/module/gcp/carbon/carbon_integration_test.go new file mode 100644 index 000000000000..f6cad0e4f1c8 --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/carbon_integration_test.go @@ -0,0 +1,41 @@ +// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +// or more contributor license agreements. Licensed under the Elastic License; +// you may not use this file except in compliance with the Elastic License. + +//go:build integration && gcp && carbon +// +build integration,gcp,carbon + +package carbon + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + mbtest "github.com/elastic/beats/v7/metricbeat/mb/testing" + "github.com/elastic/beats/v7/x-pack/metricbeat/module/gcp/metrics" +) + +func TestFetch(t *testing.T) { + config := metrics.GetConfigForTest(t, "carbon") + config["period"] = "24h" + config["dataset_id"] = "carbon_footprint" + + metricSet := mbtest.NewReportingMetricSetV2WithContext(t, config) + events, errs := mbtest.ReportingFetchV2WithContext(metricSet) + if len(errs) > 0 { + t.Fatalf("Expected 0 error, had %d. %v\n", len(errs), errs) + } + + assert.NotEmpty(t, events) + mbtest.TestMetricsetFieldsDocumented(t, metricSet, events) +} + +func TestData(t *testing.T) { + config := metrics.GetConfigForTest(t, "carbon") + config["period"] = "24h" + config["dataset_id"] = "carbon_footprint" + + metricSet := mbtest.NewFetcher(t, config) + metricSet.WriteEvents(t, "/") +} diff --git a/x-pack/metricbeat/module/gcp/carbon/carbon_test.go b/x-pack/metricbeat/module/gcp/carbon/carbon_test.go new file mode 100644 index 000000000000..a9b06cdb607a --- /dev/null +++ b/x-pack/metricbeat/module/gcp/carbon/carbon_test.go @@ -0,0 +1,89 @@ +// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +// or more contributor license agreements. Licensed under the Elastic License; +// you may not use this file except in compliance with the Elastic License. + +package carbon + +import ( + "errors" + "io/ioutil" + "log" + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +func TestValidate(t *testing.T) { + for _, tt := range []struct { + name string + + config config + expectedError error + }{ + { + name: "with an empty config", + config: config{}, + expectedError: errors.New("no credentials_file_path or credentials_json specified"), + }, + { + name: "with a period lower than 24 hours", + config: config{ + CredentialsJSON: "{}", + }, + expectedError: errors.New("collection period for carbon footprint metricset 0s cannot be less than 24 hours"), + }, + { + name: "with all required fields filled", + config: config{ + CredentialsJSON: "{}", + Period: 25 * time.Hour, + }, + expectedError: nil, + }, + } { + t.Run(tt.name, func(t *testing.T) { + assert.Equal(t, tt.expectedError, tt.config.Validate()) + }) + } +} + +func TestGetReportMonth(t *testing.T) { + for _, tt := range []struct { + name string + + now time.Time + expectedValue string + }{ + { + name: "with a date before the 15", + now: time.Date(2023, 03, 12, 0, 0, 0, 0, time.UTC), + expectedValue: "2023-02-01", + }, + { + name: "with a date after the 15", + now: time.Date(2023, 03, 16, 0, 0, 0, 0, time.UTC), + expectedValue: "2023-03-01", + }, + { + name: "with a month before matching the previous year", + now: time.Date(2023, 01, 03, 0, 0, 0, 0, time.UTC), + expectedValue: "2022-12-01", + }, + } { + t.Run(tt.name, func(t *testing.T) { + assert.Equal(t, tt.expectedValue, getReportMonth(tt.now)) + }) + } +} + +func TestGenerateQuery(t *testing.T) { + log.SetOutput(ioutil.Discard) + + query := generateQuery("my-table", "jan") + + // verify that table name quoting is in effect + assert.Contains(t, query, "`my-table`") + // verify the order by is preserved + assert.Contains(t, query, "ORDER BY usage_month ASC") +} diff --git a/x-pack/metricbeat/module/gcp/fields.go b/x-pack/metricbeat/module/gcp/fields.go index d91003c2c634..3f4a6ff99e52 100644 --- a/x-pack/metricbeat/module/gcp/fields.go +++ b/x-pack/metricbeat/module/gcp/fields.go @@ -19,5 +19,5 @@ func init() { // AssetGcp returns asset data. // This is the base64 encoded zlib format compressed contents of module/gcp. func AssetGcp() string { - return "eJzsXVtz2ziWfs+vQPVLki23eqd3ax9SW1PldqZ7UhP3uMZOV+0TGwSPJEQgwOBiR/3rt3AjQYmSKF7kuGucl9gige9ccG44gL5HG9i+QytSvUJIU83gHXr9ixArBuiGCVOgO4b1Usjy9SuEJDDACt6hHDR+hVABikhaaSr4O/TXVwgh9MvNHSpFYRi8QmhJgRXqnfvge8RxCXEq+6O3lf1dChP/kj6fvsNwDkzVf46vivwzEJ38uQNP/PG4ONVCUr5CJWhJidofeRdCCsMokIv/aH10EIr98X/M/BMb2D4JWXQOXILGBdZ4rsEtqbOMrbZKQznL0BKUMJLAZIPHgb+rGeL/fXdar1rjFsLkTrs7Ps1KXFWUr8Kj37UGP6Kdt0Ed9RprJEEbyaFASylK1FqM13cf0BcDcrvYIyunjFG+OjRfa5if/LNRNZJ3dld4mzHpWkWdiyWiIUJ5jrzaF12X1FtYb4TS7lmFKCfMFIAkrAzD8gpp/PUK4eKzUboErq8Q5gWSwvDCsh2kFHLRgYfyR0EJZKXgej0EU2SZhEpIjdw4XRNVUjhtoMWQWe782+jDeySWSK8hijXOmwMTfKWQFl2Ta6Ex65h3yQTWh2d9sK/VM+FSGK73FYyIsjIaeinYjX92RgVbUglPmLFFIUVVQbHItxpUB+2WX4dJ/8CJKC3V7nUUBkP51jE/TtJj/qzCZANaZcRx7xEz06X7PdGEwc7AQ7nSmBNYkMosJCiQj1BkREhQB8HsWbIdOL+aMgdpNdGNg+KwSHAHZ23XadDTOP8paEbhFWSalrBQQAaA+mQHQEshEWYsAKMcKSCCF6rX9Itqx3P0m/nBSkBiYn+LZGPGBMEaCnRz98mbb6oQMVIC12xrkRkFkWF9mFRQtVlIwEM1+sbqn4XnNdqO5B2JHbjXxJmoxqlxDcEO6RF8+CcSFUhsHzgqJIfiSVIN09Bvh9LAkRb9GOCmnpgDbsz+LCihFHK7yK1uCb6QuMwU/QMGQrFa6/xCMOwWlZ/BKqfVyt9uF+hhTVWw1laBBWdbhB8xZThnfrX9dhvCE++CLEPty/AjWuKSsm231z1MklFQDCTp1sNvVpkd6/moUU+4yigfqK9WPnuScWuG8oBqZUBpv4ipVkg8cWTnRKrCBJ6FWmH0lOTGRepIbCjW4jno5aCfhNwsKF9JUGoqM0yAPsaY3oIJ05yDJEQFC2eZhiOKwcUoTDAhcxTY3x5BDgQxMV/OhmOqI/HM8cmbECsEMMGGoTVWKAfgSBrOKV8dVVkPIHNmfhCMvzFcWRtqh0GKcgIRxxNWSGksNRRXSZy1QPe4rBgUCB5BbtH//GfzyfVSg0TKfk756goVWGO7ULnQ6JEqGlepqezC/MuPzat7OYd9tZKC9Eo63oeHZ8g6kiyIGaVBLtbFUi0sPC4K6FK9g4zfi/kLaiNIL3le68Pf3/9870j61U7gzRmWEPXBCp8WgHCN6DRcpYW00TfBFSZUbzvc75FY+CDuOFyN2leFaqSCRxhWgX756QykRlNG/3Ah0yiw1utUIAlwbbOHGqifZieS6IHP8DVgptfbLGeCbOaQfz0F8lNEgduneon8s8gXS0wZFDOg+yzyoJNr/AjIz2Ml3VMfLbho2eZEl66Y89Apk5c2KrkA95ydr+ezRrEnzjqfmE/QTcoyRtwN0PmEvot0mOgbpHMqQCdXB6vBFku+qCsRmQ+qs8beTWw6/+/6X7/WKaRqSiB9QFbVHLYSE00fwQPDVcXsQ5a5PRARwTWmHOQcuBygZIbTcILsDuT7Q7yzRZRKzA7dxw87ODa4KTHHqxn5Y+Ob2zDHbnTTX/0r8DsAUzHQqn0IC5qk/ItNS688967qal+Y2i7YHNDSsCVlrCnaKrKGwrBeVDxSqQ1moW47PcPD+E3Z1ErgtC8kwsb6dmhfuT3M2NPbcmuqtFhJXJ7ivstFvKcUYmO5G2BAk7C6Z+x/jLIph7OdCmH7Wkz63SN1auC3URSivk4bBjxhuiwLCiNPhaGz0O48g6qsGlLrxFb00XoJjTX0dWTPJ73UzZ0tw/rlCSXZMOUZ5Jlwo5dU060mm6n023r7OT49axpcCGJK4HpRgGX7CFP10DJQyhACSi0Nq6dAfooD3rMG4rZM5oRhJ1Bebb8YkBQUEhIxITamOgXOb2jMic7N0FFBWW0Oqs3r31cb+D3GJ95VlGkjwpN1XhJXdSvCzR2615hsCkkfQbpuhPC26ybYb69ZCune+uUff3s9oR42W9IhtHJ7e9apZaP3F29MaRh28eTN3SfkxnP5Q73b6HYbgnOvEfSrjZ2kgdGS6pHbtha2R+pGi1uVzUSj8aVlmen2Ui1uj7jvPmrYC3CMQgRzLjSCrwSgQH9BWAXhtT+w77tZpith/vjfZ3AwRJGTbc030g4jdyjnzNXaU6TOoy4Nuedsve+qjA3ZVxKwduV+zHcUJ9WaMOEz6A1UayhBYpaFiqVfhwN3Xz4Kghmqx6yroH7tUe53ZYZaiX2skW/Tog2jzoDX1YCnBev1aSzUsBM7RvZh67zbMUwKcKZVX6YEvDA/0dt4Bj5WNpRZYsP0wK3NxktUrmnKDqWuUC7FBjgqxBN3rmJbwRUq8WchXTdlSXl3E+UewHEr+7ZVVhmkjTML4FIOLCj1n9qHBZaOMK5BX6axpTZJxnL8wrKZhtpR3DV2vZJ+1/wZtPdoP8JJTaUlJMmMV8p9+lq9CWfJgIvC92KGjQOcM5gsGk7GTCLjsH7czOOxzmMNdpH3NwYXWso1Ly6aZyeiG5xi18hdg8xIXfPN8rwj/xqnZftxaKIRE4ejvu0q1bipsVPXFpMtJQztW/1ZQlrz8gNanKybqClh+0aqgf22O/pxGeAe8cCWTIf4kJZMrRkzZFceaFqWG4g2xCjjF15i3EJsNyPGCzikkQFqn3yrY7rLRv8pd8cspyYYCQuobnOezuCOD6Zn1NDYnRv7izMvx4HduQ1OvsPYun85gbvXtzsCvwKu58KuJeYqNPpMDL+iReYLF8MPjJT4K7rzhxD/eT9SVy2eEYc92ltfsT+kkoKAUvHQx1iQWYGhFPzZtpGc7Bk8AovNsx7QqJA3EjVx5h3bWnZxDkjHK1HsWYvx9uyUnahEMXKdpbitlZgec4d9mAb2o2CmnCJebBC7M24hpaiP4YSeDTvlpRx4Qt4IXf+1i650rR4n6QSuWSI1P3hHdJaDtZUp+GMR2vEq4k5pPBwprOUdMIQjWhc5HMIELnLMMCd97z34KHCBfoqvzHg6fa11pRY5JhvgRTauTN52gMmJKVxv+Ya2lL8/PNz9cO/4gjxjrCgFCjg6lbMb6bB4Z8dVR2zh0Hi+rYG4AxgdYPsAVJXgamhedpyXfujAzBrrGyERwWQNby0v4asGyTFz+N/cv+1LwKV0gDAKXCsL9TwOzyz6c8FcSsyH1kzgYxdE9l+LqAXjjkEehOmQ1Zr2cHP3w6f3d9Hn72ANetpgjl5hycTTAv0spB3A/aYQ1a8VchJu7rywQWnTM4+UloBLd6q2H/HZuOOXbSa0zmBOx4YThIw76ntcjAFNajp6UTKv6ALFM8tuMO3d6Cmff9V9+PhThy69WY6Qxdt+5Mwsi27CTi+SGuVlFkkKc16uX3oJJJR13t1EqkwpllVSfN0uCBPK3d/DORB/lmlY+SS9w6ceK7b3SkAaZEm5u8HGJZd2fd7ff0Qexkmco1bibqW04dpvt4mOGhUqPn0AjdPSw4gaOf52ex4iDk8XkCNxmdswIYoK+AQQb8JxqeSYo9E23XSno9qwpTCrtTM9J7G2EwCGNXBCj2zgdhzb6HFoo0XJtc39taS5SdNsP/UWEcyIYY7bTjWe1sDT5hp/Z4LlRKwYWNJiRSQaN2vGWPLx/m0ctbV2cyvtdNKH1yE6Psit2s9LWFFh//Nv/o3jn28deCnc2yt5OhbtsmwlNLq++UfLwAnueRV55Jh2mFFLKbi2emUtitSH9xxm48u/Hh5QCVgZaTkiJAJM1om1QTnoJwAeCcS8OGVr6hDhJS+aSER3hjvbkkKnufqyl1Ifvs6w0F4201pVs0swLc11pdbPyTh3OS3Sklb+GGhg5FVjtWLM1oqRrClrsa4zYz6V5Dwj7S3DfJDEejnFwkBbNVzmdzqCfX43pEoh9BoKR/YbylGp3jbkp274tXKMUBqTzZX3ViXlRkMrk2V4CzJkIRVWoVpZm22/HtI9kUNbF+5+9/2dju4djSE3+h66tL0yuTJ5r8HvTH5v8hn3YxTHlVoL7Tw6E6tRO5/O57lrNOozKUrhldtpdh3ShT/UXU/aA5DvM8nybeYjzosC3DsR4qUSkBxDTwRf0lVmqgJP0iZD4l18fuBwRh+RNeYrUFde4n49JdcDbCvw14CDMuw4u7kps8iN0Zn4GLmnQBKxzwxpuKQFK0DpCDnDq2H3HV6vwFnHsKn7NmqoHz7CH8LQfYAJWy8O9XxGm7yee4HJJhIy4ZqqdQOTDRdPDIqVX0rXze/11l1rrRXAqNvJt1OfRD+LjTW8hbqm5Q1ebBZ4gcKk9QdvgzxSYCeBbzVkRAzepW0x3Z9oa+78SOK9plnKNWHQ0F+iRXwEfTFCY5R2i5zCfllL3CTaKYgR9jklpQBcZAy0BjnnKqhMzqhae8bbOZGfE2lRUdKipSfwUhSZXbp2MEY5zIn+aS0UoDiTS7q87B3gW1HQ5faabN7HByZY14fIy9KrhqYidJ+CaHDTJTWBjMb1PfSD3sn8noBtoJDU0WeIXoIWxCsH09lf22Wi1gh4UQnKrVsz2vVqbUG3/EgvOgyv55qOjkt4hhBcRF9vNaghYa5gqIus3XBjUiJGx6HTEjQuhgrUOu2YX1bdSnhIB0eoYAc9c4vtAG0Ty6syjGVJ4DuLV0kI6elPXBcORgUsKaex4NP1qoKkmvFDU85oUfnDaR8avnekN7um91/4VDrQ03M5iHNK007wZxDj9CJ0nBknO7WeBZdaI6w1lFU3LvSJM7oBR4C68kVT+45rJJWIlhWDErj2mUUhwHeO51iTtb9Aso4r0L3wKUp9wQhn2+a2O8Gh9cLC33ScTCat4H2PkfteOasdor6sMX3X3bKBqwqwRKVhmlbM3/nYWbpuMbrtjafNmbvjohERxE7oMGvN8lzw47yPApjDlNphj+r7aVxcz5lGxh3jtsnTIqKwMXZohR6ROfq2PJu9XMTL38fp7qwZ/PP4/A42zqCycZK2E0FPVK8RF/x7q8vbFldpMUy32+TMqRE7RF1KDf6XiAL+OkgZzmXeJWsz7fW1X+4YV6E5Rdmzqfw+oX7DcQIi6/aZGYkKx16GoI1J56wO/1AdZ7yndwXd8dX9cSX9pJjvqueuxN/d2uDgvpStVY82mhsr0MPN1xN0X7zf6b0IVfzaLjsArprhQLw9iPglVMPCTsQgdf8260VjKPrWso8xtCjgxcy1kfa6eDERb8KfOeoPnitnlkYCpm/dZ3uYz++shy2MvQ6Ii/Qt7voTTDaontiSU1LGaH19wYO3DXU3LFXNV5PERnPXEdteJMqdJMf1kko32poU23najrdDN61CmHf1aoRWUVz3pJxV3XseNrs6W8Tf8Ns1TlIiRXQtRzW/ps3fA/fqwNytVsP7cGfcfL2GuKIjD9O/B6ZxYxXc12xgVw9NmGHVzn5Sgl6Lws0dIzXfGm1zzy5NwEav/1hgwrIcKyiyIErsvlNkGsR1BuW00ovNmQYeFCfcTLKSmGsokJ8bKcGAbVFh3AIJT17ffOyMkhsyGv80EP0nFb7Z7vrmY/otPR33ZB9GEtioKiB0SUlmkZVGj/HrO1yNvTclLlIGxRkPcmqiG5Z20OzcrbR7M9E0ujrBJUudsDu/4nkayPGeSy+doaeKd65Y8oMpVIFEuSEb0C2w9ffYMqxU62of903psQefE/dVjqjA2ytHjLs8KD4nofJHSLEOrV3hkh7fo/+IWTy3KYy/EbrAnYeoWneSujAki/7z5WjedBerxrAJM1bLMVxb942J8v8DAAD//4wfFgE=" + return "eJzsXVtz3LaSfvevQOXF8ZYy2Thb++DaOlWKfJzjOlaOKpJTtU88INgzAw8IMLhInvz6LdxIcIYzw+FlFKXWeYkkEv31Bd2NRgP8Dm1g+w6tSPUKIU01g3fo9c9CrBigGyZMge4Y1kshy9evEJLAACt4h3LQ+BVCBSgiaaWp4O/Q314hhNDPN3eoFIVh8AqhJQVWqHfuD98hjkuIpOw/va3sz1KY+Jv0+fQdhnNgqv51fFXkX4Do5NcdeOI/j4tTLSTlK1SClpSo/ZF3IaQwjAK5+I/Wnw5Csf/8LzP/xAa2T0IWnQOXoHGBNZ5rcMvqLGOrrdJQzjK0BCWMJDDZ4HHgb2qB+P++OW1XrXELYXJn3R1/zUpcVZSvwqPftAY/Yp23wRz1GmskQRvJoUBLKUrUmozXdx/R7wbkdrHHVk4Zo3x1iF5rmJ/8s9E0knd2Z3hbMOlcRZ2TJaIhQnmJvNpXXZfWW1hvhNLuWYUoJ8wUgCSsDMPyCmn89Qrh4otRugSurxDmBZLC8MKKHaQUctGBh/JHQQlkpeB6PQRTFJmESkiN3DhdhCopnDXQYgiVO/82+vgeiSXSa4hqjXRzYIKvFNKii7gWGrMOuksmsD5M9cG+VlPCpTBc7xsYwTIXvJd93bhH0QchdCUp1zMa2mTytsL2LKJljbuX1CME+9MYEPb9ETAUyEdr4sMkce/ftpJYCtkNwyi8gmOkkxHHYEh+OQCMhNVA+r+6NztInqBYg1ooIir44fwZeG/fQz/scdmD2tsFEwQfEHkvsm+7hGs9QRz4uxwrKOwkXotCMLHa9sJVYrkBPTkqP+wgTD8OBfPjmZoRy6UCrYa64kAsjNLhikVZGQ39fLF/dkYXvKQSnjBji0KKqoJikW81dPFuvddh1j9yIkqrYPc6CoOhfOsmYyTSg35WYbIBrTLiAtkjZqbLL/dEEwY7Aw/lSmNOYEEqs5BgnSMUGRES1EEwe0nlDpxfTJmDtEmBGwfFYZHgDs7apkwhZYj0T0Fz/izTtISFAjIA1Gc7gHOXmLEAjHKkgAheqF7kFxXp8hCnKD9YDUhMXIgIbGPm/BUU6Obus8+kqULESAlcs61FZhREgfURUkHVZiEBD7XoG2t/Fp63aDuSz+ntwL0IZ6IaZ8Y1BDukR/DxX0hUIJ1nP6okh+JJUg3T8G+H0sCRFv0E4EhPLAE3Zn8RlFAKuV3k1rYEX0hcZor+AQOhWKt1KXrIsS0qT8Eap7XK324X6GFNVfDW1oAFZ1uEHzFlOGd+tv12G1aKPiG0ArUvw1u0xCVlnZHwGEtGQTGQpVsPv5lldqzn40Y94SqjfKC9Wv3sacbNGcoDqpUBpf0kploh8cSRpYlUhQk8C7fC6CnZjZPUsdhwrMVz8MtBPwm5WVC+kqDUVG6YAH2M5RULJpA5B0nIChbOMw1HFJOLUZhgQuEosD89ghwIYmK5nA3HVEfymePEmxQrJDDBh6E1VigH4EgazilfHTVZDyBzbn4QjL8zXFkfaodBinICEccTVkhpLDUUV0metUD3uKwYFAgeQW7Rf/9n85frpQaJlP075asrVGCN7UTlQqNHqmicpaayE/OHt82re2sO+2olBem16HgfHp5h1ZGsgphRGuRiXSzVwsLjooAu0zso+L2cv6A2g/Sa57U9/OP9h3vH0i+WgHdnWEK0B6t8WgDCNaLTcJUW0mbfBFeYUL3tCL9HcuGDuONwNWpfoK+RCh5hWAP6+aczkBpNGf3DpUyjwNqoU4EkwLVdPdRAPZmdTKIHPsPXgJleb7OcCbKZQ/81CeRJRIW7IlEflX8R+WKJKYNiBnRfRB5sco0fAXk6VtM97dGCi55tTnTpjDkPnTJ5abOSC0jP+fmannWKPXHW64n5FN0sWcaouwE6n9J3kQ5TfYN0TgPolOpgM9hiyRd1JSLzSXXW+LuJXef/Xv/6S72EVE0JpA/IqprDV2Ki6SN4YLiqGPUlZNUDERFcY8pBzoHLAUoonIYTdHdgvT8kOltEqcbs0H3isINjk5sSc7yaUT42v7kNNHazm/7mX4HfjJ1KgNbsQ1rQLMp/t8vSKy+9q7raF0jbCZsDWhq2pIw1RVtF1lAY1ouLRyq1wSzUbacXeBi/KZtaDZyOhUTYXN8O7Su3hwV7ukNiTZUWK4nLU9J3axEfKYXYWOkGGNAsWN0z9n+MsksO5zsVwva1uOh3j9RLA7+pqRD1ddow4AnXZUVQGHkqDZ2FdxcZVGXNkNogtqKPNkporKFvIHs+7aVh7mwd1i9PqMlGKM+gz0QavbSabjXZlUq/rbcP8elZl8GFIKYErhcFWLGPcFUPLQelDCGg1NKwmgTyJA5EzxqI2zKZE4YloLzZ/m5AUlBISMSE2JjqFDi/oTEnOkeho4Ky2hw0m9f/Xm3g3zE/8aGiTHvCnmzwkriqu8Ju7tC9xmRTSPoI0jWGhbfdHvl+p+NSSPfWz//8++sJ7bDZkg6pldvbs0EtG72/eGNKw7DLJ2/uPvv+C7d+qHcb3W5DCO41gn61sZM8MFpSPXLb1sL2SN1ocauyITQaX1qWmW4v1eL2iPvuo4a9ACcoRDDnQiP4SgAK9APCKiiv/Qf7vqMyXQnz7X+dIcGQRU62Nd9oO4zcYZwzV2tPsTqPuTTsnrP1vmsyNmVfScDalfsx3zGc1GoCwWewG6jWUILELAsVSz8PB+6+fBIEM1SPWVdB/dyj3O/KDPUS+1ij3KZFG0adAa+rAU8L1tvTWKhhJ3aM7sPWeXdgmBTgTLO+TBl4YXGit/MMcqxsKrPEhumBW5tNlKhc05QdSl2hXIoNcFSIJ+5CxbaCK1TiL0K6xvaS8u5+9j2A42b2bausMsgaZ1bApQJYMOq/dAwLIh3hXIO9TONL7SIZy/ETy6401I7hrrHrlfS75s9gvUf7EU5aKi0hWcx4o9znr9WbcJYOuCh8L2bYOMA5g8my4WTMJDMO88dRHo91Hm+wi7y/M7jQVK5lcdF1dqK6wUvsGrlrkBlpa75Znnesv8ZZ2X4emljExOmob7tKLW5q7NS1xWRLCUP7Vj9ISGtefkCLk3UzNSVs30g1sN92xz4uA9wjHtiS6RAfspKpLWOG1ZUHmpblBqINOcr4iZc4t5DbzYjxAgFpZILaZ73VQe6y2X8q3THTqUlGwgSq25ync7jjk+kZLTR258b+4szrcWB3boOT7wi27l9O4O717Y7Ar4DrubBribkKjT4Tw69okfnCxfADIyX+iu78efB/3Y+0VYtnxGGP9tZX7A+ppCCgVDz0MRZkVmAoBX+2bSSnewaPwGLzrAc0KuWNTE288o5tLbs4ByzHK1HseYvx/uyUn6hEMXKepbitl5gec4d/mAb2o2CmnCJfbBC7M25hSVEfwwk9G5bkpQJ4wt4IW/+li690rh5n6QSuWTI1P3hHdpaD9ZUp+GMZ2vEq4k5pPBwprPUdMIQjWhc5HMIELnLMMCd9r6D5JHCBfoqvzHg6fa11pRY5JhvgRTauTN4OgMmJKVxv+Ya2lH88PNx9f+/kgrxgrCoFCjg6jbMb6bB8ZydUR2zh0Hi+rYG4AxgdYPsAVJXgaui67Lgs/dBBmDXWb4VEBJM1vLGyhK8aJMfM4f/2/k1fBi5lA4RR4FpZqOdJeGbVnwvmUmo+NGeCHLsgsh8X0QrGHYM8CNMhqy3t4ebu+8/v72LM38Ea7LTBHKPCkomnBfogpB3A/aQQ1a8Vchpu7rywSWnTM4+UloBLd6q2H/PZuOOXbSG0zmBOJ4YTjIw76ntcjQFN6jp6cTKv6gLHM+tuMO/d6Cmff9Z9/PRThy19uxyhizf92JlZF92MnZ4kNcrLTJIU5rxSv/QUSDjrvEaPVJlSLKuk+LpdECaUu7+HcyD+LNOw8kl6h089VmzvlYA0yJJyd4ONW1za+Xl//wl5GCdxjpqJu5XSRmq/3SY26m/g6glonJUeRtTo8bfb8xBxeLqAHolbuQ1ToqiATwDxJhyXSo45Gm2Xm+50VBu2FGa1dq7nJNb2AoBhDZzQIxu4Hcc2ehzaaHFybdf+WtLcpMtsT3qLCGbEMCdtZxpPa+Bpc42/M8FKIlYMLGuxIhKdm3VjLPnz/m0ctbd2tJV2NunT65AdH5RWHef9vYSY/b/8RsrPtw68FOntlTydiHZFthIaXd/8s+XgBPeyijJyQjssqKUUXFu7sh5F6sN7DrPJ5deHB1QCVkZaiQiJAJN14m1QDvoJgEcGMS9O+Zo6RXjJkyYy0b3CnW1KodNSfdlTqY9cZ5hoL1torarZJYSWrnWl1s8pOHdPONKSVv4YaBDkVeO1Ys7WypHihcC16DpXzKcWOc/Ie8sxH2Sxnk6xMNA2DbfyO53BPn8YUqUQeg2FY/tbylGp3jTsp2H4tXKCUBqTzZWPViXlRkNrJcvwFmRYhVRYhWpl7bb9fEj3RA5tXbhPbezvdHTvaAy50ffQ9zMqkyuT9xr8zuT3Jp9xP0ZxXKm10C6iM7EatfPpYp67RqM+k6IUXrmdZtchXfhD3TXRHoB8n0mWbzOfcV4U4N6JEK+VgOQYeiL4kq4yUxV4kjYZEu/i8wOHM/qIrDFfgbryGvfzKbkeYFuB/yIDKMOOi5ubMovSGL0SH6P3FEii9pkhDde0YAUoHSFneDXsvsPrFTjvGDZ130QL9cNH+EMEug8wEevFoZ4vaJPXtBeYbCIjE86p2jYw2XDxxKBY+al03fxcb9215loBjLqdfEv6JPpZfKzhLdQ1L9/ixWaBFygQrf/wJugjBXYS+FZDRsTgXdqW0P2JtubOjyTfa5qlXBMGDf0lWsRH0O9GaIzSbpFT2C/riZuFdgpihH9OWSkAFxkDrUHOOQsqkzOq1l7wlibyNJEWFSUtXnoCL0WR2alrB2OUw5zon9ZCAYqU3KLL694BvhUFXW6vyeZ9fGCCeX2IvSy9amgqRvc5iA43nVIT6Ghc30M/6J3C7wnYJgpJHX2G7CVYQbxyMKX+2k4TtUbAi0pQbsOa0a5Xawu6FUd68WF4TWs6Pi4RGUJyEWO9taCGhbmSoS62dtONSZkYnYdOy9C4HCpw66xjfl11G+EhGxxhgh38zK22A7xNrK/KMJYlie8sUSVhpGc8cV04GBWwpJzGgk/XqwqSasb3TTmjxeX3p2No+O5Ib3FNH7/wqeVAz8jlIM6pTUvgr6DG6VXoJDNOd2o9Cy61RlhrKKtuXOgzZ3QDjgF15Yum9h3XSCoRLSsGJXDtVxaFAN85nmNN1v4CyTqvQPfCL1HqC0Y42za33QkOrRcW/qbjhJi0ivc9Ru4Tn9Y6RH1ZY/quu2UDVxVgiUrDNK2Yv/Oxs3TdEnQ7Gk+7Zu7Oi0ZkEDupw6w1y3PBj4s+CmAOV2qHPWrvp3FxPecyMu4Yt12eFhGFzbFDK/SIlaNvy7Orl4tE+ftI7s66wb9OzO8Q4wwmG4m0gwh6onqNuODfWVvetqRKi2G23WZnTovYYepSZvA/RBTwt0HGcK7wLlmbac+v/XLHuArNKc6ezeT3GfUbjhMwWbfPzMhUOPYyBG1cdM4a8A/VccZHelfQHV/dH1fST4r5rnruSvzdrQ0O7kvZWvVoo7uxCj3cfD1B98X7nd6LUMWv/bID4KoZDsSbg4hfQjUs7EQMMvc/Z71oDEd/ttXHGF4U8GLm2kh7XryYjDeRzxz1By+VM0sjAdOfPWZ7mM8frIdNjL0OiIv0Le7GE0w2qCZs2SkpY7S+vuDB+4a6G5aq5tMksdHcdcS2J4lyJ8lxPaXSjbZmie0ibcfboZtWIcy7ejVCqyiue1LOqu49j5hdnS3ib+TtGicpkSKGlqOWX/Pm74F7dYB2q9XwPtwZN1+vIa7oyMP074Fp3HgF95kN7OqhiTCs2dm/lKDXonC0Y6bmW6Pt2rPLErDR6z8WmLAsxwqKLKgSu2+KTIO4XkE5q/Rqc66BB8MJN5OsJOYaCuRpIyUYsC0qjJsg4cnrm0+dWXLDRhOfBqL/rMKX7a5vPqVf6em4J/swkiBGVQGhS0oyi6w0ekxc35Fq7L0pcZEKKFI8KKmJbljaQbNzt9LuzUTT2OoElyx1wu78xPM0kOM9l147Q08V71yx5AdTqAKJckM2oFtg6+/YMqxU62of96X02IPPifuUIyrw9sox4y4Pis9JqPwRUqxDa1e4pMf36D9iFs9tCuNvhC5w5yGq1p2kLg3JYvx8OZY33cWqMW3CjNV6DNfW/clU+X8BAAD//4RIvEU=" } diff --git a/x-pack/metricbeat/modules.d/gcp.yml.disabled b/x-pack/metricbeat/modules.d/gcp.yml.disabled index 34482d2ca056..4a42e04b3115 100644 --- a/x-pack/metricbeat/modules.d/gcp.yml.disabled +++ b/x-pack/metricbeat/modules.d/gcp.yml.disabled @@ -63,3 +63,12 @@ dataset_id: "dataset id" table_pattern: "table pattern" cost_type: "regular" + +- module: gcp + metricsets: + - carbon + period: 24h + project_id: "your project id" + credentials_file_path: "your JSON credentials file path" + dataset_id: "dataset id" + table_pattern: "table pattern"