Migrate some Metricbeat modules to GCP package

[masci]

Goal: migrate all currently implemented Metricbeat gcp module Metricsets to Data Streams within gcp integration package and add system tests.

TODO:

• gcp.billing data stream:

  • migrate metricset: [GCP] Add GCP Billing Metricset #2141
  • add data stream tests: GCP package billing datastream tests #2312

• gcp.compute data stream:

  • migrate metricset [GCP] Add GCP Compute Metricset #2301
  • add data stream tests [GCP] add compute datastream tests #2849

• gcp.loadbalancer data stream:

  • migrate metricset: [GCP] Add GCP Load Balancer Metricset #2308
  • add data stream tests: todo

• gcp.pubsub data stream:

  • migrate metricset: [GCP] Add GCP PubSub metricset #2309
  • add data stream tests: todo

• gcp.storage data stream:

  • migrate metricset: [GCP] Add GCP Storage Metricset #2310
  • add data stream tests: todo

• gcp.metrics data stream:

  • migrate metricset: [GCP] Add GCP Custom Metricset #2314
  • add data stream tests: todo

• gcp.firestore data stream:

  • [GCP] Add firestore data_stream #2704
  • test integrated in the PR above

• ensure dashboards query the Agent stream and not the Metricbeat ones

🛑 Blockers

• gcp.billing require auxiliary files to bootstrap test data
  • copy all files for terraform system tests elastic-package#603
    • Support auxiliary files in terraform deploy spec package-spec#272
      • Use Terraform Dependecy Lock in system tests elastic-package#700
• elastic-package Terraform deployer should support GCP Cloud SDK blocks billing tests
  • Add GCP Cloud SDK to Terraform deployer image elastic-package#638
    • Use GCP ADC for auth in Terraform Deployer elastic-package#662
      • Add GCP test package elastic-package#701
• [GCP] Configuration breaking changes due to metrics data streams addition #2987
  • we decided to move forward with a new package, gcp_metrics to avoid the need for configuration breaking change. we reverted the decision and decided to wait on Kibana team to fix the upgrade issue and move forward with a single gcp package.

🏗 Implementation work

• [gcp] Migrate some gcp beat metricset to data streams #2707

❓ Open questions

• Should the new data streams be added to gcp package? As of now it contains only security related log collection.

🚨 Risks

• elastic/integrations CI does not support creating GCP infrastructure to be used during tests, but it supports the use case; support should be added.
  • copy all files for terraform system tests elastic-package#603
    • Support auxiliary files in terraform deploy spec package-spec#272
  • Add GCP Cloud SDK to Terraform deployer image elastic-package#638
    • Use GCP ADC for auth in Terraform Deployer elastic-package#662
      • Add GCP test package elastic-package#701
        With this solved we still need to migrate the CI configuration from elastic/elastic-package to elastic/integrations. The risk lies in the different CI and Stack versions in use

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[masci]

Package creation: #459

[legoguy1000]

I have some free time, I can look at getting this started.

[endorama]

@legoguy1000 thank you! I will keep an eye on it and review it.

[endorama]

Everything has being slowed down by adding tests, which required to address some limitation of our current CI setup for GCP.

The work to run GCP tests is in progress:

• copy all files for terraform system tests elastic-package#603
• Add GCP Cloud SDK to Terraform deployer image elastic-package#638
• Reduce terraform deployer build verbosity elastic-package#665
• Use Terraform Dependecy Lock in system tests elastic-package#700
• Add GCP test package elastic-package#701
• Use GCP ADC for auth in Terraform Deployer elastic-package#662

[endorama]

While adding tests we stumbled upon an issue with Elastic Agent parsing it's configuration that is affecting how we perform tests: elastic/elastic-agent#178

[endorama]

The issue has been solved, but we need to wait for release 8.1.0 to be out to merge it.

[endorama]

GCP test package is waiting on the final review but we were able to overcome the issues of the past days. I expect it to be merged beginning of next week

[endorama]

GCP test package merged today! This unblocks working on multiple data streams (probably all but billing, for which elastic/elastic-package#662 is required)

[legoguy1000]

@endorama we good to keep merging the individual Gcp metric sets?

[endorama]

No, unfortunately tests are still not working in this repo. We are migrating the configuration but facing current system tests for logs failing in #2704

[endorama]

While integrating tests in #2704 we discovered errors in the CI pipeline. As the changes are unrelated to #2704, I'm pausing the work on that one to fix the CI in #2707

[endorama]

Work has been done in a separate branch to pinpoint the issue, and we have been able to find and solve the issue.

During testing of #2707 it has been highlighted that the changes introduce a possible breaking change in the configuration that we should address.
I performed some tests and I can confirm there is a breaking change that we have to address. Further details in a separate comment.

[endorama]

In #2987 we decided to move forward by creating a separate integration package called gcp_metrics. I'll rework implementation PR to use the new package.

[legoguy1000]

@endorama let me know when the new package is in the gcp_metrics branch and I can update all my other branches with the remaining data streams and we can finish this.

[endorama]

The decision of moving forward has been reverted, in favour of a different approach: an issue has been opened on Kibana (elastic/kibana#131251), that has been closed yesterday. This issue solves the upgrade issue I found when upgrading GCP policy with a breaking change.
The next step is to ensure the upgrade works as expected and then continue migrating the metricsets as previously decided.

[endorama]

#2707 is ready for review!

[endorama]

#2707 has been updated to include latest changes and marked ready again, after positive decision in moving forward with the current implementation.

[endorama]

@legoguy1000 I'm sorry that all linked PRs have been closed! I'm not sure why it happened but was not intentional. We are testing the 2.0.0 package and we are going to work on other metricsets soon after!

[legoguy1000]

@legoguy1000 I'm sorry that all linked PRs have been closed! I'm not sure why it happened but was not intentional. We are testing the 2.0.0 package and we are going to work on other metricsets soon after!

No problem. When the branch was merged and deleted, any PR targeting it gets closed. I had to rebase it off of main anyways.