gcp.compute data stream does not collect data in correct namespace

[endorama]

As per #3188 we started removing usage of Beats lightweight modules from gcp package data streams.

To remove them, we are directly using the gcp.metrics Metribeat module.

Today we uncovered an issue with he migration: data are collected all under gcp.metrics namespace, instead of gcp.<data stream name> namespace.

The first version with the issue is 2.11.0 (#3797)

Affected versions:

• 2.11.0
• 2.11.1
• 2.11.2 (not yet promoted to production)
• 2.11.3 (not yet promoted to production)
• 2.11.4 (not yet promoted to production)
• 2.11.5 (not yet promoted to production)
• 2.11.6 (not yet promoted to production)
• 2.11.7 (not yet promoted to production)
• 2.11.8 (not yet promoted to production)

Affected data streams:

• gcp.compute in 2.11.0
• gcp.firestore in 2.11.2 (not yet promoted to production)
• gcp.storage in 2.11.5 (not yet promoted to production)
• gcp.pubsub in 2.11.4 (not yet promoted to production)
• gcp.loadbalancing in 2.11.6 (not yet promoted to production)
• gcp.dataproc in 2.11.7 (not yet promoted to production)

Fixes released in:

• [GCP] Add compute ingest pipeline #4343 2.11.10-beta.1
• [GCP] Add firestore ingest pipeline #4347 2.11.10-beta.2
• [GCP] Add storage ingest pipeline #4348 2.11.10-beta.3
• [GCP] Add pubsub ingest pipeline #4349 2.11.10-beta.4
• [GCP] Add loadbalancing ingest pipeline #4350 2.11.10-beta.5
• [GCP] Add ingest pipeline for dataproc #4344 2.11.10-beta.6

[legoguy1000]

That's my bad, I was tracking that this, https://github.com/elastic/beats/blob/main/x-pack/metricbeat/module/gcp/metrics/response_parser.go#L82, was doing all the renaming but I guess I missed the parent field renaming being done by the module

[endorama]

No worries, we missed it too. I think we missed because is not part of the metricset but is being done somewhere in libbeat.

If you could update any open PR (if there are any still open) with this change that would be appreciated.

[endorama]

While working on fixing this we also discovered an issue affecting (probably) the entire gcp package due to a invalid YAML indentation that Kibana UI complained when adding a new integration. Already fixed with #4355

[endorama]

Reopening as this relates to multiple PRs

[endorama]

All PRs have been merged. I'll release gcp 2.11.10 with all the fixes.

[endorama]

This is being released in elastic/package-storage#5786

[endorama]

We missed the ingest pipeline for gke as was not included in the listed PRs #4357

Fixes released as part of 2.11.12