[GCP] Configuration breaking changes due to metrics data streams addition

[endorama]

This issue was noticed while working on #490; it's goal is to migrate gcp beats metricsets to gcp package data streams, allowing the Agent to ingest data related to GCP metrics.

Context

• The current gcp package contains only logs related data streams.
• Variables shared by all implemented data streams are set in the Policy Template.
• Fleet UI support Policy Template level variables is yet to be implemented: [Fleet] Support input level variable for granular integrations kibana#112272

Issue

Given the missing support for Policy Template level variables in Fleet UI, some of the configuration variables must be moved: some configuration are shared by all data streams (logs and metrics one), like credentials and GCP project ID, so it make sense to move them to Package level variables and some are only pertaining to logs data streams, so it make sense to move them within the data streams themselves.

This introduce a configuration breaking change. (Thanks @andrewkroh for raising this concern)

Testing

In order to verify the breaking change and test the upgrade path, I bumped the gcp package with changes in #2707 to the next major (these changes are not yet pushed to #2707).

These were the steps:

1. Start a test cluster with 8.2.0-SNAPSHOT
$ elastic-package stack up --version "8.2.0-SNAPSHOT" -d -v
2. Install latest gcp package (1.5.0 at this moment) through Integrations page
   a. Set Project ID and Credentials JSON to non-default values
   b. Create a new agent policy with the integration
   c. Configuration screenshot
3. Build the gcp 2.0.2 package from https://github.com/endorama/integrations/tree/gcp-metrics-II
   $ cd packages/gcp && elastic-package build
4. Restart the local package registry
   $ elastic-package stack up --services package-registry -d
5. Upgrade the installed integration
   a. click on the upgrade button in the gcp integration "Settings" page
   b. from gcp integration "Integration policies" page click on upgrade button for the defined policy
   c. The upgrade steps identifies some conflicting fields.
   Note: in this screen Project ID, Credentials file and Credentials JSON fields are missing Full page screenshot
   d. Add foobar to GCP Billing Metrics Dataset ID field and click "Upgrade integration`
6. Verify if the issue discovered in 5.c persists; it does, see screenshot (opacity on bottom bar added for clarity)

Conclusions

• Changes in [gcp] Migrate some gcp beat metricset to data streams #2707 introduce a configuration breaking change.
• The upgrade path is not working as expected.

Suggested course of action

Considering that:

• introducing this breaking change with the current upgrade path is problematic for our users
• this blocks adding metrics related data streams to gcp package
• there are some metrics data streams that could be released quickly in beta

my suggested course of action would be:

1. Do not introduce breaking changes in gcp package and create a new gcp_metrics package with metrics related data stream.
2. Engage with Fleet UI team to solve the issue.
3. Continue migration from metricsets to data streams in gcp_metrics package.
4. Merge gcp and gcp_metrics at a later date.

/cc @andrewkroh @ravikesarwani @masci @elastic/obs-cloud-monitoring

[masci]

I'm +1 with point 1., we have the precedent of azure metrics showing this is the best course of action, to the point that I wouldn't even do 4.

[kaiyan-sheng]

+1 on Massi's comment. I think having gcp_metrics can be considered as a permanent solution instead of just a workaround.

[ravikesarwani]

While my personal preference would be a single package like AWS we cannot afford a breaking change since we have things like GCP dataflow templates based ingestion as well.

Did we look at all the way to see if we can keep it in a single package (gcp) but not introduce a breaking change?
cc: @ruflin to see if there are any extra suggestion here.

If there's no way to add metrics to the existing gcp package then I think we will need to go with the separate gcp_metrics package as proposed and +1 by Massi and Kaiyan.

[ruflin]

@joshdover Could you take a look at this from a Fleet perspective on what we could do here?

[andrewkroh]

I have the same opinion as Ravi. We need to avoid a breaking change and it would be nice to keep it in a single package.

I think this growth path for existing packages will be common. Packages may need to create more policy_templates or shift variables from global scope to policy_template scope and it would be good if upgrades continue to work. Maybe we should talk through some common update "use cases" that we want to handle and see what it would take to handle them in Fleet.

[endorama]

We decided to put taking a decision about this on hold, while I'm investigating with Fleet team the timeline for a fix to the mentioned Kibana issue.

[endorama]

Based on the discuss we had in Cloud Integrations weekly meeting we are holding on this decision until the Fleet UI team provides an estimation for the fix detailed in the linked Kibana issue.

[endorama]

Reporting here the answer from Fleet UI team:

If we were to add this support, it would probably take 2 weeks or so, but it is currently not on our foreseeable roadmap due to other product priorities that are higher.

[endorama]

After today Cloud Integrations weekly meeting we reached this conclusion: we are going to move forward with a separate package.

This is due to 2 main elements:

• [Fleet] Support input level variable for granular integrations kibana#112272 is not prioritized right now and we do not know when it will be available
• there may be further complexity to be explored in the upgrade path not solved by closing [Fleet] Support input level variable for granular integrations kibana#112272 as Jen suggested: “Even if we introduce support for input-level variables and templates, it does not guarantee that GCP policy upgrade will be conflict-free as it is possible that some of Fleet's data models will need to be changed.”

This issue will not be closed as the breaking change is still an issue that need to be addressed in the future when merging gcp and gcp_metrics packages (something that we want to do to provide a consistent user experience across all 3 major cloud providers), but our decision removes this as a blocker for #490

/cc @ruflin @andrewkroh @joshdover

[endorama]

The related Kibana issue has been closed, which means is possible to test the upgrade path for the integration with breaking change and expect the issue found to be solved.

[endorama]

This issue is solved in 8.x release track, still present in 7.17 as elastic/kibana#132068 has not yet been backported.

Will close once this is solved also for 7.17

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!