[Fleet] Fix datastream settings on package upgrade and package policy creation

[nchaulet]

Description

Related to #147034

That PR fix a few bugs with the synthetic source experimental datastream feature:

• we are not preserving the already set experimental features when upgrading a package => this is now fixed in the prepareTemplate function where we pass the existing datastream features
• we were not loading correctly the already set experimental feature when creating a new package policy => this is now fixed in the packageToPackagePolicy function
• we were mutating the original package policy object when editing experimental feature, the "save integration" button was
  disabled because of this as the original package policy and the updated one are not different

testing

There is no package using the source_mode: synthetic feature for now but you can edit locally a package or use this test package https://github.com/elastic/kibana/tree/main/x-pack/test/fleet_api_integration/apis/fixtures/test_packages/non_epr_fields/1.0.0

All the changes are covered by unit tests

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[hop-dev]

Great, thanks for all the tests 👍

[nchaulet]

@shahzad31 this test is always failing but it does seem to be related to any of my change here, and I was not able to get it failing locally apis synthetics CRUD routes DeleteProjectMonitors deletes integration policies when project monitors are deleted

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: d5fc256

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	897.6KB	897.6KB	+52.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	120.5KB	120.7KB	+198.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	60	66	+6
osquery	109	115	+6
securitySolution	445	451	+6
total			+20

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	69	75	+6
osquery	110	117	+7
securitySolution	521	527	+6
total			+21

History

• 💔 Build #94567 failed a8a8ae2
• 💔 Build #94565 failed 08195b3
• 💔 Build #94561 failed 253146227c44b5ee4da5bba798683843d230b3b7
• 💔 Build #94545 failed 652a0f697077b86fb23480265056530f2ab5b2ef
• 💔 Build #94487 failed 43c694c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

[ruflin]

There is no package using the source_mode: synthetic feature for now

@nchaulet As you likely understand I'm slightly concern by this. We should have tests in place that the issues found in #147034 (comment) are fixed.

[nchaulet]

We should have tests in place that the issues found in #147034 (comment) are fixed.

@ruflin I added unit tests that cover the changes here

[ruflin]

In #147034 (comment) I left a question around the following scenario:

• Synthetic source is enabled in the package
• Data starts to ship to it and synthetic source is actively disabled
• User creates a second policy with the same integration. What does the toggle in the policy show now? It should be disable because that is the setting of the data stream that already exists and must overwrite the package default.
• Same scenario with a different namespace, what happens now?

Can you comment on what the expected behaviour after this PR is? Especially interested in your thoughts around namespaces.

[ruflin]

I did test this again on the most recent -SNAPSHOT builds from today. The setting on the data_stream itself is correctly set including the template. But when I go to the UI, I see the following:

[nchaulet]

I did test this again on the most recent -SNAPSHOT builds from today. The setting on the data_stream itself is correctly set including the template. But when I go to the UI, I see the following:

@ruflin Does your package policy version is up to date with the system package installed? I think I may have found an issue here

[ruflin]

Not sure I fully understand. For testing I use this PR: elastic/integrations#4749 Then the elastic-package build and run elastic-package stack up --version=8.7.0-SNAPSHOT from the integrations directory. I made some changes on my end to the yaml to ensure the package installed (or at least the policy I see) is the one from the package itself.