[Stack Monitoring] Enable testing of latest package versions

[klacabane]

Summary

We added integration tests for Agent based data in #119658. At this moment the tests work by loading a standard es-archiver bundle containing both static mappings of the metrics-* data streams and corresponding documents (example).

Ideally each update of the mappings in the packages should be reflected in the static archive in order to validate no regression were introduced as new versions are released.

1. We could have a mostly manual, documented process to update static mappings with each package update.
2. We could replace the static mappings created by installing the packages as a setup step to the test suite. This will require spawning a local package registry and hitting the fleet api. We'll likely have to bundle the packages in the kibana repo to avoid any network reliance. With such a setup in place, we'd only have to update the version of the package installed during test setup.
3. (recommended) use xpack.fleet.developer.bundledPackageLocation to load local packages without spawning a local registry. Because loading packages can have side effects on test suites of the other kibana plugins we'll have a create a dedicated suite spawning its own kibana test server (fleet example). See [Stack Monitoring] install bundled package for tests #146355 for the blueprint

For 2. we'll have to:

• spawn a docker package registry - example in fleet api https://github.com/elastic/kibana/blob/main/x-pack/test/fleet_api_integration/config.ts#L47
• load pinned version of a package
  • the package should be bundled in the image
  • hit kibana's /api/fleet/epm/packages/${pkg}/ to install the package assets. Consider doing this at the top level of the test suite so we only have to do one setup/teardown cycle
• remove mappings.json files from all the _package archives

AC

• A process exists to validate the latest version of the package's mappings in Stack Monitoring integration tests

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

There's a way to load packages bundled on the filesystem with the xpack.fleet.developer.bundledPackageLocation. This does not require a docker registry since bundled packages take precedence and ignore the registry request. This approach is cheaper to implement and maintain while perfectly matching our requirements.

This attempt #146355 to replace package mappings with preinstalled packages is fairly successful in that Stack Monitoring tests are green. However loading the packages on the test server used by every other plugin causes side effects, so we'd have to create a dedicated test server for monitoring api tests like fleet.
There's also a CI issue with the case of the package names, we could exclude the packages folder from the glob unless there is a way to bundle packages with different names.

@kpollich is there any counter indication using xpack.fleet.developer.bundledPackageLocation setting to load packages ?

[kpollich]

@kpollich is there any counter indication using xpack.fleet.developer.bundledPackageLocation setting to load packages ?

Sorry I'm not sure I understand the question. Could you rephrase?

[klacabane]

Is the setting safe to use and is it the recommended approach to load packages for automated testing purposes ?

[kpollich]

Is the setting safe to use and is it the recommended approach to load packages for automated testing purposes ?

Thanks for clarifying. This setting will control from where Fleet loads its bundled packages. Note that bundled packages are not automatically installed - just available on disk. Bundled packages exist to support offline environments.

If you'd like to install some packages by default, then you can use xpack.fleet.packages in kibana.yml: https://www.elastic.co/guide/en/kibana/current/fleet-settings-kb.html#_preconfiguration_settings_for_advanced_use_cases

[klacabane]

Completed in #147755