Ensure that stack monitoring Metricbeat modules are tested with the next release

[jsoriano]

Metricbeat modules for stack monitoring are currently tested only with the last released version, but not with the next one. This can lead to issues like #27495, where there is some issue with a Metricbeat monitoring itself.

A possible strategy is to use snapshots with an update strategy similar to the one used for the testing environments in testing/environments/snapshot.yml. Or directly use these environments. Still, there should be tests covering released versions, at least the last one to ensure that breaking changes are not introduced.

For the specific case of the beat module, it'd be good to have unit or integration tests that check that the current code in the beat module can monitor the current endpoint in beats monitoring.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[matschaffer]

In my recent exploration of metricbeat kibana module, I was surprised to not find anything like _meta/testdata that would help ensure module functionality across a (probably small) range of stack versions.

I tried to add it, but it doesn't look like testdata is set up to support modules that require extra endpoint requests as part of the init phase (like how kibana stats module checks the version).

Is there any plan to support response fixtures & expected payload testing for more complex modules? Seems like it could be helpful in fixing this issue as well.

[jsoriano]

I tried to add it, but it doesn't look like testdata is set up to support modules that require extra endpoint requests as part of the init phase (like how kibana stats module checks the version).
Is there any plan to support response fixtures & expected payload testing for more complex modules? Seems like it could be helpful in fixing this issue as well.

There have been some internal discussions about this, but we don't have a general framework. For example for rabbitmq we have a mocked server, but it doesn't support multiple versions, and so far it is specific for rabbitmq, but maybe it can be generalized.

For Cloudfoundry we also need something like this, because Cloudfoundry is expensive to manage (and forget about testing multiple versions with real deployments 🙂). I recently opened an issue about this: #28278

For system testing of some services, we have the supported-versions.yml in some modules, this file can be leveraged by system tests, as is done in the redis module for example.

[matschaffer]

Ah, thanks @jsoriano that's really interesting. That rabbitmq mock server looks like the sort of thing I was imagining for the kibana module last week.

I guess alternatively if we used a golang kibana (and ES and logstash) client, we could also go a similar route to the app client mock you pointed to.

I'm sure that'd be faster and lighter than a full mock http server.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!