Automatic metricbeat testing against stack snapshots

[matschaffer]

The logstash image used for metricbeat module testing is quite old (7.12), which led to this PR #29440

But after some team discussion we decided it would be best to:

• Set docker-compose to 8-SNAPSHOT (for golang tests)
• Add supported-versions.yml that references 7.x-SNAPSHOT & 8.x-SNAPSHOT to test against both revisions (for python tests)
• Lock the above to a known-good docker image sha
• Update bump-stack-version to update the logstash docker-compose and supported versions files as part of the nightly snapshot flow

Subtasks:

• [Stack Monitoring] Add supported-versions testing to logstash module #30202
  • Add supported-versions and update docker-compose for logstash module #29440
• [Stack Monitoring] Build reasonable logstash module assertions #30203
• [Stack Monitoring] Add supported-versions testing to kibana module #30204
• [Stack Monitoring] Add supported-versions testing to elasticsearch module #30205
• [Stack Monitoring] Add supported-versions testing to enterprisesearch module #30206
• https://github.com/elastic/observability-robots/issues/1028
• [Stack Monitoring] Add supported-versions testing to beats module #30208

cc @jsoriano @v1v

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[matschaffer]

Also, once we have this sorted out we'll probably want to do the same for any module that monitors something we produce. (es, kibana, entsearch, ... beats?)

[matschaffer]

Heads up @jasonrhodes, @smith, @ruflin - I've turned this into an Epic and split out sub issues.

If we really want to get to an "ideal" state there's a fair bit more work than just the logstash docker-compose file getting updated.

[miltonhultgren]

@matschaffer I made this type:epic, hope that's correct!

[ruflin]

++ on driving this effort forward across all stack modules. This covers the metrics case. We should also discuss the metrics case especially around testing with "snapshot" logs.

[matschaffer]

@smith since this is an epic that will span cycles I've removed it from the current iteration

[jasonrhodes]

Doing a deep sweep of some old GH notifications that got lost.

@matschaffer @smith I don't have a good sense of the priority of this effort as it relates to other things we'd like to do with Stack Monitoring. This likely belongs in a single "Tech Debt / Testing" Work Queue that we can constantly re-order to make sure we are focused on the most important work from that stream without overdoing it, but for right now I think that means "the backlog", yes?

[matschaffer]

"backlog" works for me.

I agree with the shortcoming. We don't currently have good comprehensive testing of our modules on our own products. But given "platform observability" efforts, I'm not sure it's worth making the investment to fix it.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[smith]

We are not planning on doing this at this time.