[Discuss] Ship data to older version of Elasticsearch

[ruflin]

Elastic Agent has a limitation in place that only allows it to ship data to versions equal or newer of Elasticsearch. This is to reduce the compatibility matrix and keep it simpler. Having the version in sync was especially important for Beats that was also loading assets into Elasticsearch like templates and ingest pipeline which might contain feature not supported by older versions yet. But with the move to packages, more and more logic is removed from the edge and put into the packages itself. What long term is left is inputs and mostly bulk requests to Elasticsearch which should not contain breaking changes. If there is, we might need an Elasticsearch condition: elastic/package-spec#334

For inputs to ship data, the dependency now is not the Elasticsearch version itself but which package version is installed. For example an input shipping TSDB data in a future release requires that TSDB templates are installed. This means, it is less about the Elastic Agent and Elasticsearch version if data can be shipped than the package. If an Elastic Agent could ship to older versions of Elasticsearch, it would depend on if the matching package is installed in Fleet. It is not an Elastic Agent enforcing the Elasticsearch version, but the input.

This issue is mainly to discuss the above not to make any changes right now as it could have quite an impact on the complexity around testing and compatibility. The way I start to think of it, there might be a future where customers constantly run the most recent version of Elastic Agent and if possible packages but not always update their Elasticsearch version.

This also ties into removing lightweight modules and move then into packages: elastic/integrations#3188

[nimarezainia]

thank you for this discussion.

I haven't seen this become an issue with our users as yet. That could mainly be due to the types of users currently deploying the agent. They tend to be more leading edge and smaller in size. Once we are truly enterprise ready and have the larger organizations uptake the agent I suspect we will hit some of these issues.

Even then, I would suggest that our backward compatibility be to only one or two well known versions (that we have tested). Perhaps one in the current major.(version-1) and previous major's last. and of course some QA sanity check.

if we do go down this path there would be a level of complexity added as we always have to think about backward compatibility with any future code changes.

[rodrigc]

I mentioned this on the Elastic Slack https://elasticstack.slack.com/archives/C02J2JBS0FP/p1683700872863199?thread_ts=1683590609.706699&cid=C02J2JBS0FP

I don't know if it is possible, but it might be nice to relax the checks on the Elastic server side, so that more versions of elastic-agent can run against a particular Elastic server.

It's a pain in the neck to be required to upgrade the Elastic server + Fleet in order to run a newer elastic-agent which has stability fixes

[cmacknz]

Beats run under support the allow_older_versions=true flag on the Elasticsearch output in 8.7.1 to work around this elastic/beats#3496

We can leave this issue open as the point is that we should consider doing this by default.

[cmacknz]

We allow shipping to older versions of Elasticsearch by default as of 8.12.0 elastic/beats@e39b37e

Closing.