[Filebeat] Module incompatibility with older ES/Kibana versions

[andrewkroh]

The Elastic support matrix indicates that the latest Filebeat 7.x version works with all 7.x versions of Elasticsearch. This is an assumption I'm making based on the table pictured below. There is a "Compatibility with Beats" table but it does not include Elasticsearch or Kibana columns.

This is not true for all Filebeat modules because there are specific Elasticsearch features utilized in module ingest node pipelines that require newer ES versions. For example:

• The set processors copy_from feature. Used to copy objects.
• The convert processor's type: ip. Used to ensure strings are valid IPs for ip mapping fields.
• registered_domain is new in 7.13.
• network_direction is new in 7.12.

There are several other features where Filebeat automatically rewrites the ingest node pipeline to create a compatible pipeline for the current ES version (possibly with a reduced feature set).

• The append processors allow_duplicates.
• community_id is new in 7.12
• The set processor's ignore_empty.
• uri_parts is new in 7.12
• user_agent requires ecs: true in ES [6.7, 7.0).

How do we want to handle modules that require specific Elasticsearch versions and how do we indicate this to users in the support matrix?

• It might just be to say a particular module requires ES 7.N, documented it, add a runtime check, and then put a footnote in the support matrix that says certain Filebeat modules have additional version constraints.
• Or indicate that Filebeat requires a specific ES version like Filebeat 7.13 requires at least ES 7.12 (FB works with previous ES minor). This would mean that modules can be developed using more recent ES features and it's easy to test with one previous ES version.
• Manually fix every module pipeline that uses new features like copy_from or convert with type: ip. With this we never get to take advantage of new features in ES.
• Try to automatically rewrite pipelines for compatibility. This solution is hard to maintain and test for every edge case. And some configs cannot be automatically rewritten. I attempted this in [Filebeat] Make set processor with copy_from compatible with ES < 7.13 #26593.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

While we wait to make a decision I opened #26631 to get a quick fix for copy_from. I didn't find any usages of convert with type: ip.

[andrewkroh]

@adriansr added automatic rewriting of pipelines for these cases in #26676.

• Replaces usages of convert processor using type: ip with an equivalent grok expression.
• Removes the network_direction processor.

[andrewkroh]

@adriansr added module testing using ES 7.11 so that we can catch problems with future module changes. See #26737.

[P1llus]

Adding the fact that when a module (or package/integration) has a dashboard, this dashboard can only be installed on a Kibana that runs minimum the same version as the Kibana used to export the included dashboard.

That means if a dashboard has a hotfix, then suddenly that dashboard will no longer be available to any older versions of kibana (even though it might have worked before).

Personally I feel we should be more strict with the versions supported, we should not be more than one minor version behind the rest of the stack, and should commonly not be newer than the rest of the stack either.

There should also be some sort of Asterisk next to the versions, maybe mentioning that beat modules might require a higher minimum level for all functionality to work?

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[kvch]

There should also be some sort of Asterisk next to the versions, maybe mentioning that beat modules might require a higher minimum level for all functionality to work?

The support matrix has an asterisk that has a following clarification:

We recommend running the latest version of Beats, Logstash, Elastic Agent and ES-Hadoop; earlier versions will work with reduced functionality.

[andrewkroh]

@nimarezainia Could you help by confirming that my interpretation of the support matrix is correct? Is the current requirement that Beats 7.14 be compatible with ES/Kibana 7.0? Or have I misinterpreted it.

[ruflin]

@nimarezainia @mostlyjason @sorantis I know you start to think about this also in the context of packages.

My personal take: There are 2 layers of compatibility:

• Elasticsearch output
• Modules

Elasticsearch output stays compatible because Elasticsearch is not expected to make breaking changes in minor. For modules we should require the same version of the stack or newer.

[andrewkroh]

The merging of #27398 will mean that Beats 7.15+ require at least Kibana 7.15 to import dashboards and Kibana index patterns. The release notes will say "Beats can only import and export dasbhboards using at least Kibana 7.15."

The support matrix needs to be updating to reflect this when 7.15 is released.

[nimarezainia]

@andrewkroh this is complex as you have outlined above with many components that have their own support considerations.

to begin with, these matrices need to be pruned and show only the beats versions that are currently under support (i.e > 6.8.x)

then I propose that we only support two versions (current and one previous) as also mentioned above. If you look at the Endgame table that is a good example.

For beats we also have this footnote: "It's highly recommended to run the same minor versions across Elasticsearch, Kibana, and Beats for best monitoring functionality."

Allow me to work with @andresrc on these tables and update them and clarify accordingly.

[andrewkroh]

then I propose that we only support two versions (current and one previous) as also mentioned above.

+1

Another issue the broad compatibility affects is the ability to upgrade to ECS 1.11 which uses the flattened data type added in 7.3. #27107 (comment)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!