[Filebeat] Implement a more stable file identifier

[belimawr]

Issue

Our current default implementation implementation for file_identity uses a device ID that is not unique in some situations. One situation is when using using LVM according to Red Hat Enterprise documentation. Another problem is the use of inodes, in some situations inodes can be reused, which will make Filebeat skip some lines or whole files.

This issue is about exploring a more widely stable file_identity implementation for Filebeat and/or any Beat that requires it.

EDIT: as much as possible, the solution should be working in 7.17.

Current fix idea

Add a new fingerprint file identity option based on a computed hash from the initial bytes in the file.
The number of bytes to read should be configurable.

Definition of done

Target: 8.9 and 7.17.11

• Implement the new file identity option that can be selected.
• Experiment to find the "best" default value for the number of bytes to read.
• This new file identifier needs to at first fix the issue on Linux.
• This new file identifier needs to fix the issue on MacOS and Windows.
• Test to validate this file identifier is not impacting other in place.
• Test to confirm this file identifier is used when selected.
• Test to confirm we are not ingesting the same file when inodes changed.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

@nimarezainia: do you know users facing this situation?

[nimarezainia]

@nimarezainia: do you know users facing this situation?

~~ I am not aware, No. ~~
Update: inode issues faced are related to this.

[jlind23]

@belimawr looks like this might have some overlap with #27278 right?

[belimawr]

@belimawr looks like this might have some overlap with #27278 right?

Yes, there is some overlap. We could have a meta issue for all file identity related problems.

[nimarezainia]

@belimawr if there's any opportunity for the user to test a fix for the inode problem that would be great.

[belimawr]

I edited the issue adding information about the inode reuse problem as well. So we have a broader scope here.

[nimarezainia]

Hi, we need this fix for the inode rotation issue. Please prioritize for that use case.

[pierrehilbert]

Next step is to define the best way to fix this issue.