Filestream input: Fingerprint for inode race detection

[urso]

Log collection using the filestream or logs input sometimes can treat an old file as a new file if we are seing an inode being reused. The clean_removed setting allows us to remove the state from the registry more early (filestream input can even detect removal asynchronously), but especially with autodiscovery in place we might have the input being shutdown before we managed to detect that the file was removed.

In order to better detect the inode reuse race condition, we want the harvester/prospector to add a fingerprint to the file metadata in the registry. The fingerprint would be computed from the first 4KB (configurable). The harvester (prospector) would check the fingerprint after opening the file, in order to check that the contents matches the original file.

#19990 Also discusses fingerprinting for identity tracking. But the solution proposed here is supposed to be used in conjunction with any other identity tracking we already have in place.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[nimarezainia]

This issue will be addressed by #34419
(@belimawr & @pierrehilbert please confirm and ensure this use case is addressed in sprint 10)

[pierrehilbert]

I will close this one for now and we will reopen it if it's not fixed by #34419