FileReader - Data loss when tailing symlink file with log rotation (k8s container logs)

[rockb1017]

Describe the bug
When I do performance test, I see 98~99% ingest rate.
with log generator EPS 5000, i see 98.6% ingest rate.
with EPS 3300, I see 99.3%
with EPS 2500, I see 99.1%
So even though I reduce EPS significantly (from 5000 to 2500), I still see some data loss.
So I looked into data loss pattern.

Each row, i should have 100000 event, but data loss is happening throughout the ingestion.
Drilling further down,

and I confirm a block from 545800 to 545913 is missing.
I think this pattern can be explained that this happens with log rotation.

I am not sure if it will happen if I were to tail files directly instead of tailing a symlink file to the actual log file.

Regardless, we need filereader to continue to tail it's been tailing, then begin following a new file upon discovering file rotation.

What version did you use?
Version: 0.22.0

What config did you use?
Config:

Environment
Docker imasge

[tigrannajaryan]

@djaglowski can you please have a look?

[djaglowski]

Resolved by opentelemetry-log-collection#182, and included in collector contrib release 0.29.0