Track rotated

[rockb1017]

continuing #168
fixes #85

instead of closing all files at the end of poll cycle, keep them open till next poll cycle. Only after opening all matched files, consume files that kept open from previous poll and close. By having this "overlap" at each cycle, we make sure we have trace of moved away files and consume any logs that were written to it but not yet read before being rotated.

In addition, there were "Move/Create" rotation tests not being skipped for windows. I skipped those.

[codecov]

Codecov Report

Merging #182 (9e316c3) into main (44b6bf5) will increase coverage by 0.0%.
The diff coverage is 91.6%.

@@          Coverage Diff          @@
##            main    #182   +/-   ##
=====================================
  Coverage   75.6%   75.7%           
=====================================
  Files         95      95           
  Lines       4360    4371   +11     
=====================================
+ Hits        3299    3311   +12     
  Misses       738     738           
+ Partials     323     322    -1     

Impacted Files	Coverage Δ
operator/builtin/input/file/config.go	73.4% <0.0%> (ø)
operator/builtin/input/file/file.go	73.5% <100.0%> (+2.1%)	⬆️
operator/builtin/input/file/reader.go	65.8% <100.0%> (+0.7%)	⬆️

[rockb1017]

https://github.com/open-telemetry/opentelemetry-log-collection/pull/183/checks?check_run_id=2808466320
i tested that configuration on github actions. it failed on windows.

[djaglowski]

https://github.com/open-telemetry/opentelemetry-log-collection/pull/183/checks?check_run_id=2808466320
i tested that configuration on github actions. it failed on windows.

@rockb1017 What conclusion are you drawing from this?

[rockb1017]

I naively deleted a blocked of code that seemed to be inefficient and unnecessary.
it was for "copy/trunc" cases and we are having flakiness on "copy/trunc" tests. I added those back and it passed.
Thanks for the discussion we had today. @djaglowski

Could you run a few more times just to be sure ?

[rockb1017]

100% data ingestion rate.
(i am generating 100,000 + 3 messages per pod and i deployed 3 pod. so total 300,009

[djaglowski]

This looks good to me.

A few notes on this PR.

1. See explanation here of what was wrong with the previous attempt.
2. In addition to believing that we have a good handle on the issues involved, I've rerun unit tests about 10 times and have observed no failures.
3. @rockb1017 has incorporated this branch into a build of the collector, and run it on k8s as part of a benchmark. He observed 100% delivery rate, and has posted the results here.
4. I'm confident that this is an improvement upon the previous release because it verifiably addresses a clear problem that was identified and does not regress in any regard that is currently tested. However, it is possible that further edge cases may be identified in the future.
5. I am drafting a design doc that explains how this operator works. I'll share this and will welcome feedback, especially as it pertains to possible improvements.
6. The codebase for this operator could be improved through refactoring. This is out of scope for this PR, but will be an ongoing focus for this project.
7. Similarly, the test suite for this operator needs improvement. Some tests are currently non-deterministic. Some edge cases could be tested independently. This will also be an ongoing effort, but should not hold up this PR.

@tigrannajaryan, I'd appreciate any thoughts you have on these changes, or on the suggestion that we merge this and continue to improve from there.

[tigrannajaryan]

@rockb1017 can you please update the PR description, squash the commit into one and add a commit message that fully explains why the changes are made and how they fix the problem.
Thanks for spending time on this for your patience. @djaglowski thanks a lot for your help and for the summary in your approval.

[tigrannajaryan]

Thanks @rockb1017 and @djaglowski !