Improvements to the way delete_after_read works

[djaglowski]

• Wait until end of poll cycle and then delete files from the primary
  goroutine
• Update checkpoints, even though in thoery files are consumed fully
• If file.Close fails, log at Warn level instead of Debug

Description of Changes

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• Add a changelog entry (for non-trivial bug fixes / features)
• CI passes

[djaglowski]

Log Files	Logs / Second	CPU Avg (%)	CPU Avg Δ (%)	Memory Avg (MB)	Memory Avg Δ (MB)
1	1000	1.413805	-0.17243886	128.0171	+2.3669128
1	5000	5.051762	-0.017374516	135.01428	-0.25228882
1	10000	9.31054	-0.79299164	144.11598	+0.43144226
1	50000	46.549637	-1.6237831	173.47777	-4.254593
1	100000	91.671104	-3.053276	249.20111	+16.885788
10	100	1.9655274	-0.034494042	136.33055	+2.344696
10	500	5.931107	+1.0345206	137.62218	-1.7978058
10	1000	11.68993	+0.7068777	144.23532	-2.011444
10	5000	53.897842	-1.3963852	181.43413	-0.4315796
10	10000	99.83008	-1.4278488	217.24286	-2.6743011

[djaglowski]

Log Files	Logs / Second	CPU Avg (%)	CPU Avg Δ (%)	Memory Avg (MB)	Memory Avg Δ (MB)
1	1000	1.4655347	-0.17248833	127.22912	+0.7413788
1	5000	4.810548	-0.1723404	135.26562	-5.7043304
1	10000	9.569025	-0.36200333	145.0784	+1.3406525
1	50000	50.8118	+3.449379	174.35898	-2.6182556
1	100000	93.05227	+12.214462	239.27048	+4.4637756
10	100	1.9138172	+0.068917155	136.21767	+3.8603058
10	500	5.4827824	-0.7759476	140.10385	+0.49353027
10	1000	10.965808	-0.27578545	146.60991	+0.28529358
10	5000	51.17245	-5.191658	179.54378	-1.1139526
10	10000	102.877014	+3.5897064	221.32098	+2.0651855

[codecov]

Codecov Report

Merging #432 (2dd4496) into master (f596ffb) will increase coverage by 0.12%.
The diff coverage is 80.00%.

@@            Coverage Diff             @@
##           master     #432      +/-   ##
==========================================
+ Coverage   72.53%   72.65%   +0.12%     
==========================================
  Files         124      124              
  Lines        8158     8160       +2     
==========================================
+ Hits         5917     5928      +11     
+ Misses       1731     1721      -10     
- Partials      510      511       +1     

Impacted Files	Coverage Δ
operator/builtin/input/file/file.go	76.85% <77.78%> (+0.23%)	⬆️
operator/builtin/input/file/reader.go	60.87% <100.00%> (ø)
operator/builtin/output/newrelic/newrelic.go	74.38% <0.00%> (+1.65%)	⬆️
operator/builtin/output/forward/forward.go	64.20% <0.00%> (+2.47%)	⬆️
operator/builtin/output/otlp/otlp.go	68.13% <0.00%> (+3.30%)	⬆️
logger/parser.go	70.37% <0.00%> (+7.41%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f596ffb...2dd4496. Read the comment docs.