-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cherry-pick #21450 to 7.x: Add new log file reader for filestream input #21470
Conversation
Pinging @elastic/integrations-services (Team:Services) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This breaks the windows builds
💔 Tests FailedExpand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
## What does this PR do? This PR adds the new refactored version of the previous `Log` reader of the `log` input called `logFile`. The differences between the two readers are the following: * `logFile` calls `Stat` only once to avoid too many system calls (`Log` calls `Stat` 3 times after every read) * `logFile` starts separate go routines to check if `close.after_interval` or `close.inactive` has elapsed. `Log` checks `close_inactive` after every `Read`. Thus, if the output is blocked, it cannot stop the reader. * `logFile` does not check if the file has been removed or renamed if `close_removed` or `close_renamed` are enabled. Instead it is checked separately in the prospector, so if the output blocks, the reader can be closed. This prevents Filebeat keeping too many open files if the output is blocked. (The code mentioned is not yet included in any PR.) ## Why is it important? This is the improved version of the previous `log` reader. (cherry picked from commit 1945373)
d979cdf
to
ad62274
Compare
💔 Tests FailedExpand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
Cherry-pick of PR #21450 to 7.x branch. Original message:
What does this PR do?
This PR adds the new refactored version of the previous
Log
reader of thelog
input calledlogFile
. The differences between the two readers are the following:logFile
callsStat
only once to avoid too many system calls (Log
callsStat
3 times after every read)logFile
starts separate go routines to check ifclose.after_interval
orclose.inactive
has elapsed.Log
checksclose_inactive
after everyRead
. Thus, if the output is blocked, it cannot stop the reader.logFile
does not check if the file has been removed or renamed ifclose_removed
orclose_renamed
are enabled. Instead it is checked separately in the prospector, so if the output blocks, the reader can be closed. This prevents Filebeat keeping too many open files if the output is blocked. (The code mentioned is not yet included in any PR.)Why is it important?
This is the improved version of the previous
log
reader.Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration files- [ ] I have added tests that prove my fix is effective or that my feature works- [ ] I have added an entry inCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Related issues
Rebased on #21444