Sane flush

[nukemberg]

Added flush cycle at configurable interval and inactive files close cycle every 10 seconds.

Logstash is blocking so flushing many files together in the event handler thread may be a problem for heavy setups that write a lot of data to a lot of files. I didn't think this edge case is worth complicating the code so i didn't handle it.

[jordansissel]

Eek, monkeypatching. Bad mojo. I'd much prefer something that was state held in the plugin instead of in a monkeypatched part of the File or GZipWriter.

[jordansissel]

If you're going to wrap things and track state like activity, it might be simpler to write your own IO-wrapper class

class MyFile
   def initialize(io)
       @io = io
   end

    def write(*args)
        @last_write = TIme.now
        @io.write(*args)
    end

    ...
end

And delegate flush/etc to @io, but add activity checks and stuff

[jordansissel]

overall this looks pretty good, I added some in-line comments on specific issues.

Thanks for your work! :)

[nukemberg]

Erk!!! my mailbox feels like messanger!

Thanks for the review. The reason for the flushes is C habits (although
i would have synced, not flushed before close). As for io.flush this is
because the docs and reality don't agree:

$ ruby test_flush.rb
size betfore write: 0
size before flush: 39194
size after flush 47264
size after io flush 47264
size after close 47274

$ jruby test_flush.rb
size before write: 0
size before flush: 35380
size after flush 35380
size after io flush 38984
size after close 47209

my guess is that Java is doing some dark IO magic somewhere...
As for the monkey patching, i felt it was cleaner not to create a new
object for the state, it takes more memory and the code looks dirtier.
Monkey patching is mostly a problem when you are replacing existing methods.

On Sat 28 Jan 2012 01:41:29 AM IST, Jordan Sissel wrote:

overall this looks pretty good, I added some in-line comments on
specific issues.

Thanks for your work! :)

---

Reply to this email directly or view it on GitHub:
#91 (comment)

[jordansissel]

Righto then. Can you add a big comment explaining what exactly you're monkeypatching and why, then for every method you invoke later (fd.active, etc) put a comment there saying explicitly that you are invoking a monkeypatched method - this'll help future debugging efforts :)

[nukemberg]

I removed the monkey patching as per your suggestions.

[jordansissel]

Thanks a bunch for this :)