XML SAX push parser leaks memory on JRuby, but not on MRI

[Burgestrand]

Hay!

I’m reading streaming XML from a remote server in JRuby using the SAX push parser in Nokogiri. It starts with an opening <events> tag, and then follows with a lot of streaming XML <event>data</event> coming in.

The following code leaks memory in JRuby (attached visualvm heap screenshot), but not in MRI.

Me and @jnicklas tracked it down to two internal variables in nokogiri.internals.NokogiriHandler which have content added to them for every new element, that is never released:

nokogiri/ext/java/nokogiri/internals/NokogiriHandler.java

Line 145 in 2adeeb6

lines.add(locator.getLineNumber());

nokogiri/ext/java/nokogiri/internals/NokogiriHandler.java

Line 146 in 2adeeb6

columns.add(locator.getColumnNumber() - 1); // libxml counts from 0 while java does from 1

require "nokogiri"

class Parser < Nokogiri::XML::SAX::Document
  def initialize
    @parser = Nokogiri::XML::SAX::PushParser.new(self)
  end

  def <<(data)
    @parser << data
  end

  def start_element(name, attrs)
  end

  def end_element(name)
  end
end

puts "Open up your GC monitor. Press enter to start."
gets

parser = Parser.new

parser << "<events>"
event = "<event></event>"
loop { parser << event }

[yokolet]

Please use parse.finish like in the example http://nokogiri.org/Nokogiri/XML/SAX/PushParser.html .

[knu]

@yokolet I think the point made here is that a streaming parser engine is not expected to consume memory proportional to the amount of total data processed.

[Burgestrand]

@yokolet @knu is correct. I do not expect the streaming parser to use an unbound amount of memory if the SAX handler is not retaining any data.

The data I am parsing is never ending, so there is no natural place for me to call #finish. I would like it to run for days, weeks, happily parsing data without using more and more memory for every minute it runs.

[quoideneuf]

@Burgestrand
In case this is of any use, here's an example here of a workaround to them problem you're seeing (I think!):
https://github.com/archivesspace/archivesspace/blob/master/migrations/lib/xml_sax.rb#L88-90

[eljojo]

Hi, I'm seeing a leak very similar (jruby as well), but this time when using Nokogiri::XML::Reader (which AFAIK is impossible to use parse.finish).

My graphs show the same leak.
This should be enough to reproduce it.

reader = Nokogiri::XML::Reader(File.open('hugefile.xml'))
reader.each { |node| node.name }

This is what the eclipse memory analyzer shows:

I assume this line is the responsible for the leak, but it's just a guess.