JabRef should release unused heap more frequently

[lenhard]

It seems that we have a memory leak with the entry editor on current master, latest commit 0326d77

JabRef 4.0-dev
Windows 7 6.1 amd64
Java 1.8.0_121

To reproduce:

• Open entry editor and select entry in the main table
• Press down key to cycle through the database
• Watch RAM consumption grow and not go down again

@JabRef/developers Please see if you can reproduce this on your system. This seems quite serious to me and a blocker for 4.0.

Any hints where this might come from are very much appreciated. @koppor Could you maybe try tracking this down with git bisect?

[lenhard]

@tobiasdiez I suspect that bindings are the cause of this all.

I know your time is limited, but can you please have a look at this? It's really a game changer...

[LinusDietz]

Confirmed. Had about 2.5GB allocated RAM after 5 minutes of clicking around in a Bibtex File with about 50 entries. Closing the EntryEditor did not help:/

JabRef 4.0-beta3
Mac OS X 10.12.6 x86_64
Java 1.8.0_144

[tobiasdiez]

The main reason for the memory leak is that the event listener in the source tab is never unregistered: https://github.com/JabRef/jabref/blob/master/src/main/java/org/jabref/gui/entryeditor/SourceTab.java#L55

After uncommenting this line, the memory still increases a bit for each new entry but not as much as before. I tried to further investigate the issue but couldn't really pinpoint the cause.

First of all, mostly int arrays are left over after opening a bunch of entries (the screenshot shows the difference in the memory consummation/allocated objects between a clean start and after the entry editor was opened for a few entries):

However, most of these arrays are actually no longer referenced and can be removed by garbage collection (no idea, why these are kept):

It appears that these numbers (and the other objects like HashMaps) have their origin in JavaFX. The bigger int arrays are a kind of raster image of older entry editors. Moreover, a few other objects are referenced by a cache in the JavaFX style manager (but again, these are weak hash maps).

Sorry, I don't have time to have closer look or even provide a fix. But I can definitely recommend the YourKit Java Profiler (see e.g. their documentation about memory leaks; license is in admin rep)

[lenhard]

@tobiasdiez Thanks for looking into this! I can confirm that the source pane is the most critical. Oh boy, I am happy that this was due to my changes, because it means that it's easy to repair.

#3116 repairs the event deregistration in the source tab. I would suggest the following: When #3116 is merged, the obvious blocker is gone and we can remove this issue from the 4.0 milestone. But we should keep investigated why the heap size doesn't really go down. After all, most of the heap reserved for JabRef is actually unused.

[LinusDietz]

I wonder if we could add some regression test that does the memory profiling automatically for us. Like selenium, just for java apps.

[lenhard]

Since the PR is merged, I've renamed this issue. Our main problem is not that JabRef consumes too much memory, but that it doesn't release unused heap. If there's a peak consumption, then JabRef somehow doesn't give memory back to the operating system when it's no longer needed.

EDIT: And since the memory leak is fixed, we can remove this from the 4.0 milestone.

[j0hannes]

I can confirm this issue with the 4.0-dev version. After using JabRef for a while, memory consumption goes up from 5.8% (start) to 18.1% and more. That's when I close and reopen it.

[lenhard]

To take some information from the PR linked above, the typical memory usage look like in this picture:

Sometimes there is a peak in memory usage (i.e. cycling through the entry editor) which makes Java reserve a lot of memory. When garbage collection happens, the used heap goes down a lot. For some reason, Java does not release the unused heap again. If we manage to make the VM do this, then the memory problem would be solved.

Unfortunately, I don't know how to do this.

[tobiasdiez]

https://stackoverflow.com/questions/30458195/does-gc-release-back-memory-to-os ?

[j0hannes]

I tried to limit memory consumption by adding -Xmx750M to the JabRef script, but it already uses 1.2 GB and doesn't seem to stop growing. Any ideas?

[Siedlerchr]

@j0hannes Did you pass the arguments to the jar file?
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/index.html

[j0hannes]

@Siedlerchr, I changed my /usr/bin/jabref:

#!/bin/sh

# Force usage of latest java version
JVM_PATH=$(find /usr/lib/jvm/* -maxdepth 0 -type d | sort | tail -n1)
exec $JVM_PATH/jre/bin/java -Xmx750M -jar /usr/share/java/jabref/JabRef.jar "$@"

[j0hannes]

Seems that this is no solution to the problem. The entry editor is broken after a while (won't display entries completely) and memory consumption readily exceeds the given limit.

[j0hannes]

Is there a way for JabRef to detect when memory consumption is running riot and restart itself?

[lenhard]

Well, we know what triggers the peaks. It is when you cycle through the main table with up/down keys when the entry editor is open or when you click around in the main table really fast. Every selection change in the main table will trigger the creation of a new entry editor and this is what's consuming memory.

I'm not sure that restarting JabRef would be good from a usability point of view, though. I think the long term solution is to not create entry editors all the time, but rather to update the parts that should change. Unfortunately, that is really quite a big refactoring. The start is here #3187 but it is still quite a long way to go.

[j0hannes]

For now, an automatic restart or a restart button would be very helpful. If I've used JabRef a lot (many entries opened in entry editor) and forget to close it before compiling it will crash my system due to insufficient memory. I could somehow replace the compile commands with scripts that close JabRef on compiling a document and restart it afterwards but that wouldn't be hassle-free neither.

[lenhard]

@j0hannes There have been some significant performance improvements during the last days. You might want to update to the most recent development version, which could already help your workflow.

On top of that, more performance improvements are already in the pipeline.

[lenhard]

With the merge of #3333, we have driven down the memory consumption in the entry editor significantly. With the new memory profile, GC tends to kick much sooner and more often than previously (at least on some systems). Hence, this issue can be closed.

In case of new performance problems, we can open a new issue.

[j0hannes]

@lenhard The new version runs smoothly on Linux. I've even seen the process giving back memory to the system. On OS X, I don't see memory issues with the new version either. It tends to freeze frequently, but that's probably a different story.

[lenhard]

That is great to hear. Thank you very much for reporting!

[javicacien]

This issue also appears in the release version. When JabRef 4.0 starts its allocated memory is 363MB and keeps climbing all the time. After 30-40min JabRef becomes so slow that it results necessary to force closing it.
My machine:
Intel core i7 with 8GB RAM at 2.93GHz
Windows 7 enterprise 64 bit
Windows 7 6.1 x86
Java 1.8.0_151
JabRef 4.0

[lenhard]

@javicacien Yes, of course the problem is present in 4.0. That is exactly what this issue was about.

The problems described are fixed in the current dev version, which is not yet released. That version is available at http://builds.jabref.org/master/

[javicacien]

Sorry, I misunderstood the posts, I was thinking that they were referring to the "dev" version prior to 4.0

[wilthan]

The memory problem is still not solved in JabRef 4.2 )Windows 7 6.1 amd64, Java 1.8.0_161)
I start the process with com 360MB and after working for a while on my database-file with 12k entries the process is at 6GB ( My machine has 16GB RAM)

[mgmverburg]

Currently my JabRef is using 2.5GB of RAM for a library with less than 50 entries. I only installed and started using JabRef yesterday and today started wondering why my PC became slow (8GB of RAM only currently), and noticed JabRef was using this much.

JabRef 4.3.1
Windows 10 10.0 amd64
Java 1.8.0_191

[Siedlerchr]

Please use the development version available at http://builds.jabref.org/master/
Please make a backup of your library.