Memory leak in GUI [timebox: 4.5 days]

[FreddieAkeroyd]

After monitoring the GUI with VisualVM the enclosed trace was attached. It seems that at 15:03 on 1st July the GUI allocated a lot of memory and went into a garbage collect loop high cpu loop that froze the GUI.

[FreddieAkeroyd]

Note from scientist "I tried opening the reflectometry server in a device screen and this caused it to freeze. In the end I needed to kill the client and restart it."

[FreddieAkeroyd]

visual VM will be installed locally on polref, also gui patched to allow nagios monitoring

[rerpha]

visualvm installed on polref and inter along with gui patch for jmx handles

[FreddieAkeroyd]

Nagios now monitoring NDXPOLREF and NDXINTER, "IBEX CLIENT Java Heap Memory" item

[Tom-Willemsen]

This wasn't really a memory leak, but rather some highly excessive resource consumption by the reflectometry OPI, caused by large numbers of rules being implemented in javascript.

I've added a mechanism that lets us completely bypass the javascript layer for a number of the most common rules. Most rules actually use a relatively small number of distinct conditions (see output of grep -roP "bool_exp=\".*?\"" | cut -d ":" -f 2 | sort | uniq -c | sort -nr). By implementing these common conditions in java, and only falling back to javascript for uncommon conditions, we can save a lot of memory and cpu.

This was exacerbated on INTER and POLREF by previously allowing them to open the reflectometry OPI in three different places - refl perspective, device screen, and synoptic. At ~400MB each time, this got POLREF to ~1.2G of memory used purely for the reflectometry OPI, and INTER similarly got to ~800MB (they only had perspective + synoptic, not device screen). It is then easy to see how INTER ran into their 1G heap limit with a little consumption from other OPIs/scripting etc. I believe this has since been fixed, and the REFL opi can no longer be opened from multiple places. Combined with the changes in this ticket, this should mean the reflectometry opi now uses reasonable amounts of resources.

---

Although it wasn't a primary goal of this ticket, I also did a quick benchmark of CPU time using the old & new rule implementations. For executing the following (reasonably representative) rule:

importPackage(Packages.org.csstudio.opibuilder.scriptUtil); 
var pvInt0 = PVUtil.getLong(pvs[0]);
if(pvInt0 == 1)
	widget.setPropertyValue("border_color",ColorFontUtil.getColorFromRGB(255,0,0));
else
	widget.setPropertyValue("border_color",ColorFontUtil.getColorFromRGB(0,128,255));

• For the first execution (e.g. when OPI first opens & pvs connect), RhinoWithFastPath takes 27 μs, while executing with JS takes 45407 μs on my machine. This represents a performance increase of about 1645x.
• For subsequent executions (e.g. when a linked PV updates), RhinoWithFastPath takes 14 μs while executing with JS takes 2792 μs. This represents a performance increase of about 200x.

The significant difference between first and subsequent executions for JS is because it needs to compile the JS and then run it, whereas subsequent iterations can reuse a cached result of the compilation.

---

PRs:

• Reflectometry performance ibex_gui#1458
• Implement fast path cs-studio#9
• https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/OPI-programming-tips-and-limitations#improving-the-runtime-performance-of-rules-in-opis
• Ticket7212 #7279