Suggestions on improvement for memory performance regarding Regex matching

[georgetayqy]

What feature(s) would you like to see in RepoSense

Currently, some areas of RepoSense use Regex checks within iterations.

For example, consider StringsUtil::filterText.

public static String filterText(String text, String regex) {
    String[] split = text.split("\n");
    StringBuilder sb = new StringBuilder();
    for (String line: split) {
        if (line.matches(regex)) {
            sb.append(line + "\n");
        }
    }

    return sb.toString();
}

As observed, the method String::matches is called for as many times as there are lines in the input text, which is not performant as String::matches creates and compiles a new Regex pattern every time it is called. This behaviour is evident in the source code for the String::matches method, which makes a call to Pattern.matches(regex, string).

It is recommended by the documentation for the Pattern class as well as some external sources to avoid such a pattern of code since it causes many unnecessary Regex compilations, which takes more time and consumes more memory.

After implementing some fixes for this behaviour, and from some preliminary profiling using Intellij's built-in Java Profiler, I have observed that the improvements lead to a small decrease in runtime and a significant decrease in memory usage when Regex patterns are precompiled first before being used in a for loop.

Here are some of my findings:

Command used to test

./gradlew run -Dargs="--repos https://github.com/reposense/RepoSense.git https://github.com/CATcher-org/CATcher.git https://github.com/MarkBind/markbind.git --output ./report_folder --since 31/1/2017 --formats java adoc xml --view --ignore-standalone-config --last-modified-date --timezone UTC+08 --find-previous-authors"

Runtime

Before Improvement	After Improvement

Some key points to note:

• The runtime being inspected is the CPU time (time that the code is executed, ignoring gradle builds)
• 11.69% decrease in runtime

Memory Consumption

Before Improvement	After Improvement

Some key points to note:

• 43.32% (!) decrease in memory usage
• It also appears that StreamGobbler consumes a significant chunk of memory, perhaps we could also look into reducing the memory usage of that class?

---

Limitations

• These preliminary profiling tasks are done on a M1 Mac with MacOS. These performance values may vary across different OS and processors
• Only 3 profiling samples were taken, more samples might have to be taken, across multiple platforms and processors, to decide if these enhancements have a positive impact on the performance of RepoSense

Is the feature request related to a problem?

This feature is not exactly related to a particular problem in general, but it is linked to the overall goal of increasing RepoSense's performance and memory usage.

If possible, describe the solution

The solution involves the pre-compilation of the offending Regex pattern into a Pattern object, and using a Matcher object created from Pattern to test if the string matches the Regex pattern.

The fixes are found in a draft PR here.

If applicable, describe alternatives you've considered

No other viable alternatives have been explored or tried at this time. It seems on preliminary inspection that these Regex operations are crucial and cannot be removed or replaced with other built-in String methods.

Additional context

N/A

[gok99]

Great findings @asdfghjkxd! Perhaps you could create a separate issue for investigating StreamGobbler.

[georgetayqy]

@gok99 Thank you! I will work with the team to create another issue to investigate StreamGobbler and find out why it consumes a significant amount of memory.

Regarding the link, it should be fixed. Apologies if it didn't work in the first place, the draft PR was not created before this issue and hence I couldn't link to it initially!