-
Notifications
You must be signed in to change notification settings - Fork 318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OutOfMemoryError when running the scanner #3091
Comments
Hello, just for your information, I'm using for some projects up to 10g ! |
Thanks for your feedback, @fb33. |
it works with docker. I pass it with an env var.
where $ORT_JAVA_OPTS contains the options |
I'm running some tests with the newest version (supporting ScanCode 3.2.1rc2) to check if the problem persists. |
When I run the scanner on my project for the first time, it will take about 20 hours and will finish successfully. If I run the scanner a second time on the same project, leaving any project files, ORT caches, and downloaded files untouched, it will crash due to insufficient memory as described above. This seems weird to me as the second run should use the cached results and finish quickly. Any ideas? |
Interesting. Which scan storage have you configured? "FileBasedStorage with XZCompressedLocalFileStorage" as mentioned above? |
I assume I use the default local file storage as I have no
but this seems to be something else... |
Ok, I was asking because our log above shows
But now you say you have |
Sorry, I think I mixed up the scanner archive storage and the scanner storage. Based on this reference config file, they are not the same. I'll try to add a scanner storage with no compression and run the scanner again. |
Same result with the uncompressed local file storage: the second run terminates with an OutOfMemoryError. Do you think it might make sense to try the Postgres storage to avoid such an error? |
Using Postgres anyway is a good idea, and it might help us to narrow down the root cause, so yes. |
Alright, give me about 24 hours and I'll report back. |
Same result with Postgres, unfortunately. So it does not seem to be related to the file storage. Naively, I'd expect the initial scan to use more memory than the subsequent one (which uses cached results), yet the initial scan runs fine whereas the second runs crashes. Is there anything else I could test? |
Again, interesting. To me that means that the potential memory leak could be in the general code that deserializes already stored scan results. Is there a way you could profile memory use / create a heap dump and inspect which objects take up that much memory? |
I'll see what I can do. Meanwhile, here's the log output for the second scan: scan.log I removed the irrelevant parts in which the scanner looks for stored results. What's interesting to me is that the ScanCode version is requested multiple times before the OutOfMemoryError occurs. |
I see that you rely on ScanCode being bootstrapped, which is something I do not recommend to do, mostly for performance reasons. Could you install / configure ScanCode 3.2.1rc2 beforehand, put it in |
Thanks for all of your recommendations. I finally made it work by setting my I used VisualVM to monitor the memory consumption of the ORT scanner. The scan runs for about 10 minutes. The consumed memory is quite low (< 300M) for the first 5 minutes, then it quickly goes up to about 1G, and a minute later it jumps to about 7.5G - all while it is looking for stored scan results. I could provide a heap dump if you're interested but the file is about 1G. The top 3 classes by size of instances are
I don't quite understand why the memory consumption jumps significantly or why this second scanner run takes up much more memory than the first run, but for now I'm happy it works at all. Thanks again for your support. It helped me a lot. |
Maybe it would be good to archive that heap dump for a little while, if possible. We have some (probably unrelated, but still) changes to the scan storage coming soon; maybe you could also test whether the issue can still be observed afterwards. |
Sure, no problem. Let me know when I can run a re-test. |
@mnonnenmacher realized that in many stack traces So, can any of the affected reporters please check if the |
Thanks for the feedback. I'm running a scan of a large project using the ORT code, and I will report back when it's done. |
My scan has finished successfully, I have seen no OOMEs. Thanks! |
Quite remarkable, thanks for testing! I'll still keep this open for a bit more time, until we're certain the issue does not occur anymore. |
Hello, |
FYI, additionally @oheger-bosch has filed a PR at #3412 that aims to reduce the scanner's memory usage in general. |
Regarding the So each usage of the |
Sorry for getting a bit off-topic, but FYI, the reason why files are used in the first place to redirect output is to avoid the need for "pumper" threads to pull data out of the |
I'm closing this for now; please reopen if the issue reoccurs. |
Hi,
I have a fairly large Maven project (about 500 entries in the dependency tree) that I want to scan with ORT. The analyzer works as expected and produces an
analyzer-result.json
file of about 1 M.When I run the scanner, though, It runs for a while and then terminates with the following error message:
The exact point of failure varies with every run, but generally it happens at a rather advanced stage of the scan (say at 392/486 packages scanned).
A typical stack trace looks like this:
But I have seem these stack traces, too:
(Sorry for pasting so excessively.)
Based on this and this issue, I put a
gradle.properties
next to thepom.xml
file of the project to be scanned and, to be on the safe side, I tried settingJAVA_OPTS="-Xms1024M -Xmx6g -XX:MaxMetaspaceSize=2g -Dkotlin.daemon.jvm.options='-Xmx2g'"
before running the scanner, but without success.Is there an error somewhere in my memory options? Do I need to allow for even more memory?
I work with the latest ORT code on Ubuntu 20.04 which runs in a Virtual Box environment under Windows. The VM has 10 G available as main memory. A colleague of mine tried to scan the project under Windows and ran into the same problem.
I did not experience this issue when I was scanning the same project with ORT a few months ago. I don't know what has changed since then.
I'd appreciate any help.
Thanks and keep up the great work.
The text was updated successfully, but these errors were encountered: