OSS Detect Backdoor goes OOM on a package

[maxboone]

On the package https://github.com/lxyeternal/pypi_malregistry/tree/main/aioconsol the OSS Detect Backdoor goes haywire on my system and eats up all the memory in seconds. This happened on multiple (x86) machines using the latest binary release.

Possibly there is a memory leak scanning there?

[scovetta]

Wow, thanks for reporting this. I was able to repro, ate up 126 GB of memory before getting killed. It's definitely a bug, getting stuck in the 22 MB long line in the setup.py file.

Not sure if this is within the regex engine or our code, but either way, we'll take a look.

[gfs]

My suspicion would be one of the patterns used is exponential. Maybe worth trying enabling non-backtracking mode for the app inspector engine in characteristic tool.

[gfs]

Starting to investigate this, I've tested with Non-backtracking regex (WIP commit: #453) and that seems to resolve the memory usage/infinite loop issue, but this sample also revealed an unrelated issue in ApplicationInspector (microsoft/ApplicationInspector#571). Will need to incorporate the new version of appinspector with that fix to be fully confident that the non-backtracking change will resolve this, and will require a little plumbing work to build the tools with newer .net and likely expose the configuration parameter for non-backtracking as well. Likely will be able to resolve this this week.

[gfs]

I believe the linked change in #453 should fix this. I scanned all 3 versions of the package mentioned above at the same time (not sure if its only an issue with a specific version) and see a max memory usage of 8GB for ~2 million matches.

[gfs]

We had some pipeline issues that should now be resolved and the fix for this is now available in the latest release. If you can still replicate the issue please reach back out. Thanks.