external-sources: GetMavenPackageBySha can be rate limited by maven central, grype will silently fail which results in inconsistent scan results
#2383
Labels
bug
Something isn't working
Comments
rawlingsj
added a commit
to rawlingsj/grype
that referenced
this issue
Jan 21, 2025
…ate limited for large sets of java depenencies When [external-sources](https://github.com/anchore/grype?tab=readme-ov-file#external-sources) are enabled if an Image contains a large number of Java dependencies Grype can get rate limited by maven central. This change will: - add a rate limiter to throttle requests at 300ms per second to produce reliable results - if a normal artifact not found error is returned by maven central the existing debug logging happens - any other error from maven central will result in an error being logged - adds an integration test that can be used to verify the rate limiter so we can verify against the real external api settings Related to issue anchore#2383 Signed-off-by: James Rawlings <[email protected]>
|
Also, in case it is of any use, here's a link to a branch I used to debug why scan results were missing on some runs rawlingsj@ed51394 |
|
I also came across #2216 which could help too, though I still think there's a risk of rate limiting. |
rawlingsj
added a commit
to rawlingsj/grype
that referenced
this issue
Jan 22, 2025
…ate limited for large sets of java depenencies When [external-sources](https://github.com/anchore/grype?tab=readme-ov-file#external-sources) are enabled if an Image contains a large number of Java dependencies Grype can get rate limited by maven central. This change will: - add a rate limiter to throttle requests at 300ms per second to produce reliable results - if a normal artifact not found error is returned by maven central the existing debug logging happens - any other error from maven central will result in an error being logged - adds an integration test that can be used to verify the rate limiter so we can verify against the real external api settings Related to issue anchore#2383 Signed-off-by: James Rawlings <[email protected]>
wagoodman
added a commit
that referenced
this issue
Jan 22, 2025
…ate limited for large sets of java dependencies (#2384) * external-sources: throttle requests to maven central to avoid being rate limited for large sets of java depenencies When [external-sources](https://github.com/anchore/grype?tab=readme-ov-file#external-sources) are enabled if an Image contains a large number of Java dependencies Grype can get rate limited by maven central. This change will: - add a rate limiter to throttle requests at 300ms per second to produce reliable results - if a normal artifact not found error is returned by maven central the existing debug logging happens - any other error from maven central will result in an error being logged - adds an integration test that can be used to verify the rate limiter so we can verify against the real external api settings Related to issue #2383 Signed-off-by: James Rawlings <[email protected]> * incorporate review feedback, add a unit test for the rate limiter behaviour Signed-off-by: James Rawlings <[email protected]> * fix linting issues Signed-off-by: Alex Goodman <[email protected]> --------- Signed-off-by: James Rawlings <[email protected]> Signed-off-by: Alex Goodman <[email protected]> Co-authored-by: Alex Goodman <[email protected]>
|
fixed via #2384 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
We are scanning using Grype with external-sources enabled which reaches out to maven central. For some applications that have a large number of java dependencies I noticed inconsistent scan results between runs. In one case a critical vuln was reported in derby for one run, the next run it wasn't listed by Grype which lead to confusion.
After digging into the code and hacking a little reproducer detailed below I found that maven central (or the ELB infront) will rate limit requests based on IP and return a 403, blocking requests for ~5 mins.
I also found that when these 403 errors occur there was no error logged, vulns that would have been discovered are just omitted.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
I've put together a little reproducer test that is not intended to be part of the code base, just shows the error we get today after around 2-3 iterations of requesting a large number of sha requests. The test can take ~ 4-7 mins to fail.
Reproducer Code
Adding to
./grype/db/v5/matcher/java/Anything else we need to know?:
I had a little tinker while doing the reproducer above and tried a few things like setting http client connection settings, keep-alive etc but nothing seemed to reliably work until I added a rate limiter to throttle requests. I'll pop a PR up to show where I got to as a potential solution and see what folks think.
Environment:
The text was updated successfully, but these errors were encountered: