Replies: 27 comments 58 replies
-
|
I get a similar error.
Can someone please look into this ASAP? |
Beta Was this translation helpful? Give feedback.
-
|
I am experiencing the same problem, even after upgrading to version 0.56.1: After running my CI/CD pipeline 8 times, it finally worked, but repeatedly rerunning it shouldn't be the solution. |
Beta Was this translation helpful? Give feedback.
-
|
Exactly the same problem here |
Beta Was this translation helpful? Give feedback.
-
|
The rate limits are applied per organization, so creating a new repository (e.g., This is happening just because Trivy has too many users and reached the rate limits. The same problem could occur in other registries. We are trying to find a way, but currently, the only ways are to use ECR Public (we're still evaluating ECR Public, though) or self-host DBs on another registry, including a separate GitHub organization like this. |
Beta Was this translation helpful? Give feedback.
-
|
@knqyf263 thanks for your reply but... trivy is a security tool, you can not seriously ask users to self-host the list of vulnerabilities 🥹 Or to use mirrors not managed by trivy team, that would add security risks to a security tool 🥹 If ECR hosting is stable, it should be called by default: this is not for a user to decide which trivy vuln database to use, but for trivy to enforce to its users which database to use 🥹 |
Beta Was this translation helpful? Give feedback.
-
Why not? |
Beta Was this translation helpful? Give feedback.
-
|
I agree with @knqyf263 - especially for everyone relying on the open source tooling and not paying to cover the costs of ensuring access. I agree that provenance validation for the databases is on the critical path, but replicating from a known good source announced by the team to a private registry that is in the control of the consumer helps to mitigate the load concerns and isn't any worse than what exists today. Of course consuming anything from unknown random mirrors without validation is always a risk - and the deeper question is how does Trivy itself validate the vulnerability databases it is fetching and using - I haven't looked, but if it doesn't, then the well could always be poisoned and that is the much deeper concern to address. |
Beta Was this translation helpful? Give feedback.
-
|
I am myself maintainer of MegaLinter which is free and open source tool: this is not about open source or not, this is about breaking changes, security and ease of use :) MegaLinter works because it's easy to setup and configure, if tomorrow I force my thousands of users to mirror docker images so it continues to work, they'll be very unhappy, even if they don't pay 🙃 |
Beta Was this translation helpful? Give feedback.
-
|
Over the past 1-2 days, I've started seeing CI/CD scan job failures with this error as well in our on-prem infrastructure (with a GitLab Runner using the Kubernetes executor). A big difference is that we scan via a Trivy server deployment (using the Trivy Helm chart), so the errors don't arise from pulling Java DBs, etc. |
Beta Was this translation helpful? Give feedback.
-
|
Actually, inspecting our Trivy server deployment logs, I'm also seeing some errors there, of the sort discussed above, e.g., But this doesn't break our scans, since the old DB etc. is still available in the Trivy server. |
Beta Was this translation helpful? Give feedback.
-
|
I set up a new deployment for trivy but the operator won't work because every single image pull attempt fails for me with TOOMANYREQUESTS :( |
Beta Was this translation helpful? Give feedback.
-
|
Same errors in our GitLab pipelines. v0.56.1 @knqyf263 Any info/docs on how exactly to configure trivy to do the workaround you mentioned (ECR)? Or better to wait? |
Beta Was this translation helpful? Give feedback.
-
|
This is a workaround, not a solution , because it still requires to change the same way all calls to trivy in the whole universe :/ |
Beta Was this translation helpful? Give feedback.
-
|
The Trivy Operator helm chart does not support multiple repositories. Is it acceptable to use only public.ecr.aws/aquasecurity/trivy-db and public.ecr.aws/aquasecurity/trivy-java-db ? |
Beta Was this translation helpful? Give feedback.
-
|
@ErikLundJensen Yeah, using one is just fine; the CLI commands were updated to accept a priority list, so supplying both just tells it to try 3562529#diff-7bfcac56497a113e6a3d15e967ba52d22a30c232a0b641728eda7dc85fe32137R32 |
Beta Was this translation helpful? Give feedback.
-
|
Trivy version: Setting these variables seems not to have worked, the fallback is not used in our case: |
Beta Was this translation helpful? Give feedback.
-
|
williamwissemann's suggested workaround works fine for us. docker run \
--rm \
-v "$(realpath .):/opt/src" \
-v /run/docker.sock:/var/run/docker.sock \
-v /tmp/trivy-cache:/cache \
-e "TRIVY_DB_REPOSITORY=public.ecr.aws/aquasecurity/trivy-db" \
-e "TRIVY_JAVA_DB_REPOSITORY=public.ecr.aws/aquasecurity/trivy-java-db" \
-w /opt/src \
aquasec/trivy:0.56.2 --cache-dir /cache image --quiet "$image"Et voilà. |
Beta Was this translation helpful? Give feedback.
-
|
Just adding my context: We're moving to using private AWS ECR Pull Through Caches for containers so that we have more visibility into the containers that are running, and so that we can disconnect environments where things are running from having public internet access. I'd love if a job running trivy could do everything it needs to do, without connecting to the public internet. AWS requires credentials to set up these caches for the upstream, and does sane things with ratelimiting. I was already working to have trivy pull things through caches instead of "upstream", but there is different config required for each of the different ways of using trivy (config, image, etc). I'd be totally happy for trivy to have a single configuration method for "use_this_repo" to allow us to maintain our own cache, but it'd be even better to have a |
Beta Was this translation helpful? Give feedback.
-
|
@chriskellyrelay
|
Beta Was this translation helpful? Give feedback.
-
|
It's a complex thing. |
Beta Was this translation helpful? Give feedback.
-
|
I'd like to add one more aspect that was touched upon but not made explicit by several comments above. If registries like GHCR and ECR make it difficult for Trivy users to access the DBs during high load, then why not ensure that the latest DB images are also hosted on Docker Hub? When they started their rate limiting nearly four years ago, the tooling and hosting ecosystem quickly adapted and introduced features like Harbor's Proxy Cache, GitLab's Dependency Proxy, and others, which are designed to mitigate upstream registry rate limiting, but work specifically and exclusively with Docker Hub at the moment, and not with GHCR etc. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Have the same problem but with the bundle-checks repository when running misconfiguration. |
Beta Was this translation helpful? Give feedback.
-
|
With misconfiguration scanner, the checks bundle is embedded within Trivy. While its not ideal that it fails to download latest, the scan will still be successful with the embedded version of the checks that Trivy contains. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
I am using the helm chart to deploy the operator on EKS and pointing to the AWS ECR for images and vulnerability DBs. For some reason, it still keeps going to ghcr.io. The CM seems to have been updated properly though. Below is my values.yml And below is the error I am seeing still |
Beta Was this translation helpful? Give feedback.
-
|
You are specifying the wrong path in your values. To adjust the registry for
|
Beta Was this translation helpful? Give feedback.
-
|
Most people here are talking about issues with the |
Beta Was this translation helpful? Give feedback.
-
|
@knqyf263 any chance this image can get pushed up too? would you be open to a pr in checks that replicates this? https://github.com/aquasecurity/trivy-db/pull/440/files Or is there other behind the scenes reasoning to these images not yet being hosted? |
Beta Was this translation helpful? Give feedback.
-
|
Ah infact an open pr exists already aquasecurity/trivy-checks#262 |
Beta Was this translation helpful? Give feedback.
-
|
i have the same issue with latest trivy version while executing stage with jenkins pipeline: |
Beta Was this translation helpful? Give feedback.
-
|
This is still an issue for me and my team. We updated all commands in our CI/CD scripts to specify the AWS ECR ( This tool is a critical part of our CI processes and it is very difficult for us to remove it. Are there any other workarounds for this issue that we can try? |
Beta Was this translation helpful? Give feedback.
-
you may have forget setting --java-db-repository as well. Basically : --db-repository public.ecr.aws/aquasecurity/trivy-db and -java-db-repository public.ecr.aws/aquasecurity/trivy-java-db |
Beta Was this translation helpful? Give feedback.
-
|
Ah theres a different flag for the Java DB! My bad, I'll try that out. |
Beta Was this translation helpful? Give feedback.
-
|
This is still an issue for us. Is there any timeline of when this issue will actually be solved? |
Beta Was this translation helpful? Give feedback.
-
|
@daehiff You've reached your limit as an anonymous user. Try authenticating for ECR. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick response. Can I reach this limit when being authenticated? |
Beta Was this translation helpful? Give feedback.
-
|
@nikpivkin so we used an aws account, but still ran into the same rate limit. This is quite blocking for us, is there any other option asides from rotating accounts? |
Beta Was this translation helpful? Give feedback.
-
|
Even with the workaround of setting Example command: $ TRIVY_DB_REPOSITORY=ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db trivy fs .The failures happen with both and with only one of the repos enabled:
Failure example: Trivy version: |
Beta Was this translation helpful? Give feedback.
-
$ trivy --version
2024-10-25T13:39:25+02:00 INFO Loaded file_path="trivy.yaml"
Version: 0.56.2
Vulnerability DB:
Version: 2
UpdatedAt: 2024-10-25 06:16:24.134613424 +0000 UTC
NextUpdate: 2024-10-26 06:16:24.134613063 +0000 UTC
DownloadedAt: 2024-10-25 11:33:33.169512 +0000 UTC
Java DB:
Version: 1
UpdatedAt: 2024-10-21 02:52:13.021715717 +0000 UTC
NextUpdate: 2024-10-24 02:52:13.021715596 +0000 UTC
DownloadedAt: 2024-10-21 15:42:53.42322 +0000 UTCUpdated original comment. |
Beta Was this translation helpful? Give feedback.
-
|
If you specify multiple repositories, Trivy will try to load them all. You have only one repository in your logs. 2024-10-25T17:40:18+06:00 INFO [vulndb] Downloading vulnerability DB...
2024-10-25T17:40:18+06:00 INFO [vulndb] Downloading artifact... repo="ghcr.io/aquasecurity/trivy-db:2"
2024-10-25T17:40:19+06:00 ERROR [vulndb] Failed to download artifact repo="ghcr.io/aquasecurity/trivy-db:2" err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 722.467µs, allowed: 44000/minute\n\n"
2024-10-25T17:40:19+06:00 INFO [vulndb] Trying to download artifact from other repository...
2024-10-25T17:40:19+06:00 INFO [vulndb] Downloading artifact... repo="public.ecr.aws/aquasecurity/trivy-db:2"
54.81 MiB / 54.81 MiB [-----------------------------------------------------------------------------] 100.00% 5.50 MiB p/s 10s
2024-10-25T17:40:32+06:00 INFO [vulndb] Artifact successfully downloaded repo="public.ecr.aws/aquasecurity/trivy-db:2" |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, because that's not how trivy behaves here. Note how the command enables two repositories, yet trivy terminates after the first one fails to download. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, this was a false positive. Your comment #7668 (reply in thread) pointed me in the right direction. In our case, trivy is run in a scripted setup. There was a problem that the TRIVY_DB_REPOSITORY variable got scrambled, which resulted in "losing" the second repository in the list. This is now fixed in our setup, and if Sorry for the false report, and thanks a lot for your help to resolve this! |
Beta Was this translation helpful? Give feedback.
-
|
I run Trivy in a similar way: Is it in your logs? 2024-10-25T17:40:18+06:00 INFO Adding schema version to the DB repository for backward compatibility repository="ghcr.io/aquasecurity/trivy-db:2"
2024-10-25T17:40:18+06:00 INFO Adding schema version to the DB repository for backward compatibility repository="public.ecr.aws/aquasecurity/trivy-db:2" |
Beta Was this translation helpful? Give feedback.
-
|
Hello @nikpivkin Could you please guide me on this? |
Beta Was this translation helpful? Give feedback.
-
|
What version are you using? Support for multiple repositories was added in 0.56.0 |
Beta Was this translation helpful? Give feedback.
-
|
A lot of this doesn't make sense to me, my guess is the whole reason the databases are being distributed like this, is its some scheme to avoid paying transit costs, that has obviously broken down now that GitHub is rate limiting. This would probably have been better done by distributing the file via HTTP, via a good CDN like Cloudflare, where there are better tools for caching available. My workaround is to use an AWS ECR pull-through cache rule: You just need the public_repo permission on the GitHub access token, and the token is required, but unfortunately having a token doesn't get you past the rate limit, because thats being imposed on the aquasecurity organization. One you have created the cache rule you need to "prime the pump" since you may be rate limited several times before getting a successful pull. When successful, you will see the message: "unsupported mediatype application/vnd.oci.empty.v1+json" because the Docker tools do not know how to parse these particular container images. If you are interested in inspecting the contents, use the command Once the images exist you can do something like: And you have to repeat that for every account/region combo where you need it because cross-account/cross-region access to ECR seems to be hard or not supported. But this is a PITA to have to set up and maintain so longer term I think aquasecurity ought to re-think if this overly elaborate scheme to distribute tar files is really the right way. |
Beta Was this translation helpful? Give feedback.
-
|
I get this error on CI even when the last run was a day ago. Maybe it's time to setup mirrors instead of just relying on GitHub. I'm up for hosting one. |
Beta Was this translation helpful? Give feedback.
-
|
cc @knqyf263 , a good samaritan offers to host a mirror :) |
Beta Was this translation helpful? Give feedback.
-
|
My workaround for those with their own OCI compatible image registry (such as harbor).
oras doc: https://oras.land/docs/commands/oras_cp/ |
Beta Was this translation helpful? Give feedback.
-
|
Just to add my feedback, in order to be able to pull down from the aws ecr I had to do a firewall request. Which worked. but its now failing on trying to pull down one of the layers of the image - (snippet from the error) - oci download error: failed to fetch layer : GET https://d2glxk2uabbnd.cloudfront.net/..... now clearly that url is blocked at present. Now this isn't an issue I have seen before with the normal trivy db download... So any explanation or indeed expectation around additional firewall permissions be useful.. That said I think I'll just implement what @resamaraschi has done above. |
Beta Was this translation helpful? Give feedback.
-
|
Tagging my discussion because @miguno said above
but that doesnt seem to be case from what Ive seen in my personal git pipeline. |
Beta Was this translation helpful? Give feedback.
-
|
Still getting this error as of today. |
Beta Was this translation helpful? Give feedback.
-
|
Same |
Beta Was this translation helpful? Give feedback.
-
|
Can someone please cleanup this discussion? Finding the actual solution/workaround from #7668 (reply in thread) is quite hard... |
Beta Was this translation helpful? Give feedback.
-
|
Such workaround doesn't always work, and is just a workaround, not a fix :/ |
Beta Was this translation helpful? Give feedback.
-
|
The workaround works fine for me when not downloading from GitHub as primary. TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db,aquasec/trivy-db,ghcr.io/aquasecurity/trivy-db
TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db,aquasec/trivy-java-db,ghcr.io/aquasecurity/trivy-java-db |
Beta Was this translation helpful? Give feedback.
-
|
Additional note: There is a PR in progress that aims to fix these problems: #7679 |
Beta Was this translation helpful? Give feedback.
-
|
It is 28 days old, do we have an ETA ? :) |
Beta Was this translation helpful? Give feedback.
-
|
I encountered the same issue while using GitHub Actions. Caching the Trivy database in the GitHub Workspace and reusing it during the scan process resolved the problem for me. Here's the caching guide: https://github.com/aquasecurity/trivy-action?tab=readme-ov-file#updating-caches-in-the-default-branch |
Beta Was this translation helpful? Give feedback.
-
|
We are using this setup now, too. That is, 1x workflow to download and cache the Trivy vulnerability database(s), and 1x workflow to run the actual scans using the cached database(s). The caching guide link explains this setup in copy-pastable detail. |
Beta Was this translation helpful? Give feedback.
-
|
Works only on github :/ |
Beta Was this translation helpful? Give feedback.
-
|
This worked the first day but started failing the second day... seems like it still tries to pull, even though we used the caching example (and see that it has cached the database?) |
Beta Was this translation helpful? Give feedback.
-
|
Same as @PapaPeskwo, I cached and thought "yeah problem solved". Until a few mins ago, I got again the |
Beta Was this translation helpful? Give feedback.
-
|
Same issue with GitlabCI, this is starting to get very problematic. |
Beta Was this translation helpful? Give feedback.
-
|
TOOMANYREQUESTS error is also happening to me today. |
Beta Was this translation helpful? Give feedback.
-
|
Jenkins Pipeline: Same issue..... Why not have a user create a free user account, confirm email. and then distribute a token ? Attach the token to the ENV variables and have trivy pick it up.. 16:29:41 2024-11-12T13:29:41-08:00 INFO [vulndb] Need to update DB |
Beta Was this translation helpful? Give feedback.
-
|
@DmitriyLewen What do you suggest if someone is using a pipeline in GitLab or Jenkins and this issue is occurring more frequently? |
Beta Was this translation helpful? Give feedback.
-
|
The "quick" workaround has been suggested above already - #7668 (reply in thread).
|
Beta Was this translation helpful? Give feedback.
-
|
@Darwiner - would aws pull through cache mechanism work? Use GHCR as the upstrea url? And then I can point my trivy to use this instead? |
Beta Was this translation helpful? Give feedback.
-
I haven't tried it, but according to a few comments above, it does seem like it could/should work. |
Beta Was this translation helpful? Give feedback.
-
|
can confirm it worked:
The only "Issue" with this is that you'll need to create a jenkins job to run step 3 every 6 hours or so because pull-through cache isn't smart enough to monitor the ghcr repository for changes and automatically pull the latest image. |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, we found 1 more way - #7538 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
v0.57.1 with |
Beta Was this translation helpful? Give feedback.
-
|
been using it for a few days now and seems to work. thx |
Beta Was this translation helpful? Give feedback.
-
|
@knqyf263 please would it be possible to merge my PR in trivy-action , so the fix is applicable to the github action ? |
Beta Was this translation helpful? Give feedback.
-
|
I can also confirm that switching to |
Beta Was this translation helpful? Give feedback.
-
|
Please see the summary of this issue, including resolution: #8009 |
Beta Was this translation helpful? Give feedback.
and others
-
Description
Hi, I'm the maintainer of MegaLinter, which heavily uses trivy in its core, but also in its own CI/CD jobs
For some weeks now, there are many fatal errors because of rate limit when downloading database, and this is impacting MegaLinter stability, at a point that we have to advise to disable trivy "until it is stable again"
init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:9eed63035e251610986ab4666a7ef45cfcf74e7f02daee94fe9958d1f7f2749b: TOOMANYREQUESTS: retry-after: 544.488µs, allowed: 44000/minuteI've read all issues & discussions about it, and all I see are workarounds
MegaLinter users can run MegaLinter on:
So the GITHUB_TOKEN solution does not work.
People also have existing MegaLinter config with their own arguments to call trivy, so I can't either force the AWS image to be used
The best solution I imagine is the following:
That way, Trivy would continue to work great without forcing user to implement clumsy workarounds :)
Do you think it's doable ? :)
Desired Behavior
Trivy to work perfectly everytime as before :)
Actual Behavior
If you are lucky, it work, but if you are not , it fails :/
Reproduction Steps
Target
Filesystem
Scanner
Vulnerability
Output Format
None
Mode
Standalone
Debug Output
`init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:9eed63035e251610986ab4666a7ef45cfcf74e7f02daee94fe9958d1f7f2749b: TOOMANYREQUESTS: retry-after: 544.488µs, allowed: 44000/minute`Operating System
Alpine & ubuntu
Version
Checklist
trivy clean --allBeta Was this translation helpful? Give feedback.
All reactions