Dependency-Track takes over 30 minutes to process BOM and Jenkins plugin times out #2857
Open
2 tasks done
Labels
Comments
|
Additional Details
Could you please: HTTP Access Logs Jenkins logs Dependency-Track Logs |
|
Is this happening consistently, and is reproducible, or was this a one-time occurrence? Unless the metrics update is putting your database under heavy stress, it should not have caused the stalling you observed. What's interesting is that you are getting 500s from OSS Index occasionally: Dependency-Track uses a retry mechanism with exponential backoff in case requests to OSS Index fail. You can control the parameters for that behavior via application properties / env variables: dependency-track/src/main/resources/application.properties Lines 402 to 415 in a7e5122 If you suspect this could be the reason, you could try setting |
2 tasks
|
@nscuro this is happening periodically, but consistently. The database never appears to be significantly stressed. Is this error logged only when all retry attempts have been exhausted? Can you speculate on the consequences of changing these backoff settings? It seems this vulnerability source is being skipped anyway when the HTTP 500 is persistent. |
|
We also see this behavior so it's a little comforting to see we are not alone. I had to remove the checks in CI. |
|
So few things I am looking at here:
@37b That's honestly sad to hear. I'm hoping I can address this is 4.9. Worst case we at least have more info as to where the problem is exactly. |
|
I think (although not 100% sure) we are facing the same issue. We are creating and uploading SBOMs to a project on a daily basis. We then wait for the SBOM to be processed (using /v1/bom/token/{uuid} API calls) after which we apply additional VEX documents. Sporadically, waiting for the SBOM will timeout after 20 minutes. I am yet to find anything useful in the logs regarding this. |
|
@Jasper-Ben Yeah that's entirely too long. Do you have Prometheus monitoring set up? As of 4.10 (I believe) we expose metrics as to how long processing of certain events takes: https://docs.dependencytrack.org/getting-started/monitoring/#event-and-notification-system If there's something blocking for >= 20min you should be able to observe a spike in those metrics. The example Grafana dashboard has a widget for this particular metric, see #3173 (comment) |
Ah sweet! Yes, we do. Unfortunately, checking them for the first time now, this is how the metrics currently look like for DT 😅:
I am in the process of updating DT 4.10.0 to 4.10.1, in case that makes any difference. If not, then I'll have to figure out why the metrics do not end up where they should be. Noteworthy: We are currently using the Helm chart from SaturnMediaMarkt, maybe there is something wonky with the metrics stuff there, but we'll see. |
|
Ok, the Prometheus metrics are fixed, so I can start monitoring the issue now 😅
- Our Kubernetes Cluster probably |
|
Forgot to link it, but this issue seems at least partially related to #3454, which will impact large(r) portfolios more than smaller ones. |
|
We only have 4 or so projects in DT, so I don't think it is relevant in our case
|
|
Just pinging @nscuro to let you know that we have yet to encounter this problem again since the update from 4.10.0 to 4.10.1 |
|
Thanks for the update @Jasper-Ben! But I don't think we (knowingly) changed anything in that version that could be related. But hopefully the next time it appears, you'll be able to spot some irregularities in the metrics that could provide further hints. |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behavior
Jenkins pipeline job with Depndency-Track step times out after 30 minutes (Java: 206 components)
Steps to Reproduce
Run Jenkins job with a step that calls dependencyTrackPublisher() from the OWASP Dependency-Track Plugin
Expected Behavior
Should complete in under a minute.
Dependency-Track Version
4.8.0
Dependency-Track Distribution
Container Image
Database Server
PostgreSQL
Database Server Version
No response
Browser
Google Chrome
Checklist
The text was updated successfully, but these errors were encountered: