Error while processing bom: Duplicate key #4027
Comments
|
This is really odd since we de-duplicate component properties multiple times such that there can't be more than one property for each I can not reproduce this with the component data you provided, by uploading a BOM with those multiple times to the same project. The fact that there are multiples of those combinations in your case leads me to believe there is some sort of race condition. Are you uploading multiple BOMs to the same project in close succession? Also, can you please share the component properties for the problematic component in DT? You can find them by clicking on a component in the list, then View Details, and then Properties on the bottom of the Component Details modal. There should not be any duplicate properties. |
|
I have a similar issue, yet the stack trace of the exception specifically refers to the The uploaded BOM represents a container image and is quite big. However, I checked the licenses and could not find any duplicate license with the same |
|
@turbolocust This is caused by #3957, which is already fixed via #3958. But the fix was not yet released. If this is a more widespread issue we'll definitely look into pushing another bugfix release. |
|
Hey! I may have a similar issue:
Dependency-Track v4.11.5 I generated the SBOM with https://github.com/anchore/syft |
|
I've tried to provide you with more detailed SBOM but couldn't replicate the issue. My fix: I've deleted the System.ValueTuple components and reuploaded the SBOM - it was processed successfully. Thanks for all the work for all DependencyTrack. |
|
If you are looking for a SBOM which causes this, this is mine: sbom.json
I just tried that. This didn't help in my case. |
|
Again, I have similar case to my other SBOM - maybe completely coincidence but this time the problematic component is: System.Security.AccessControl (thinking if the System.* maybe a problem?):
After removing the problematic Components new issue came (again System.*)
The same route and the same problem arises - all with System.* components in version 4.5.0, like those:
|
|
I also experience the duplicate key issue. I'm on the latest DependencyTrack version (v4.11.7) and I have the BOM Processing V2 experimental setup enabled. As a note, I only get this error for projects I generate the SBOM with anchor/syft tool |
|
I did a quick smoke test and I can reproduce the issue with @simonknittel's BOM. First upload succeeds, for the second the For anyone curious, the code in question is here: It fails on line |
|
Same issue in our setup:
|
|
I did some more testing yesterday. I was not able to reproduce it with BOM Processing V2 enabled. I also noted that the Syft BOM shared above has lots of duplicate components, which BOM Processing V2 does a better job of getting rid of. Upon first import, there were no duplicate properties in the database. During the second import, some components did have duplicates, which leads me to the assumption that properties are unintentionally being added outside of the intended "happy path" (a snippet of which I shared above). Since that code takes care of de-duplicating properties, there must be another place where properties end up being added without proper de-duplication. |
|
I just enabled the experimantal option for v2 processing but still run into the same error. However, when I delete the component it complains about, the next run it won't complain about that component again but a different one. First it was pkg:generic/6lowpan, then it was pkg:generic/8021q, now it is pkg:generic/8139too. I can probably continue deleting them until everything passes |
|
Yeah if there's already duplicate properties it will run into the same issue. But it should not cause new duplicates. |
|
I ended up just deleting all components for the affected project (nearly 3000) since deleting one by one was never ending. Now the import processed successfully |
I get the same issue within our setup ( Our Jenkins generates multiple Docker images that may share the same components. Multiple sboms of these Images (generated via anchor/syft) are - by chance - pushed in parallel to DT. So, if there is actually a race condition in deduplication we might need to change our process. |
|
@mbxsuite Projects in DT are not meant to hold more than one BOM at once. It doesn't do any merging - if you observed that happening before, it was by accident since there's currently no locking on the project level for processing uploaded BOMs. I think we need to add locking to make this more obvious. |
Ok, I may have described that not exactly enough: It is a single Jenkins Job to build and scan multiple Docker Images in parallel, from each Image one sbom will be generated and uploaded to DT. The Job is not waiting for results from DT. All images contain a c# microservice. Any service/image may contain and use shared nugets (models, etc). We don't push different sboms to a same project. Every docker image/microservice has it's own DT Project, so do the base projects that are scanned by CycloneDX. |
|
@mbxsuite OK, in that case you should be good process-wise. Now to address the issue at hand, do you have BOM Processing V2 enabled in the settings? If not, give it a try, as it appears to resolve this issue for new projects. As mentioned by @simonknittel though, it doesn't repair existing projects. I guess we need a solution for that still, as I don't think forcing users to delete components is acceptable. |
|
@nscuro I did both, enabled V2 now and we were able to delete the projects. Loooking at the issue, we stumbled across the naming of the detected components. Is it intended that the nuget package's name is used instead of the package id to identify it? vs |
|
Where is that output from? If it is from a log of some kind, the reason why dependency-track/src/main/java/org/dependencytrack/model/Component.java Lines 875 to 889 in 7ce8e04 |
|
I see. .... Origin was the Stacktrace of this issue, when analysing Docker->layer -> dll -> nuget-package The Github Purl package utilizes a "name", whereas c# / nuget world differentiates between a name and package id. There was no inherent requirement in our project to have the "name" unique, which led DT to seeing multiple different packages with similar Purls. We have now set names to equal the package ids and will see how this will behave. Thanks for the insight. |
|
We are facing the same issue with 4.11.4 |
|
I'll add a mechanism to v4.12 that will "auto-repair" this. |
nscuro
added
p2
Fixes DependencyTrack#4027 Signed-off-by: nscuro <[email protected]>
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Current Behavior
Hi,
Our process makes daily scans of all our projects, create the SBOM and load it to Dependency Track, Unfortunately, for some projects we are getting errors like this:
ERROR [BomUploadProcessingTask] Error while processing bom java.lang.IllegalStateException: Duplicate key Identity[group=aquasecurity, name=trivy:PkgID, [email protected]] (attempted merging values ComponentProperty{id=1450648, component=pkg:nuget/[email protected], groupName=aquasecurity, propertyName=trivy:PkgID, [email protected], propertyType=STRING, uuid=5e7640f8-2e76-4a09-8817-86cba50e0e05} and ComponentProperty{id=1450649, component=pkg:nuget/[email protected], groupName=aquasecurity, propertyName=trivy:PkgID, [email protected], propertyType=STRING, uuid=701f26af-fc1f-407e-966c-270c7e7325e8}) at java.base/java.util.stream.Collectors.duplicateKeyException(Unknown Source) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Unknown Source) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(Unknown Source) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) at org.dependencytrack.persistence.ComponentQueryManager.synchronizeComponentProperties(ComponentQueryManager.java:925) at org.dependencytrack.persistence.QueryManager.synchronizeComponentProperties(QueryManager.java:608) at org.dependencytrack.parser.cyclonedx.util.ModelConverter.convert(ModelConverter.java:613) at org.dependencytrack.parser.cyclonedx.util.ModelConverter.convertComponents(ModelConverter.java:464) at org.dependencytrack.tasks.BomUploadProcessingTask.inform(BomUploadProcessingTask.java:157) at org.dependencytrack.tasks.BomUploadProcessingTaskV2.inform(BomUploadProcessingTaskV2.java:151) at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Already there was one System.ValueTuple loaded in version 4.5.0:
In SBOM there was 4 separate instances of this component:
{ "type": "library", "bom-ref": "684a5aa1-535f-4fd2-ba87-3ca30b505e7c", "name": "System.ValueTuple", "version": "4.5.0", "purl": "pkg:nuget/[email protected]", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "nuget" } ] }, }, { "type": "library", "bom-ref": "a895ab7c-cee3-414b-8811-0c7a9f01b968", "name": "System.ValueTuple", "version": "4.5.0", "purl": "pkg:nuget/[email protected]", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "nuget" } ] }, { "type": "library", "bom-ref": "b4ace403-0b23-407c-9f9f-9a103de5d6f8", "name": "System.ValueTuple", "version": "4.5.0", "purl": "pkg:nuget/[email protected]", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "nuget" } ] }, { "type": "library", "bom-ref": "0ddf76c9-bc51-4287-81b8-4a7690cfde09", "name": "System.ValueTuple", "version": "4.5.0", "purl": "pkg:nuget/[email protected]", "properties": [ { "name": "aquasecurity:trivy:PkgType", "value": "nuget" } ] },I had the same issue as per other issue, that the API/UI responds with 200 okay response to uploading this SBOM.
Loading this SBOM to empty project looks fine:
Any ideas how to go over that?
Steps to Reproduce
Expected Behavior
The SBOM should be correctly replaced/updated.
Dependency-Track Version
4.11.4
Dependency-Track Distribution
Container Image
Database Server
PostgreSQL
Database Server Version
13.14
Browser
Microsoft Edge
Checklist
The text was updated successfully, but these errors were encountered: