Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unique constraint violation while mirroring NVD via feed files #3663

Closed
2 tasks done
nscuro opened this issue May 1, 2024 · 1 comment · Fixed by #3664
Closed
2 tasks done

Unique constraint violation while mirroring NVD via feed files #3663

nscuro opened this issue May 1, 2024 · 1 comment · Fixed by #3664
Assignees
@nscuro
Labels
defect Something isn't working size/S Small effort
Milestone
4.11

Comments

@nscuro
Copy link
Member

nscuro commented May 1, 2024

Current Behavior

Mirroring the NVD via feed files can fail due to unique constraint violations in the AFFECTEDVERSIONATTRIBUTION table:

 javax.jdo.JDODataStoreException: Insert of object "org.dependencytrack.model.AffectedVersionAttribution@65e2bfba" using statement "INSERT INTO "AFFECTEDVERSIONATTRIBUTION" ("FIRST_SEEN","LAST_SEEN","SOURCE","UUID","VULNERABILITY","VULNERABLE_SOFTWARE") VALUES (?,?,?,?,?,?)" failed : ERROR: duplicate key value violates unique constraint "AFFECTEDVERSIONATTRIBUTION_COMPOSITE_IDX"

Mirroring via API does not seem to be affected. Taking an educated guess here, the logic that processes the feed files can sometimes create duplicate Vulnerability <-> VulnerableSoftware relationships. I had to deal with that while implementing the API mirroring:

dependency-track/src/main/java/org/dependencytrack/tasks/NistApiMirrorTask.java

Lines 268 to 273 in fd82e58

// Get all VulnerableSoftware records that are currently associated with the vulnerability.
// Note: For SOME ODD REASON, duplicate (as in, same database ID and all) VulnerableSoftware
// records are returned, when operating on data that was originally created by the feed-based
// NistMirrorTask. We thus have to deduplicate here.
final List<VulnerableSoftware> vsOldList = persistentVuln.getVulnerableSoftware().stream().distinct().toList();
LOGGER.trace("%s: Existing VS: %d".formatted(persistentVuln.getVulnId(), vsOldList.size()));

Steps to Reproduce

  1. Mirror NVD via feed files

Expected Behavior

Mirroring should not fail. We should not create duplicate records that cause unique constraint violations.

Dependency-Track Version

4.11.0-SNAPSHOT

Dependency-Track Distribution

Container Image, Executable WAR

Database Server

N/A

Database Server Version

No response

Browser

N/A

Checklist
@nscuro nscuro added defect Something isn't working size/S Small effort labels May 1, 2024
@nscuro nscuro added this to the 4.11 milestone May 1, 2024
@nscuro nscuro self-assigned this May 1, 2024
nscuro added a commit to nscuro/dependency-track that referenced this issue May 1, 2024
@nscuro
Fix unique constraint violation during NVD mirroring
1cbef14 
The fix is achieved by using the same logic for persisting `Vulnerability` and `VulnerableSoftware` records that `NistApiMirrorTask` was already using. It handles duplicate records.

This should also yield a performance boost (did not benchmark because that wasn't the focus of this change), since the transaction commit frequency is reduced compared to the previous logic.

Fixes DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
@nscuro nscuro mentioned this issue May 1, 2024
Merged
2 tasks
nscuro added a commit to nscuro/dependency-track that referenced this issue May 1, 2024
@nscuro
Fix unique constraint violation during NVD mirroring
c747044 
The fix is achieved by using the same logic for persisting `Vulnerability` and `VulnerableSoftware` records that `NistApiMirrorTask` was already using. It handles duplicate records.

This should also yield a performance boost (did not benchmark because that wasn't the focus of this change), since the transaction commit frequency is reduced compared to the previous logic.

Fixes DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
nscuro added a commit to nscuro/dependency-track that referenced this issue May 1, 2024
@nscuro
Fix unique constraint violation during NVD mirroring
df1174d 
The fix is achieved by using the same logic for persisting `Vulnerability` and `VulnerableSoftware` records that `NistApiMirrorTask` was already using. It handles duplicate records.

This should also yield a performance boost (did not benchmark because that wasn't the focus of this change), since the transaction commit frequency is reduced compared to the previous logic.

Fixes DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
nscuro added a commit to nscuro/dependency-track that referenced this issue May 2, 2024
@nscuro
De-duplicate CPEs in NVD feed file parsing
7d7f0ed 
Some CVE records contain duplicate CPEs. The original parsing logic did not de-duplicate those, consequently causing duplicate `VulnerableSoftware` records in the database.

De-duplication was already handled in `NistApiMirrorTask`, but not `NistMirrorTask`.

Relates to DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
@nscuro nscuro mentioned this issue May 2, 2024
Merged
2 tasks
rkg-mm pushed a commit to rkg-mm/dependency-track that referenced this issue May 5, 2024
@nscuro @rkg-mm
Fix unique constraint violation during NVD mirroring
be034e1 
The fix is achieved by using the same logic for persisting `Vulnerability` and `VulnerableSoftware` records that `NistApiMirrorTask` was already using. It handles duplicate records.

This should also yield a performance boost (did not benchmark because that wasn't the focus of this change), since the transaction commit frequency is reduced compared to the previous logic.

Fixes DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
rkg-mm pushed a commit to rkg-mm/dependency-track that referenced this issue May 5, 2024
@nscuro @rkg-mm
De-duplicate CPEs in NVD feed file parsing
1eaa700 
Some CVE records contain duplicate CPEs. The original parsing logic did not de-duplicate those, consequently causing duplicate `VulnerableSoftware` records in the database.

De-duplication was already handled in `NistApiMirrorTask`, but not `NistMirrorTask`.

Relates to DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 2, 2024

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 2, 2024
MM-msr pushed a commit to MM-msr/dependency-track that referenced this issue Jun 18, 2024
@nscuro @MM-msr
Fix unique constraint violation during NVD mirroring
c13dbdd 
The fix is achieved by using the same logic for persisting `Vulnerability` and `VulnerableSoftware` records that `NistApiMirrorTask` was already using. It handles duplicate records.

This should also yield a performance boost (did not benchmark because that wasn't the focus of this change), since the transaction commit frequency is reduced compared to the previous logic.

Fixes DependencyTrack#3663

Signed-off-by: nscuro <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@nscuro nscuro

Labels
defect Something isn't working size/S Small effort
Projects
None yet
Milestone
4.11
Development

Successfully merging a pull request may close this issue.

Fix unique constraint violation during NVD mirroring via feed files nscuro/dependency-track
1 participant
@nscuro