CVE feed ingestion

[RaitoBezarius]

I should be able to seed the app with a tarball of CVE feeds via a management command and then the webapp should be able to download the rest of the history by itself and keep itself synchronized.

As an administrator, I would like to know since when the CVE feed has been fetched for diagnostics purpose and this information should be publicly shown in the web application to security team.

[raboof]

Focusing on CVEs seems like a good first step. You probably want to get them from the NVD feed, since NVD associates them with CPEs for matching against software packages.

Further down the line, we might also want to ingest OSV and GHSA feeds: instead of using CPEs, those associate ecosystem names (like this advisory applies to the package called 'pyarrow' in PyPI, etc).

A useful tool to fetch those might be vulnz (https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz): that project has incremental updates in scope (though it might not be implemented yet - if not we might want to contribute it).

[RaitoBezarius]

Is there any reason to not pursue the official dump of CVE from the automation group as shown here: https://github.com/CVEProject/cvelistV5/blob/main/cves/2023/34xxx/CVE-2023-34000.json which has daily releases + deltas releases every hour?

[RaitoBezarius]

(Of course, when we will be able to, we can enrich the metadata with OSV/GHSA/etc.)

[raboof]

Is there any reason to not pursue the official dump of CVE from the automation group as shown here: https://github.com/CVEProject/cvelistV5/blob/main/cves/2023/34xxx/CVE-2023-34000.json which has daily releases + deltas releases every hour?

Let's test the theory, but I suspect that the 'raw' CVE feed will have relatively little machine-readable package identification metadata. While optimizing for minimizing the time between a CVE being published and it showing up in our trackers is useful, I think the NVD feed should not be that far behind and provide CPEs which are useful for matching..

[raboof]

ah I see there's already https://github.com/Tom-Hubrecht/nix-security-tracker/blob/main/src/website/shared/fetchers.py

[ApolloUnicorn]

working branch: https://github.com/Nix-Security-WG/nix-security-tracker/tree/dg/cvenix

[raboof]

While optimizing for minimizing the time between a CVE being published and it showing up in our trackers is useful, I think the NVD feed should not be that far behind

Anecdotally: a CVE I pushed to Mitre this morning is already in the NVD feed, so the lag was at least less than 4 hours in this case.

[Tom-Hubrecht]

So, on https://github.com/Nix-Security-WG/nix-security-tracker/tree/tracker-dev we now have bulk cve ingestion for starting the tracker database as well as ingestion of delta diffs published by mitre ont the cvelistV5 repo.
Those delta are taken everyday, I don't know if we need more precision than this

[fricklerhandwerk]

Closed by #68