Use vulnerabilities from Composer package repositories

[valentijnscholten]

Current Behavior

As mentioned in #2337 (comment), Compose package repositories also provide vulnerability information for packages.

For some vulnerabilities this is the only way to get those vulnerabilities, for example SA-CONTRIB-2023-052. Neither OSS, NVD, GHSA, Snyk, OSV report this vulnerability. But for companies running lots of Drupal projects, it's important to have visibility of these vulnerabilities.

The composer package repository provides vulnerability information in two places:

• Package meta data response, i.e. https://packages.drupal.org/files/packages/8/p2/drupal/mollie.json
• API https://packagist.org/api/security-advisories/?packages%5B%5D=drupal/mollie

The API part is unclear what the URL for the Drupal 8 composer repository is. The provided example url is from the main packagist repository. This doesn not return the vulnerability.

More info about the API (and other repo functions): https://packagist.org/apidoc#list-security-advisories

Proposed Behavior

There are multiple options Depenendcy Track can do:

• During metadata processing in ComposerMetaAnalyzer the vulnerability information is already returned by the package repository. These vulnerabilities could be stored in Dependency Track and used by the Internal Analyzer. Currently this is not straightforward as meta data analysis runs in parallel with or possibly after vulnerability analysis. Downside is also that it could confuse users that the repository is also a vulnerability source (could be optional flag).

• A new analyzer could be created i.e. ComposerVulnerabilityAnalyzer that uses the API to retrieve vulnerabilities. At the time of writing it's unclear if the Drupal specific package repository supports this API.

I am curious about other peoples thoughts.

Please note that the Composer Package Repository doesn't support PURL or CPE. But since Dependency Track already retrieves meta data by just group and name, it should be OK to do the same for vulnerabilities?

I think somewhere here on Slack or Github somebody pointed us towards a library that maps PURL/CPEs to namespace/group/name parts.

I also raised composer/packagist#1497

Checklist

• I have read and understand the contributing guidelines
• I have checked the existing issues for whether this enhancement was already requested

[nscuro]

During metadata processing in ComposerMetaAnalyzer the vulnerability information is already returned by the package repository. [...] Currently this is not straightforward as meta data analysis runs in parallel with or possibly after vulnerability analysis.

Agree with this assessment. Also in general the repository meta analysis is a lot slower than vulnerability analysis, since no repository supports batch request (does Composer?). Trying to somehow couple repo meta and vuln analysis is impractical.

I'd say we make this another vulnerability scanner that can be enabled, or is even enabled by default. It is a bit unfortunate that we'd end up querying the API endpoints more than once for the same component, but I don't see a better option.

Is there any rate limiting imposed on this API?

[valentijnscholten]

From their docs

Best practices when using the Packagist.org API

• If you do scheduled jobs, avoid running things at midnight or once an hour at XX:00. Most people do so and we do see traffic peaks every hour. Pick a "random" time by hand for your cron jobs, or even better (if you can) is to make it run on a really randomized schedule.
• Send a User-Agent header with all your requests including an email or twitter or some sort of contact information so we can reach out to you if we have an issue with the way you use the API. If not you can leave us with no choice but to block IPs which we'd rather not do.
• If you send requests in parallel, be a good citizen and do a maximum of 10 concurrent requests, or up to 20 if you only fetch static files. Huge bursts in requests can cause issues for other regular users so try to spread out the load.
• Use an HTTP/2-capable client and make sure you enable that, it is much more efficient than reconnecting for every request.

In another spot they mention:

You can also send If-Modified-Since headers to limit your bandwidth usage and cache the files on your end with the proper filemtime set according to our Last-Modified header.

Something even more incremental is also possible:

https://packagist.org/metadata/changes.json?since=17344729990000

I haven't looked at the DT data model for this yet, but maybe we can store (or deduce) a value for If-Modified-Since to help reduce the load on their side. We could also look at a shared cache between the Meta analyzer and the vulnerability analyzer. A cache would be good anyway because the meta data response contains the metadata for all package versions at once.

However, the .../p2/xxx/yyy.json files are static files on their end, so it might just be good enough to request those in a new analyzer without any optimizations.

[nscuro]

be a good citizen and do a maximum of 10 concurrent requests, or up to 20 if you only fetch static files.

If concurrent requests become a requirement, we could look at https://resilience4j.readme.io/docs/bulkhead to rate limit on our end. We already use resilience4j elsewhere for retries etc.

[valentijnscholten]

I have looked into this some more and it's not straightforward. The packagist repo has an api to retrieve vulnerabilities: api/security-advisories. But this URL doesn't seem to be available on the Drupal package repository above. Neither does the Drupal package repository have the list.json that contains the names of all the packages in the repository. Or maybe it does have it, but on a different URL. We could fall back to the vulnerabities that are returned in the single package metadata response, but I'm not sure if that is reliable/supported by all repositories.

A colleague of mine was trying to implement the same functionality in a custom built SCA tool, but ran into the same problems.
Ultimately he resorted to just running composer audit and use the output from there.

composer audit has all the logic implemented to find all applicable vulnerabilities in the composer ecosystem. It uses the FriendsOfPhp advisories and vendor specific advisories such as from the Drupal repo (and probably Magento, Shopware, etc).
It also use GitHub Security Advisories to match with packages.

The Auditor and ComposerRepository classes are non-trivial: https://github.com/composer/composer/blob/main/src/Composer/Advisory/Auditor.php

Do we really want to mimic and maintain all this in Dependency Track?

Wouldn't it be a lot easier/more reliable if we could just ingest the output from composer audit?

+-------------------+----------------------------------------------------------------------------------+
| Package           | drupal/mollie                                                                    |
| Severity          |                                                                                  |
| CVE               | NO CVE                                                                           |
| Title             | Mollie for Drupal - Moderately critical - Faulty payment confirmation logic -    |
|                   | SA-CONTRIB-2023-052                                                              |
| URL               | https://www.drupal.org/sa-contrib-2023-052                                       |
| Affected versions | <2.2.1                                                                           |
| Reported at       | 2023-11-15T14:24:12+00:00                                                        |
| Advisory ID       | SA-CONTRIB-2023-052                                                              |
+-------------------+----------------------------------------------------------------------------------+

When looking at the VEX functionality in Dependency Track I noticed that it cannot be used to add new vulnerabilities to DependencyTrack. It can only apply analysises to vulnerabilities already present in Dependency Track and attributed to the Project where the VEX is being applied.

I understand importing/trusting vulnerabilities from outside Dependency Track doesn't really fit in the model of using the SBOM as source of truth and do all scanning inside DependencyTrack. But this way it might impossible to even get the same results as these scanners/audit tools native to the ecosystems.

EDIT: I am not aware of any Composer Repository Proxy / Aggregator that could be used to achieve the same. It looks like Nexus has some support for Composer repositories. I doubt it supports all the vulnerability related stuff. Even if it would, it would be quite an extra requirement for users to have a Nexus install available.

What possibly could work is to make a little wrapper around the composer audit command / code to expose the audit functionality as a web service to be queried by Dependency Track. Similar to running a local Trivy server. The web service would need to be aware of the repositories used/needed by a project. I don't think these are always in the SBOM. Maybe it could work if it just has "all known" composer repositories configured.

[nscuro]

Usually vulnerableCode is quite extensive with this sort of thing, but it doesn't look like they support PHP / Drupal either: https://github.com/aboutcode-org/vulnerablecode/tree/main/vulnerabilities/importers

Do we really want to mimic and maintain all this in Dependency Track?

Is it possible to download all data, or does the whole thing depend on ad-hoc queries for each package? If the former, you could just write a mirror task similar to the OSV one. Even if it's non-trivial, I feel like it'd still be better than writing a bespoke external service that users would then need to deploy and operate.

It would also fit the agenda of #4122.

Wouldn't it be a lot easier/more reliable if we could just ingest the output from composer audit?

We won't be supporting ingest of tool output like Defect Dojo does, but if you were to transform this into a CycloneDX VDR file we could import that, see below.

I understand importing/trusting vulnerabilities from outside Dependency Track doesn't really fit in the model of using the SBOM as source of truth and do all scanning inside DependencyTrack.

It doesn't currently work because it simply was not implemented yet, not because we don't want to support it. There's even a placeholder / TODO for it here:

dependency-track/src/main/java/org/dependencytrack/tasks/BomUploadProcessingTask.java

Lines 272 to 276 in 45982a2

	// Note: If we need to synchronize vulnerability details (i.e. import of embedded VEX or VDR),
	// that should happen OUTSIDE the transaction below, likely before.
	// Synchronizing all components and services in a single TRX is viable because they are "sharded"
	// by project; This is not the case for vulnerabilities. We don't want the entire TRX to fail,
	// just because another TRX created or modified the same vulnerability record.

That being said, until #4122 lands, we need to be a bit careful when it comes to vulnerability sources. We wouldn't want VDR imports to overwrite CVE data if it has already been mirrored from the NVD. And Vulnerability#source is currently enum-based, so we can't allow any source value either.

[valentijnscholten]

In theory it should be able to mirror all the vulnerabilities. At least the main packagist.com has APIs to do that incrementally, even providing a feed with new/changed advisories since a certain timestamp that can be provided as a parameter.
But it's unclear if all the third party repositories support this. Some repositories maybe just static files generated by Satis.
I guess we'll have to interpret the composer audit code to understand what it's doing before we know.

In theory we could also mirror all the metadata for all packages. There is a changes.json feed where we can do timestamp based mirroring of package metadata. It has to run at least once per 24h.

I am currently looking at which vulnerability identifiers we can expect. There will be vulnerabilities that have an ID generated by the composer auditor (example PKSA-w57n-mhp6-c9sd). An advisory can have multiple "remote" IDs as well, such as CVE, GHSA. But sometimes there are no other identifiers. We could introduce a new source COMPOSER of COMPOSER_AUDIT for these PKSA vulnerabilities. But there maybe new remote IDs in these vulnerabilities from third parties, for example Drupal advisotries have an ID like SA-CONTRIB-2023-052 (And no PKSA ID). I guess this needs to be another new source?

There maybe more sources as Magento, Shopware may have their own IDs as well. Every new sources means a new column in the Aliases table unfortunately. As we don't know all possible sources, some of them will have to be stored under UNKNOWN.

[valentijnscholten]

Turns out there's a lot of stuff happening in composer or composer audit we don't really need. Every repository has a metadata file in their root:

https://packages.drupal.org/8/packages.json

{
  "notify-batch": "/8/downloads",
  "metadata-url": "/files/packages/8/p2/%package%.json",
  "available-package-patterns": [
    "drupal/*"
  ],
  "search": "/8/search.json?s=%query%",
  "warning": "Drupal.org packages ARE NO LONGER AVAILABLE FOR COMPOSER 1!!! Composer 1 is NOT supported. You should upgrade to Composer 2 now! See https://www.drupal.org/drupalorg/blog/ending-packagesdrupalorg-support-for-composer-1",
  "warning-versions": "\u003C1.99",
  "security-advisories": {
    "metadata": true,
    "api-url": "https://packages.drupal.org/8/security-advisories"
  }
}

This specifies where security advisories can be obtained (or not).

https://wpackagist.org/packages.json doesn't provide a security advisory endpoint:

{
  "packages": [],
  "providers-url": "/p/%package%$%hash%.json",
  "provider-includes": {
    "p/providers-2011$%hash%.json": {
      "sha256": "a0951d6abac7bb6e8cbf94302c37bfe8dc3e21a7becce0404c27de9e36775658"
    },
    "p/providers-2012$%hash%.json": {
      "sha256": "a6600709e8f655af4d9f5db1e504f4338a72868642c5c43903d9cca3eed61176"
    },
    "p/providers-2013$%hash%.json": {
      "sha256": "cf116339d3ac5212148341941576402b5cda7acd959feb6f94457ff8ae96538f"
    },
    "p/providers-2014$%hash%.json": {
      "sha256": "53e5915305da57cc9e4ba57073f56ad9f95e89c643d4a11a63840d5f5eeb5e30"
    },
    "p/providers-2015$%hash%.json": {
      "sha256": "370bf8d70bae3cd768a4c9e85c0d42d57621d5fd1f07f793cdc66cee09eeef0f"
    },
    "p/providers-2016$%hash%.json": {
      "sha256": "c9ff23218b5b50386710e20ea8cf883a0413b45fa5ba22784f32a0ecb9ffe414"
    },
    "p/providers-2017$%hash%.json": {
      "sha256": "3db004fdb6634d8a7217c06ab79341dc4427e39f4b6a2f8596cb5c75facb4ead"
    },
    "p/providers-2018$%hash%.json": {
      "sha256": "2d480f43c05e76e4706663bc03663d74a7846f6ddce9f08efaa485663a2aee07"
    },
    "p/providers-2019$%hash%.json": {
      "sha256": "1ec75b6e9be93755d0f41a7ded4b782b03ffa1b1b28fb3b889c85ee79aff6e2a"
    },
    "p/providers-2020$%hash%.json": {
      "sha256": "a77dd757ba3da195e6626536782dd6843cff2f1452b8b50bf1c115173fe69996"
    },
    "p/providers-2021$%hash%.json": {
      "sha256": "8464710bea5a7a51339178215c93ab0a102c3390c39d017a507464466ce7161c"
    },
    "p/providers-2022$%hash%.json": {
      "sha256": "6833d08e62748c90fc33869af9e997b8f7e0940909ec507ad5c89cba635a2788"
    },
    "p/providers-2023$%hash%.json": {
      "sha256": "574cf9aaff94e51a3b114f6c5ce7baa33f713a102d7e69c346d1339dde5e3534"
    },
    "p/providers-2024-03$%hash%.json": {
      "sha256": "bb0bda2b8a547e77254724bb025a84fc4b907b5c1bc1c340e0d501bbff76e90e"
    },
    "p/providers-2024-06$%hash%.json": {
      "sha256": "a861c3aadcbebbdf8a2db9d0447864cdd0573e758686f2b60bc5f1e40c6ae062"
    },
    "p/providers-2024-09$%hash%.json": {
      "sha256": "93228aa5b730ef351c05c117c015b7c9b0dd76fa05fc30cb9f287ea764022d97"
    },
    "p/providers-2024-12$%hash%.json": {
      "sha256": "453d810a14e2aa0ed84a5b9c9657105e3ac40069361801ec1b0f60cab37d68ce"
    },
    "p/providers-old$%hash%.json": {
      "sha256": "e730658f3f4054559aa116639c7b13d893093f93cf871548d9dbb8f5cb6a50a1"
    },
    "p/providers-this-week$%hash%.json": {
      "sha256": "d1ec9fbac2aebd38c6901a18e65d67255545e3f3a62c0a5a4b663af85f2cd91b"
    }
  },
  "available-package-patterns": [
    "wpackagist-plugin/*",
    "wpackagist-theme/*"
  ]
}

The available-package-patterns is interesting to us as it can be used to not spam repositories with unrelated queries.

Some repositories such as wpackagits also do not provide a metadata endpoint. I haven't looked into that, just focussing on the vulnerability aspects here.

In theory it's possible to use the security advisory API to build a complete mirror. I'm unsure if we can run into problems if we combine the vulnerabilitiy data from multiple repositories into one SOURCE / mirror on the DT side.

[valentijnscholten]

The vulnerabilities coming in from the composer repostiories do not have a PURL or a CPE. Currently the Internal Analyzer in DT only detects VulnerableSoftware that has at least one of those.

I also see code in OsvDownloadTask and GitHubAdvisoryMirrorTask preventing the creation of VulnerableSoftware entities without a PURL. Should we try to recreate the PURL as is being done in GitHubAdvisoryMirrorTask?

Strange thing is that in my deptrack databases I can see lots of VulnerableSoftware without a PURL and without a CPE:

deptrackuat=> SELECT COUNT(*) FROM "AFFECTEDVERSIONATTRIBUTION" JOIN "VULNERABLESOFTWARE" VS ON VS."ID" = "VULNERABLE_SOFTWARE" WHERE "PURL" IS NULL AND "CPE23" IS NULL LIMIT 15;
 count
-------
 20505

   ID    |         FIRST_SEEN         |         LAST_SEEN          | SOURCE |                 UUID                 | VULNERABILITY | VULNERABLE_SOFTWARE |   ID   | CPE22 | CPE23 | EDITION | LANGUAGE | OTHER | PART | PRODUCT | PURL |   PURL_NAME   | PURL_NAMESPACE | PURL_QUALIFIERS | PURL_SUBPATH | PURL_TYPE | PURL_VERSION | SWEDITION | TARGETHW | TARGETSW | UPDATE |                 UUID                 | VENDOR | VERSION | VERSIONENDEXCLUDING | VERSIONENDINCLUDING | VERSIONSTARTEXCLUDING | VERSIONSTARTINCLUDING | VULNERABLE
---------+----------------------------+----------------------------+--------+--------------------------------------+---------------+---------------------+--------+-------+-------+---------+----------+-------+------+---------+------+---------------+----------------+-----------------+--------------+-----------+--------------+-----------+----------+----------+--------+--------------------------------------+--------+---------+---------------------+---------------------+-----------------------+-----------------------+------------
    5957 | 2022-12-19 15:46:26.599+00 | 2023-12-12 16:30:08.16+00  | OSV    | a079bcf7-9237-4a8c-8969-62d81b897263 |        187343 |              338892 | 338892 |       |       |         |          |       |      |         |      | path          | @irrelon       |                 |              | npm       |              |           |          |          |        | af1eeff4-5b0e-4985-8188-007ec20eaa00 |        |         | 4.7.0               |                     |                       |                       | t
   71919 | 2022-12-19 16:02:48.36+00  | 2024-12-19 14:37:51.236+00 | GITHUB | fe5c1791-c166-4dff-9161-ce965b70a572 |        187343 |              338892 | 338892 |       |       |         |          |       |      |         |      | path          | @irrelon       |                 |              | npm       |              |           |          |          |        | af1eeff4-5b0e-4985-8188-007ec20eaa00 |        |         | 4.7.0               |                     |                       |                       | t
    3320 | 2022-12-19 15:43:23.665+00 | 2023-12-12 16:29:33.765+00 | OSV    | 296f08db-71fd-4746-a64f-1ad68a6de4e3 |        187344 |              338893 | 338893 |       |       |         |          |       |      |         |      | connie-lang   |                |                 |              | npm       |              |           |          |          |        | 5ac3b1d8-37b3-4d55-977f-2c8b0694cc27 |        |         | 0.1.1               |                     |                       |                       | t
   71774 | 2022-12-19 16:02:46.941+00 | 2024-12-19 14:31:49.427+00 | GITHUB | f4fc631e-c81f-4046-ac96-8aeb136c2bfa |        187344 |              338893 | 338893 |       |       |         |          |       |      |         |      | connie-lang   |                |                 |              | npm       |              |           |          |          |        | 5ac3b1d8-37b3-4d55-977f-2c8b0694cc27 |        |         | 0.1.1               |                     |                       |                       | t
   71833 | 2022-12-19 16:02:47.144+00 | 2024-12-19 14:31:52.402+00 | GITHUB | 26c000cf-3a19-4060-b26b-6ff73bd00e66 |        187345 |              338894 | 338894 |       |       |         |          |       |      |         |      | property-expr |                |                 |              | npm       |              |           |          |          |        | 4fed263d-57a2-448f-88b5-f3df27e6ed9d |        |         | 2.0.3               |                     |                       |                       | t
    2365 | 2022-12-19 15:41:37.682+00 | 2023-12-12 16:29:18.491+00 | OSV    | d7e75254-2638-4437-8a76-4785b923ac58 |        187345 |              338894 | 338894 |       |       |         |          |       |      |         |      | property-expr |                |                 |              | npm       |              |           |          |          |        | 4fed263d-57a2-448f-88b5-f3df27e6ed9d |        |         | 2.0.3               |                     |                       |                       | t
 1842552 | 2022-12-21 13:45:45.944+00 | 2023-12-12 16:28:53.48+00  | OSV    | 40f3dd5a-d3aa-40eb-bbae-f0aa407447d8 |        213017 |              338895 | 338895 |       |       |         |          |       |      |         |      | safe-eval     |                |                 |              | npm       |              |           |          |          |        | 8c9e2074-8d30-43ba-8981-ad2e9f7f93ac |        |         |                     | 0.4.1               |                       |                       | t
   33037 | 2022-12-19 15:56:17.944+00 | 2024-12-19 14:38:02.359+00 | GITHUB | a6a754fa-d032-40c9-9c89-2ac6478aa247 |        187346 |              338895 | 338895 |       |       |         |          |       |      |         |      | safe-eval     |                |                 |              | npm       |              |           |          |          |        | 8c9e2074-8d30-43ba-8981-ad2e9f7f93ac |        |         |                     | 0.4.1               |                       |                       | t
    5254 | 2022-12-19 15:46:14.461+00 | 2023-12-12 16:30:04.865+00 | OSV    | 863089c7-04bc-4dc0-8d33-454b75abf196 |        187346 |              338895 | 338895 |       |       |         |          |       |      |         |      | safe-eval     |                |                 |              | npm       |              |           |          |          |        | 8c9e2074-8d30-43ba-8981-ad2e9f7f93ac |        |         |                     | 0.4.1               |                       |                       | t
 1842935 | 2022-12-21 13:56:41.551+00 | 2024-12-19 14:34:33.542+00 | GITHUB | ee73d49f-736e-418d-ad01-74976ab41add |        213017 |              338895 | 338895 |       |       |         |          |       |      |         |      | safe-eval     |                |                 |              | npm       |              |           |          |          |        | 8c9e2074-8d30-43ba-8981-ad2e9f7f93ac |        |         |                     | 0.4.1               |                       |                       | t
   44228 | 2022-12-19 15:58:52.942+00 | 2024-12-19 14:33:42.661+00 | GITHUB | 3e2973ad-8f31-44a2-91a1-0cc2806f999b |        187347 |              338896 | 338896 |       |       |         |          |       |      |         |      | dot-notes     |                |                 |              | npm       |              |           |          |          |        | 2060f816-c25e-4770-893c-9a2278d96880 |        |         |                     | 3.2.0               |                       |                       | t
   11535 | 2022-12-19 15:47:57.108+00 | 2023-12-12 16:30:26.367+00 | OSV    | 916a63ed-63c3-4893-9730-bd2ef7a07857 |        187347 |              338896 | 338896 |       |       |         |          |       |      |         |      | dot-notes     |                |                 |              | npm       |              |           |          |          |        | 2060f816-c25e-4770-893c-9a2278d96880 |        |         |                     | 3.2.0               |                       |                       | t
    3041 | 2022-12-19 15:42:50.732+00 | 2023-12-12 16:29:28.864+00 | OSV    | 86cf8417-c1db-4a0c-9a55-04c20c5102dd |        187348 |              338897 | 338897 |       |       |         |          |       |      |         |      | deep-get-set  |                |                 |              | npm       |              |           |          |          |        | 17ca69dc-e93d-4034-b7e4-9affff9ecfc9 |        |         | 1.1.1               |                     |                       |                       | t
   39954 | 2022-12-19 15:58:04.259+00 | 2024-12-19 14:33:50.134+00 | GITHUB | c210b841-fca8-4661-9cee-6fd640f7f1c5 |        187348 |              338897 | 338897 |       |       |         |          |       |      |         |      | deep-get-set  |                |                 |              | npm       |              |           |          |          |        | 17ca69dc-e93d-4034-b7e4-9affff9ecfc9 |        |         | 1.1.1               |                     |                       |                       | t
    9433 | 2022-12-19 15:47:12.72+00  | 2023-12-12 16:30:18.435+00 | OSV    | 9de7b4c1-3dc6-49f2-a89a-5bc4d96d0504 |        187349 |              338898 | 338898 |       |       |         |          |       |      |         |      | nodee-utils   |                |                 |              | npm       |              |           |          |          |        | b2516448-3a54-4be7-8fdb-464ce331c16c |        |         | 1.2.3               |                     |                       |                       | t

[nscuro]

Strange thing is that in my deptrack databases I can see lots of VulnerableSoftware without a PURL and without a CPE

PURL_TYPE, PURL_NAMESPACE, and PURL_NAME is what we query on during analysis. Storing the entire PURL mostly just bloats the table, I'm not sure why that column even exists given we also have dedicated columns for qualifiers and subpaths (which are not relevant to any vulnerability source I know of anyway).

[valentijnscholten]

Ah, I there are rows in my databases that do have the PURL column populated, guess that put me on the wrong foot. It's only 96k rows I checked just now :-)

So what do you think would be the best option @nscuro. Going for a new external analyzer which is easier to implement and can be enabled/disabled or going for a full mirror that is more work and needs more safeguarding to make sure it doesn't fail too often and doesn't miss any vulnerabilities. Currently there's also the downside that once mirrored, the Internal Analyzer will keep detecting/reporting the vulnerabilities, even after the mirroring is disabled.