fix: retrieve registry keys via TUF

[bdehamer]

Updates the audit signatures command to retrieve the registry keys from the Sigstore TUF repository. The keys are published as a delegated target under the registry.npmjs.org namespace.

The published keys.json uses a slightly different format than the /-/npm/v1/keys endpoint:

{
  "keys": [
    {
      "keyId": "SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA",
      "keyUsage": "npm:signatures",
      "publicKey": {
        "rawBytes": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1Olb3zMAFFxXKHiIkQO5cJ3Yhl5i6UPp+IhuteBJbuHcA5UogKo0EWtlWwW6KSaKoTNEYL7JlCQiVnkhBktUgg==",
        "keyDetails": "PKIX_ECDSA_P256_SHA_256",
        "validFor": {
          "start": "1999-01-01T00:00:00.000Z"
        }
      }
    },
    {
      "keyId": "SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA",
      "keyUsage": "npm:attestations",
      "publicKey": {
        "rawBytes": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE1Olb3zMAFFxXKHiIkQO5cJ3Yhl5i6UPp+IhuteBJbuHcA5UogKo0EWtlWwW6KSaKoTNEYL7JlCQiVnkhBktUgg==",
        "keyDetails": "PKIX_ECDSA_P256_SHA_256",
        "validFor": {
          "start": "2022-12-01T00:00:00.000Z"
        }
      }
    }
  ]
}

For now, we're transforming this to match the existing key file format used on the registry so that we don't have to make any changes in pacote. In the future we'll update pacote to take advantage of some of the additional metadata available in the newer format.

This new scheme will support third-party registries which may also want to publish their keys to the Sigstore TUF repository, but will fallback to the /-/npm/v1/keys endpoint for backward compatibility.

[ljharb]

Does this mean that npm audit will now have a runtime dependency on two servers instead of one??

(to be clear, adding an additional network request location to npm feels like a breaking change to me; lots of places that run npm will lock down network traffic to known servers)

[npm-cli-bot]

no statistically significant performance changes detected

timing results

app-large	clean	lock-only	cache-only	cache-only peer-deps	modules-only	no-lock	no-cache	no-modules	no-clean	no-clean audit	show-version	run-script
npm@9	43.315 ±1.43	24.451 ±0.18	23.528 ±0.09	26.709 ±0.96	3.468 ±0.03	3.733 ±0.15	3.057 ±0.15	16.761 ±0.46	2.991 ±0.04	4.279 ±0.08	0.448 ±0.00	0.470 ±0.01
#6418	45.772 ±0.36	24.787 ±0.05	24.530 ±0.25	28.190 ±0.75	3.592 ±0.06	3.653 ±0.08	3.046 ±0.05	17.150 ±0.19	3.150 ±0.12	4.489 ±0.32	0.460 ±0.01	0.491 ±0.02

app-medium	clean	lock-only	cache-only	cache-only peer-deps	modules-only	no-lock	no-cache	no-modules	no-clean	no-clean audit	show-version	run-script
npm@9	30.496 ±0.39	18.543 ±0.22	17.667 ±0.36	18.779 ±0.11	3.246 ±0.03	3.252 ±0.01	3.117 ±0.01	11.924 ±0.12	2.822 ±0.01	3.909 ±0.03	0.450 ±0.02	0.490 ±0.02
#6418	30.496 ±0.93	18.657 ±0.32	17.854 ±0.30	19.178 ±0.21	3.286 ±0.04	3.235 ±0.02	3.190 ±0.06	12.265 ±0.19	2.827 ±0.00	4.189 ±0.42	0.451 ±0.00	0.489 ±0.01

[feelepxyz]

Very nice 👍 would we need to handle a trailing slash on the path? e.g. https://verdaccio-clone.org/npm/

[bdehamer]

Good call. Will add a test case to make sure that's handled properly.

[feelepxyz]

adding an additional network request location to npm feels like a breaking change to me; lots of places that run npm will lock down network traffic to known servers)

@ljharb this is good context, makes sense that you might lock down network traffic around npm.

This change should not add any more network requests to audit signatures as we've already implemented attestation verification support in pacote that perform the same network requests there to update the Sigstore trust root (this now includes npm's public keys) 😬

Would have been good to surface this at the time, but I'm also not aware of any complaints around the pacote change so this might mean the audit signatures command still hasn't gotten a lot of usage.

Good to surface all these constraints before we consider turning some of these checks on by default when running npm audit.

[ljharb]

People largely don’t run anything that’s not run by default; I’d urge you to consider setting up an npm registry endpoint for this rather than forcing clients to connect independently to sigstore.

[bdehamer]

I wonder, would we want to make the same change to pacote so it re-uses the same cache?

@feelepxyz I've got a pacote PR open with the changes necessary to use a consistent TUF cache between the two uses cases.

[feelepxyz]

Awesome work on this!

[ljharb]

To reiterate, any CI that’s using something like https://github.com/step-security/harden-runner will immediately fail when npm audit runs (including 400+ of my packages), so this will be a very disruptive breaking change if it’s by default.

[wraithgar]

Again, this doesn't run on audit. This is part of npm audit signatures.

[ljharb]

ahh, thanks, i misunderstood that - I thought this PR was affecting npm audit itself. Appreciate the clarification.

[wraithgar]

The pacote PR that added the tuf cache param parsing landed and is published.

[bdehamer]

@wraithgar this is ready to go for the next release