RFC: Audit Policies and Filters

[lukekarrys]

There have been many past RFCs addressing npm audit and it's need for configurability. Some of the prior art here includes:

• Audit Assertions (RFC: audit assertions #422) which at a high level was proposed to allow module authors and users to configure which advisories npm audit surfaced applied to a particular project
• Audit Queries/Filters (RFC: Add --query to npm audit #636) proposed using --query flag to configure which packages in a tree would get audited
• Audit Policies (RFC: Add Audit Policies Support #637) proposed a configurable set of audit queries that would gate the dependency graph reification process to fail if packages prohibited by a policy were attempted to install
• Only Registry Dependencies (RFC: Only Registry Dependencies #593): proposed a flag that would cause npm install (and similar commands) to fail when attempting to install packages not from a specific registry
• Recommend Unique for packages ([RRFC] Add a field to package.json recommending that the package be unique. #649) proposed a package.json flag to warn a user that a package should be unique within a dependency graph
• Audit Licenses (RFC: npm audit licenses #182) proposed a way to audit the license of installed packages

A big thanks to the authors of those RFCs and everyone who participated in their discussion, including @bnb, @ljharb, @thescientist13, @wesleytodd, @naugtur!

This RFC does not specifically supersede any of the above RFCs, but instead is being used as a high level outline of the future plans of npm audit and net new behavior of blocking installs based on auditable packages. If an existing can be implemented fully after the ratified version of this RFC, then it will be closed with a link to this RFC and a comment showing how it could be implemented.

The purpose of this RFC is to create a configurable and sustainable base that will give the ecosystem the building blocks to implement features like "Only Registry Dependencies", "Recommend Unique" and a subset of "Audit Assertions".

Exit Criteria

• An audit configuration config that can live in a file at the root of a project or within its package.json
• An audit configuration spec that can be parsed or written to by humans, npm, or other tools
• A CLI flag for audit that will act as a single audit filter
• A way to gate installs/updates to fail or warn if audit policies are not met
• A :vulnerable query that can match the current behavior of a plain npm audit call

Implementation

• The audit configuration will live in a top-level key audit within package.json or .audit-config.json file

The existence of a valid non-empty policy config will cause npm audit to only run the specified policies. In order to easily match the current behavior of npm audit, a :vulnerable query selector will be available

The audit policy spec is mostly unchanged from it's proposal in #637. See that RFC for more examples of possible queries that can be used as policies.

type AuditConfig = {
    policies: {
        name: String
	query: String
        type?: 'error' | 'warn' | 'info' // defaults to error
	blockInstall?: Boolean // defaults to true if type is error, otherwise false
	expectEntries?: Boolean | Number | String // defaults to true
    }[]
}

Each type maps to a loglevel that will be shown when a package matching the policy is found, either during reification (install, update, etc) or audit. A type of error will also set the exitCode to 1 during audit.

If the blockInstall property is true the reification process will be halted during any command that could update the tree without making any changes to the tree on disk. This is a separate property so policies with type: 'error' can be opted out from halting the install process.

expectEntries can be a boolean, number, or string (such as >=5) that further filter whether the policy is matched. The default is true meaning any result from the query will match. For example, if expectEntries: >=2 as specified in the config, the policy would only be matched 2 or more matching packages were found.

Each policy config item maps to an individual npm audit command that can be run. For example, npm audit --query=".peer:not(:deduped)" --expect-entries=2 --audit-level=error would be equivalent to the policy config: { query: '.peer:not(:deduped)', expectEntries: 2, type: 'error' }.

Non-Goals

While this RFC pulls from many many existing RFCs, it doesn't aim to implement all the proposals in each one. Specific non-goals from the above RFCs are called out below. Note that none of the above RFCs are not being closed by the creation of this RFC, so discussion can continue in those or new RFCs for areas outlined below.

The theme of these non-goals is "can they be solved in the future on top of this RFC?" If the answer is yes, they should stay as non-goals. If the answer is no, then they should be discussed in the context of "what needs to be added to this RFC?" so they answer can be yes.

• Audit Assertions (RFC: audit assertions #422)
  • The concept of trusted entities providing assertions is outside the scope of this RFC. The goal of this RFC is to provide configurable building blocks that can be updated/shared in an initially ad-hoc way. The existing RFC is the best place to continue discussion of specifics around around sharing and trusting policies and filters.
• Audit Policies RFC: Add Audit Policies Support #637
  • Running or sharing custom policies from installed packages RFC: Add Audit Policies Support #637 (comment). This RFC is prioritizing creating an initial policy spec using only query with the idea the future additions to the spec can be done in an additive and backwards compatible manner vis future RFCs.
  • Since the policies aim to be configured as human readable JSON in well-known locations, processes can be setup to write to those files. We would like to see use cases around sharing these configs, with a plan to pave those paths as they come up.

[lukekarrys]

This will be discussed in todays weekly office hours if anyone would like to join the discussion: https://github.com/npm/rfcs#open-office-hours

[bnb]

We would like to see use cases around sharing these configs, with a plan to pave those paths as they come up.

I have hundreds of internal JavaScript repos. Being able to have one source that can be our internal source of truth for all those repos is wildly preferable than having to have each team maintaining a subset of those repos have to know enough about policies to write them themselves either puts additional burden on them or puts additional burden on those who are responsible for health of the JavaScript packages across the company to educate multiple teams.

Being able to have a single source of truth that I can point to and say "you are now required to add this" vs. having to do education will dramatically speed up adoption - weeks vs. years. I can also very easily imagine this being basically 1:1 mirrored in open source GitHub orgs as well.