[RRFC] Build Google Wireit's script runner into npm

[justinfagnani]

Motivation ("The Why")

There are a number of tools, and RRFCs (#190, #610 and #548, #691) that relate to running scripts in series, parallel, etc.

I opened #691 as a proposal to add pluggable script runners to npm to make using tools like Wireit easier. While I think that is a good idea, I honestly think that an even better solution is to build Wireit directly into npm, giving all npm uses the ability to define a graph of scripts and utilize caching, watch mode, and long-running services.

Wireit has a very robust feature set and implementation at this point, and has been designed to fit very seamlessly with the npm CLI's interface, configuration and workflow. It is not a tool that takes over and works over-top of npm, but one that tries to be as much of a part of npm as possible. As such, I think it should be quite easy to integrate with npm compared to tools that have their own CLI, config, and implementation (sometimes in non-JS languages).

Adding a full script graph with caching would solve all of the open issues around more powerful script running, and several issues that third-party tools cover now. For example, output caching can save significant build time for developers and CI. Services with watch mode make rebuilding and restarting servers on code changes (similar to nodemon) fast and seamless.

Example

It's probably best to view the Wireit repo for many examples of its features.

A basic Wireite configuration with dependencies between scripts looks like:

{
  "scripts": {
    "build": "wireit",
    "bundle": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc"
    },
    "bundle": {
      "command": "rollup -c",
      "dependencies": ["build"]
    }
  }
}

Integrated into npm, this could look like:

{
  "scripts": {
    "build": "tsc",
    "bundle": {
      "command": "rollup -c",
      "dependencies": ["build"]
    }
  }
}

How

Current Behaviour

Currently, scripts are just strings with the command to run.

Desired Behaviour

I propose making scripts objects that describe their command, dependencies, inputs and output files (for caching) and more.

References

• [RRFC] npm run-series && npm run-parallel #190
• [RRFC] Parallel script execution when value is set to an array of text. #610
• [RRFC] Add flag for running NPM commands in transitive dependencies #548
• [RRFC] Pluggable script runners #691

cc @aomarks, the author of Wireit

[bhouston]

I super support this! Wireit is the most complete solution, even includes caching.

But given the non-response to previous RRFCs on these topics, how can we push this one forward where the others haven't succeeded? Do we just have to implement it? Do we have to run a publicity campaign on the various social networks?

How can this RRFC succeed where the previous similar (but less complete) ones haven't?

[bivens-dev]

Seconding that I have used this in a large number of projects at varying levels of complexity and found it to be a great balance between ease of learning and productivity. It’s my default in all projects at this point

[ruyadorno]

How can this RRFC succeed where the previous similar (but less complete) ones haven't?

I believe the right place to start is getting a champion behind it to write a proper RFC and get the attention from the npm cli team to review it and get it approved.

I think this sounds like a great idea in order to unlock all these useful workflows and improve default workspace support!

[justinfagnani]

I think the real path forward here, including if Wireit were to be built in, is to really identify the use cases and requirements.

I personally believe that this would show that a couple of primitives to run scripts in parallel, series, or in topological order, would be insufficient for many use cases, and that a true dependency graph, caching, and services are needed for a large number of projects.

That in turn should help show that the level of complexity of a tool like Wireit (because although it's simple compared to other similar tools, it is certainly more complex than a npm-run-parallel command) is justified, and that it has a good, as-big-as-necessary / as-minimal-as-possible feature set and design.

The thing is that this is the kind of PM / eng work that community commentators (such as myself!) are unlikely to have the time to do throughly, so I think we have to hope an npm employee gets inspired enough to do it.

[justinfagnani]

I want to note some potential use cases to consider, for either being in or out of scope for these related feature requests.

Shared script dependency of multiple scripts in a single-package repo

In this case scripts A an B both depend on script C running first. There is no monorepo, so all scripts are in the same package.json.

I think this is the simplest case that's not covered by any of the run-parallel, run-series, and topological-sort features, and points to needing a true script dependency graph.

Diamond script dependency in a monorepo

Scripts A an B both depend on script C running first, all are in separate packages. The top level package has a script T that runs A and B. C should run only once.

A server and client with proper watch behavior

A server depends on server and client builds. We want a script that runs the server after automatically building the client and server code, rebuilds server and client code automatically, but only restarts the server if the server code changes. Client code changes are handled by a manual reload, automatic reload, or hot-module-replacement.

This case currently requires a utility like nodemon. Topologic sort only helps if the server and client are separate packages. It's tricky to have a single command that can conditionally watch or not watch because of the need to thread flags to the build commands and server.

Automatically build and run tests if the output changes, in a test runner with watch mode

This is basically identical to the server + client case. The test runner (assuming it has it's own watch mode) is the server, the test runner config is the server code, the tests and code under test is the client code.

Automatically build and run tests if the output changes, in a test runner without watch mode

In the case the script runner needs to re-run the test script itself when any input to it changes.

nodemon might insufficient in cases where you add a new test that would have been picked up by a glob pattern.

---

I can collect these and other important and unique use cases that people can contribute up into the issue body.

[trusktr]

I can't switch to npm workspaces for script running. A feature like this would be huge for NPM!

[justinfagnani]

Deno is now building Wireit-like features into Deno tasks: denoland/deno#26462