Repackage node2nix generated packages using buildNpmPackage

[ehllie]

Issue description

Currently a vast majority of NPM packages available in Nixpkgs are generates using node2nix. It generates a nix expression form the node-packages.json file. However it's very hard to maintain. Any time a new package is added to that list or updated, node2nix has to update every single package listed in node-packages.json using a code-gen script. That forces PRs that touch that part of the ecosystem to rerun the code-gen script each time only other PR is merged. This is further exacerbated by the time the script takes to run. As of writing this post, there are 408 packages in node-packages.json, and it takes ~4h to run the generation script.

Proposed solution

Nowadays an alternative tool, buildNpmPackage, is available, but plenty of the packages already listed there were added there before its creation. Not all of the packages can be build using the new tool, but I think it would be worthwhile to review how many of them could be repackaged that way. That should at the very least make the time of running the code-gen script shorter.

As such, I've listed all the packages that are build using node2nix. My intention is this issue could be used to collaborate on the repackaging efforts. If any package gets successfully repackaged, or determined to not be possible to be built using builtNpmPackage, I'd appreciate having that mentioned here so that I can update the list and move that package to an appropriate location.

List moved to https://github.com/orgs/NixOS/projects/83.

Want to contribute? Here's a general walkthrough of how you can do that

• Find a package that's not listed as repacked in this post (and better yet search for it in nixpkgs, I try to update this post somewhat frequently but I'm not infallible)
• Check what tool it's using upstream. npm/ pnpm/ yarn? does it keep it's lock file checked out in vcs? If it's using both npm and has a checked out package-lock.json it might be a good candidate
• Given the happy scenario try to make a callPackage derivation using buildNpmPackage and put it somewhere appropriate in the nixpkgs package tree.
• Remove the old package from node-packages.json and add it to aliases.nix

In case the package does not follow the happy scenario you can mention it here so I can move it to the appropriate group of packages. For an example of a package repackaged this way you can check #229512

I've discussed this idea with @SuperSandro2000 in another PR briefly, but I'd appreciate other people responsible for maintaining the node ecosystem to weigh in as well. @NixOS/node would this be the right way to go about this? Where should the repackaged packages be put? Is doing this worthwhile?

[winterqt]

Feel free to move packages out of the set, this is a net good. Not really sure how/if we want to add aliases for the old names, though. (cc @lilyinstarlight)

As a rule of thumb, more or less anything with a package-lock.json can be packaged, but some are impossible due to npm cursedness.

Feel free to request my review directly on packages, I'm happy to land things quickly and work with you on this :)

[natsukium]

Just the other day, I came across a PR to change from nodePackages to a standalone build. (#228854)
In this PR, an alias was added to override.nix to keep the nodePackages name. b4b15b6

I would also like to help migrate the package.

[winterqt]

Ah, forgot about the overrides file. That looks good, do that. Again, please request my review on these, I'm more than happy to review and get them landed.

Please do keep in mind that we expect you to maintain a package if you add it, so only migrate packages that you are okay with maintaining and testing.

[lilyinstarlight]

@winterqt Should we make a node-aliases.nix kinda like Python has now (either in top-level or still in the node-packages dir)?

That way they are easy to find and turn into throws later (as for other aliases) and we can just make the node-packages default.nix optionally include it on config.allowAliases

[winterqt]

Great catch, yes, let's do it.

[ehllie]

I've gone over a couple of the packages so far, and it seems to be not as straightforward as I initially thought it would be 😛

• @angular/cli and @commitlint/cli use yarn, and do not have a package-lock.json
• @antfu/ni and @astrojs/language-server use pnpm, and likewise have no package-lock.json
• @nestjs/cli and @squoosh/cli fail due to issue with building node-gyp, but those seem to be addressed in buildNpmPackage: patch npm to work around various roadblocks #206477, I'll investigate if that PR makes them possible to build

[SuperSandro2000]

I think we should start with everything that is easy to get done and has a top-level alias. Maybe that is even enough to get nodePackages into shape and we can save ourselves time fixing the really hard and weird cases.

[ehllie]

@natsukium thanks a lot for compiling these packages. I've edited the original post

[lilyinstarlight]

I've added aliases for nodePackages in #229639

[malob]

Just want to note that a missing package-lock.json in the source isn't necessarily a blocker for using buildNpmPackage.

The package-lock.json file can be generated and committed to nixpkgs. This is what I had to do to add github-copilot-cli:

nixpkgs/pkgs/tools/misc/github-copilot-cli/default.nix

Lines 14 to 16 in 1e237b7

	postPatch = ''
	cp ${./package-lock.json} package-lock.json
	'';

(There is precedent for this type of approach in nixpkgs with Rust packages.)

[Mic92]

Can we please NOT add 5000 lines of generated packages.json locks per node package?
https://github.com/NixOS/nixpkgs/pull/231615/files
Otherwise we would end up with ~19.1M lines of code just for 408 nodes packages. We currently have 2.8M lines of code in nixpkgs. This would significantly multiply the size of nixpkgs. I am strongly against that and would like to kindly ask for a different solution that does not bump the nixpkgs tarball to 100MB and also make it no longer possible to run nixpkgs-review on an average laptop (because of memory consumption)

[Mic92]

Last time I checked node2nix was doing a lot of the fetching in serial. However one could speed it up by pulling with 32-threads instead of just one (7min instead of 4hours based on my napkin math)

[natsukium]

Indeed, in the example above, it was undesirable to commit the 5,000 lines of package-lock.json.
On the other hand, if the upstream keeps the package-lock.json, we don't have to add it ourselves and can benefit from a much smaller number of lines in the node-package.nix.

Also, even if we could fetch in parallel, my laptop (macbook pro) only has eight threads, so it is unlikely to be that fast. (although it is enough compared to 4 hours).

[RaitoBezarius]

Can we get statistics / figures on what it would add to nixpkgs before anything?

[winterqt]

FWIW @Mic92, not every package requires that we add a package-lock.json in-tree, only some do. I'd argue that we should migrate all the ones that don't require us to do that first, and then re-evaluate.

[RaitoBezarius]

Apologies for my confusion, I didn't understand it was about adding only those who had package-lock.json in tree. 👍 for the approach!

[Mic92]

I am ok with this approach when we do this for packages that provide a package-lock.json

[SuperSandro2000]

We can also upload the package-lock.json to somewhere with buildNpmPackage.

Last time I checked node2nix was doing a lot of the fetching in serial. However one could speed it up by pulling with 32-threads instead of just one (7min instead of 4hours based on my napkin math)

That doesn't fix that it is broken in other ways and confuses dev and prod dependencies.

[ehllie]

We can also upload the package-lock.json to somewhere with buildNpmPackage.

That's definitely one way to do it, but we'd need to decide on a hosting platform for said lock files. It's probably better not to leave the choice up to the individual contributor as that would possibly make maintenance even harder than it is now, with the upstream fetcher endpoints going offline and needing to be updated. I'm also not sure if we wouldn't run into IFD issues by doing that.

But that seems to be a general issue with monorepo approach that Nixpkgs has. Any user of the repository needs to clone the entire repository regardless of how much of it they will actually use. It's obviously not cloning the binaries, but Nixpkgs is continuously growing, and I don't know if that approach is sustainable long term. However that question is beyond the scope of this issue and should probably be addressed in an RFC.

I feel like sticking to packages with pre-existing package-lock.json files and making node2nix make requests in a concurrent manner like @Mic92 suggested is the best course of action we can take atm.

As a side note, I apologise for starting the issue and then disappearing out of the blue. I had just undergone surgery and didn't have much time in the days leading up to it. I should be able to contribute a bit more now that I'm stuck in a hospital bed however ^^

[tomodachi94]

To repackage thelounge plugins, we will need to reimplement the theLoungePlugins attrset:

https://github.com/NixOS/nixpkgs/blob/02289590e9ff14e8e3aebeedd54eddccebef61bc/pkgs/top-level/all-packages.nix#L12798-L12806

[pyrox0]

To repackage thelounge plugins, we will need to reimplement the theLoungePlugins attrset:

https://github.com/NixOS/nixpkgs/blob/02289590e9ff14e8e3aebeedd54eddccebef61bc/pkgs/top-level/all-packages.nix#L12798-L12806

I think it should be implemented as two separate attrsets. one called theLoungePlugins and one called theLoungeThemes. the previous one seemed to be theLoungePlugins.plugins and theLoungePlugins.themes, which feels very clunky. Perhaps just 2 separate folders in thelounge's package folder with nix files for each? should be the simplest solution, hell if they all use the same build system, could we do a common.nix approach?

[tomodachi94]

I don't think a common.nix approach will work. The packages are packaged using a smattering of different tooling:

https://github.com/orgs/NixOS/projects/83/views/2?filterQuery=thelounge-+-status%3A%22Not+checked%22+