Init deno language support

[zebreus]

Description of changes

This PR adds a proof of concept for a buildDenoApplication function similar to the buildRustCrate function. This is still work in progress.

---

I went through the Deno manual, issues, and code and tried to collect all the information relevant to importing things and fetching dependencies based on the lockfile:

Imports

In ECMAScript modules, files are imported via import specifiers. They can be either relative (./file.ts), bare (idk/file.ts), or absolute (prot:/absolute/path/file.ts). Deno supports importing via relative and absolute specifiers. Relative specifiers are evaluated relative to the current file. Deno supports the file, data, blob, node, http, https, npm, and jsr protocols for absolute specifiers.

Deno also supports import maps, which enable us to define bare specifiers. Import maps are specified when running a script, usually via deno.json, which means that libraries cannot use import maps.

Deno can use lockfiles to lock the versions and verify the integrity of all external imports. Similarly to import maps, the lockfile is specified when running a script, so libraries cannot bring their own lockfiles.

Relative imports and imports from file: specifiers

Local imports are unproblematic because they are not dependencies but our project's source. Local imports outside our src repo are out of scope for now.

Import from node: specifiers

Deno can import Node.js built-in modules. I am unsure how Deno does this, but it doesn't produce any files in the $DENO_DIR or the vendored directories. The Node.js built-in modules are also built into Deno.

Imports from data: specifiers

Do not need to be locked or included in dependencies.

Imports from blob: specifiers

Blob imports do not need to be locked or included in dependencies.

Imports from https: (and http:) specifiers

Imports from URLs. The deno.lock version 3 solves the issue with redirects and directly lists all URLs we need to fetch. The more significant issue is caching only the fetched file contents but not the received HTTP headers. This can be a problem because Deno seems to change its behavior based on the headers, as documented in denoland/deno#19512. We can mitigate these issues by just ignoring all headers. I am not even sure if this is an issue anymore because I could not find any trace of the headers in the output of deno cache --vendor --node-modules-dir, so when we do deno --cached-only run afterward, the behavior should be unaffected by the headers. Not sure if this leads to failing applications because the behavior changes when using --cached-only, but that would be a general deno problem, not a nix problem.

Imports from npm: specifiers

Deno can import packages from npm registries. npm imports consist of a package name, a version requirement, and an optional path in the package.

If the version requirement of a package is a specific version, we can assume that this package is deterministic. I think this is an even stronger assumption than deterministic content behind URLs, as all relevant public registries have the policy that the content of a specific version is immutable ( (npm.js, deno.land/x, jsr.io). However, the dependencies of a version are usually based on relatively broad version requirements, which means that they need to be locked.

The Deno lockfile v3 sufficiently locks the packages. However, it seems like Deno only supports one npm registry at a time. The lockfile tracks the hash for every required package/version combination and the exact package/version combination of every resolved dependency. This is different from npm package-lock.json files, which track incomplete versions for dependencies if multiple packages request different versions of the package. Deno does all the heavy lifting for us here; we don't need to determine which version a package uses.

Deno supports using a vendored node_modules directory in the project directory. We need to create that directory with Nix as a part of the dependencies. All npm package sources live in directories like node_modules/.deno/PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME. There is no node_modules folder in that folder. I am surprised that this works, but it does. See the list below for more details on the exact structure. There also are relative symlinks at node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME for every direct dependency of our packages that point to the correct version of the package in node_modules/.deno. I think the structure is relatively sane, as we have a clean directory without dependencies for every package. Everything else is just symlinks.

MANGLED_PACKAGE_NAME refers to a package name in which all slashes have been replaced with +. For example, @webview/webview becomes @webview+webview. As + is not allowed in normal package names, this can not produce collisions.

PACKAGE_NAME refers to a package's name. If it is a scoped name, it does not include the scope; for @scope/name, this is just name.

MAYBE_PACKAGE_SCOPE refers to the scope of a package. If it is not a scoped package, this is just empty. For example, for @scope/name, this is @scope. For an unscoped package like tsx, this is nothing ``.

The dependencies are stored and symlinked in the node_modules folder as follows:

• node_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME is the directory that contains the actual npm package. It only includes the files fetched from npm and no node_modules folder.
• node_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME is a symlink to node_modules/.deno/MANGLED_DEPENDENCY_NAME@RESOLVED_VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME. These links exist for all direct dependency of the package in node_modules/.deno/MANGLED_PACKAGE_NAME@VERSION/node_modules/MAYBE_PACKAGE_SCOPE/PACKAGE_NAME. I am surprised that this structure works with Node.js imports, but it does.
• node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME is a symlink to node_modules/.deno/MANGLED_DEPENDENCY_NAME@RESOLVED_VERSION/node_modules/MAYBE_DEPENDENCY_SCOPE/DEPENDENCY_NAME. It exists for every direct dependency of our Deno project.

However, there are more files in node_modules.

• node_modules/.deno/.setup-cache.bin is a binary file that keeps track of the symlink structure in node_modules. Having this file isn't strictly necessary, as Deno will recreate it at runtime. However, Deno will fail if it cannot create that file at runtime, which it will if we load node_modules from the Nix store. In Deno 1.44, the file is also not deterministic because the entries are ordered randomly. I submitted a patch; this will be fixed in Deno 1.45.
• node_modules/.deno/.deno.lock and node_modules/.deno/.deno.lock.poll are probably used as a mutex. They are deterministic. Deno does not mind if they are missing. Deno also does not mind if it can not write these. We don't need to include them in our node_modules folder.
• node_modules/.bin/EXECUTABLE_NAME are symlinks to all entries in the bin sections of all direct and indirect dependencies. I am still determining how Deno handles conflicts if multiple packages export executables with the same name, but it seems deterministic.
• node_modules/.deno/PACKAGE_NAME@VERSION/.initialized is an empty file for every npm package. I assume Deno uses these to track whether a package folder is ready to be used. These files do not strictly need to exist, as Deno will create them at runtime. However, Deno will fail if it can not create them because the directory is in the Nix store.

I think these two lists are exhaustive, but I have not checked Deno's source code to verify that.

A bit tricky with using vendored node_modules is the .setup-cache.bin in the vendored folder, which maps module names to module names with versions, to keep track of the symlink structure in node_modules. It currently is a binary file without a clearly defined structure. I submitted a patch to deno to make it at least deterministic.

Imports from jsr: specifiers

Deno can resolve jsr: specifiers to import packages from jsr.io. Similar to npm: imports, jsr: imports consist of a package name, a version requirement, and an optional path in the package.

Deno will then decide the actual version for each jsr module and store them in the lockfile. Unlike npm packages, the concept of packages is only used to resolve the versions of dependencies. Afterward, Deno maps JSR imports to https URLs. If we know the resolved version, we can essentially replace a jsr import with the correct https import. So, for example, jsr:@std/path/join is equivalent to https://jsr.io/@std/path/0.225.2/join.ts. Note that it also added the .ts extension to the URL. JSR package version metadate can contain an export map that maps paths in the specifier to files in the package. In this case, the export map contains an entry like "./join": "./join.ts".

JSR imports are basically a layer of syntactical sugar on top of https: imports, with added support for version requirements. It also brings a few QoL improvements like the ability to use import maps to library authors, at the cost of only being able to publish at JSR because you need a build step (for various reasons).

When vendoring jsr: imports, the downloaded files are stored in the vendor directory like https: imports. In fact, every file is placed in the correct location, corresponding to its URL. However, in addition to the TypeScript files, the vendor directory also contains vendor/jsr.io/SCOPE/PACKAGE/meta.json for metadata about the available versions and vendor/jsr.io/SCOPE/PACKAGE/VERSION_meta.json for every version of the package that we use. The version metadata file also includes an import map for the package.

Annoyingly, files imported via jsr: specifiers are not represented like https imports in the lockfile but in a data structure similar to npm modules. packages.specifiers tracks the specifiers for all directly imported jsr and npm packages and their resolved versions. Each specifier's versions and resolved dependencies are tracked in packages.npm and packages.jsr, respectively. Both are objects with packagename@resolvedversion as keys. The values are objects with two fields; integrity contains a checksum for the package, and dependencies contains an array of specifiers of its dependencies. I have not yet found documentation on calculating the integrity checksum of JSR packages. Unlike npm modules, the specifiers in dependencies are not necessarily absolute, so something like jsr:@std/assert@^0.226.0 instead of jsr:@std/[email protected] is possible. We can resolve these via the packages.specifiers map, but that is an extra step.

I found the following reproducibility issues with the current vendoring strategy for JSR modules:

1. The vendor/meta.json contains all published versions of the package and is constantly changing because of this. Which is not deterministic at all. A file like this should not exist in the vendor directory because the information is not relevant for vendoring; it is only for caching when using vendored dependencies and only when updating them. This file must exist when running with --cached-only deno. For npm imports, this type of information is always stored in $DENO_DIR in registry.json files. jsr imports should handle this in a similar fashion.
2. The vendor directory only contains the files actually used by the script, but the lockfile does not include information about which files they are. That is a problem because the output does not depend on the lockfile. We could, however, always just fetch all files from the package.
3. The lockfile only contains the integrity checksum for the whole package but not the individual files. This makes integrity checks impossible if we are not fetching the complete package.

My preferred solution for problems 2 and 3 is to include all files in vendor with their checksums in deno.lock. This way, we also don't have to bother with the packages.jsr structure, as that is only needed for version resolution, which Deno already did for us.

With this behavior, Deno reintroduces a few of the problems Node.js imports have.

Ways of triggering imports

I tested various ways of triggering imports to see if they make a difference for the lockfile.

JSX import with comment

Deno offers a few weird ways of importing jsx runtimes

/** @jsxImportSource https://esm.sh/preact */

export const Thing = <span>hi</span>

In the above, /jsx-runtime is appended to the path and is treated like a normal import from https://esm.sh/preact/jsx-runtime. We can also use imports with an npm specifier, in which case nothing is appended.

The jsx source can also be defined in deno.json; in that case, it is implicitly imported from all jsx and tsx files. This means that it is only imported if we import at least the tsx file.

{
  "compilerOptions": {
    "jsx": "react-jsx",
    "jsxImportSource": "https://esm.sh/preact"
  }
}

If we don't define a jsx import in deno.json or via @jsxImportSource comment, deno will import nothing and fail with a React not defined error when using tsx files.

Dynamic imports

Deno supports dynamic imports via the import function. When doing dynamic imports with string literals as URLs, deno will cache and lock the dependencies. When an expression is used as the URL, deno is (obviously) unable to lock and cache that dependency.

However, when we are running deno with --cached-only (which we are)

Caches dependency, does not require networking permissions at runtime, works at runtime

const dynamicLiteral = (await import('https://deno.land/[email protected]/version.ts'));

Caches dependency; does not require networking permissions at runtime; works at runtime

if (false) {
  const dynamicLiteral = (await import('https://deno.land/[email protected]/version.ts'));
}

Does not cache dependency; requires networking permissions at runtime; fails at runtime because the module is not in cache.

const url = 'https://deno.land/[email protected]/version.ts'
const dynamicExpression = (await import(url));

Caches dependency; works at runtime

import staticLiteral from 'https://deno.land/[email protected]/version.ts';
const url = 'https://deno.land/[email protected]/version.ts'
const dynamicExpression = (await import(url));

Caches dependency; works at runtime

const dynamicLiteral = (await import('https://deno.land/[email protected]/version.ts'));
const url = 'https://deno.land/[email protected]/version.ts'
const dynamicExpression = (await import(url));

I am not sure about the behavior of dynamic imports. The table below shows some results of different combinations in different environments.

Testing the behavior of different dynamic imports

cached-only	Already cached	literal import	expression import	literal before expression	Crashes	Asks for network	Cached after deno cache
no	no	no	no	no	no	no	no
no	no	no	yes	no	no	yes	no
no	no	no	unreachable	no	no	no	no
no	no	yes	no	no	no	no	yes
no	no	yes	yes	no	no	no	yes
no	no	yes	yes	yes	no	no	yes
no	no	yes	unreachable	no	no	no	yes
no	no	yes	unreachable	yes	no	no	yes
no	no	unreachable	no	no	no	no	yes
no	no	unreachable	yes	no	no	no	yes
no	no	unreachable	yes	yes	no	no	yes
no	no	unreachable	unreachable	no	no	no	yes
no	no	unreachable	unreachable	yes	no	no	yes
no	yes	no	no	no	no	no	yes
no	yes	no	yes	no	no	yes	yes
no	yes	no	unreachable	no	no	no	yes
no	yes	yes	no	no	no	no	yes
no	yes	yes	yes	no	no	no	yes
no	yes	yes	yes	yes	no	no	yes
no	yes	yes	unreachable	no	no	no	yes
no	yes	yes	unreachable	yes	no	no	yes
no	yes	unreachable	no	no	no	no	yes
no	yes	unreachable	yes	no	no	no	yes
no	yes	unreachable	yes	yes	no	no	yes
no	yes	unreachable	unreachable	no	no	no	yes
no	yes	unreachable	unreachable	yes	no	no	yes
yes	no	no	no	no	no	no	no
yes	no	no	yes	no	yes	yes	no
yes	no	no	unreachable	no	no	no	no
yes	no	yes	no	no	yes	unknown	yes
yes	no	yes	yes	no	yes	unknown	yes
yes	no	yes	yes	yes	yes	unknown	yes
yes	no	yes	unreachable	no	yes	unknown	yes
yes	no	yes	unreachable	yes	yes	unknown	yes
yes	no	unreachable	no	no	no	no	yes
yes	no	unreachable	yes	no	yes	unknown	yes
yes	no	unreachable	yes	yes	yes	unknown	yes
yes	no	unreachable	unreachable	no	no	no	yes
yes	no	unreachable	unreachable	yes	no	no	yes
yes	yes	no	no	no	no	no	yes
yes	yes	no	yes	no	no	yes	yes
yes	yes	no	unreachable	no	no	no	yes
yes	yes	yes	no	no	no	no	yes
yes	yes	yes	yes	no	no	no	yes
yes	yes	yes	yes	yes	no	no	yes
yes	yes	yes	unreachable	no	no	no	yes
yes	yes	yes	unreachable	yes	no	no	yes
yes	yes	unreachable	no	no	no	no	yes
yes	yes	unreachable	yes	no	no	no	yes
yes	yes	unreachable	yes	yes	no	no	yes
yes	yes	unreachable	unreachable	no	no	no	yes
yes	yes	unreachable	unreachable	yes	no	no	yes

Various other things

HTTP headers

We have previously seen that Deno sometimes treats files differently depending on the received http headers. However, some package registries also treat us differently depending on the headers we send.

$ curl https://esm.sh/[email protected]/jsx-runtime
/* esm.sh - [email protected]/jsx-runtime */
import "/stable/[email protected]/esnext/preact.mjs";
export * from "/stable/[email protected]/esnext/jsx-runtime.js";

If we set our user agent to Deno/1.44.4, we get a different result:

$ curl -H 'User-Agent: Deno/1.44.4' https://esm.sh/[email protected]/jsx-runtime
/* esm.sh - [email protected]/jsx-runtime */
import "/stable/[email protected]/denonext/preact.mjs";
export * from "/stable/[email protected]/denonext/jsx-runtime.js";

If we set our user agent to Deno/1.13.0, we again get a different result with deno instead of denonext:

$ curl -H 'User-Agent: Deno/1.13.0' https://esm.sh/[email protected]/jsx-runtime
/* esm.sh - [email protected]/jsx-runtime */
import "/stable/[email protected]/deno/preact.mjs";
export * from "/stable/[email protected]/deno/jsx-runtime.js";

This shouldn't be a problem if we send the same headers as Deno. But which version? It's probably best if we either decide on one and keep it or use the same version as Deno.

This should not be a big problem as there are not many sites that do this and even on esm.sh we get responses with denonext for all recent versions and that is unlikely to change. I think we should just pin it to 1.45.0 and add a switch in the future. If we would use the correct Deno version, we would also have to write the version number into the output, which would mean that the outputHash would change on every Deno update.

package.json support

I will just ignore that for now and abort if a project contains a package.json and no deno config. Although it should be possible and not that hard.

Private npm registries

We will need to add support for these, but not now.

https://docs.deno.com/runtime/manual/node/private_registries/

Proxies, private repos, and auth tokens

Deno supports HTTP and HTTPS proxies. I don't know how Nix handles proxies; further investigation is needed.

https://docs.deno.com/runtime/manual/basics/modules/proxies/

Deno provides a way to specify an auth token for specific domains. Further investigation is needed.

https://docs.deno.com/runtime/manual/advanced/private_repositories/

FFI / native code

Deno allows calling into native code using Deno.dlopen but does not provide a way of importing or packaging binaries. So, that is not part of the lockfile.

Deno once had a plugin system, but that was removed in 1.13. Good, this way, we do not have to worry about that

WebAssembly

WebAssembly is usually fetched at runtime, so we don't need to worry about it. However, Deno may add it to the lockfile in the future, so we might need to keep an eye out for that.

TODO

Tasks in Deno

None of the these are strictly required and we should be able to work around all of them. However having those features in Deno would make things a lot cleaner and less hacky.

• Lockfile format changes
  • Add the hashes of individual files fetched from JSR packages to the lockfile
  • Adjust the dependencies of JSR imports in the lockfile to always use the same specifier that is also used as the key in packages.jsr
• Changes to files in vendored directories
  • Tread the meta.json file like the registry.json files and always store them $DENO_DIR
  • Make sure all non-downloaded files in the node_modules directory are deterministic
  • Convert node_modules/deno/.setup-cache.bin to a JSON file
  • Sort the keys in vendor/manifest.json (feat: make the cache manifest more deterministic denoland/deno_cache_dir#53)
• Make Deno work with an empty $DENO_DIR and read-only vendor directories
  • Don't require meta.json files with --cached-only --lock
  • Don't require registry.json files with --cached-only --lock
• Add a way to check if a script is completely locked by the lockfile without internet access
• Add a way to verify vendored dependencies against the lockfile

Tasks in nixpkgs

• Figure out a good interface for the buildDeno... functions
  • Look into the available deno environment variables and configuration options and if they are relevant for us
  • Maybe use finalAttrs to make overriding the dependencies hash easier than in buildRustCrate
  • Learn from the whole dependency story in buildRustCrate
• Figure out whether we are able to do Typescript type-checking at all, or if we are missing Deno features.
• Fetch with the correct user-agent

Classic nixpkgs checklist

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[emilazy]

The last time I looked into this, it was almost possible to process the lock files directly in Nix and download each dependency directly in a granular fashion rather than having to bundle it all in one large fixed‐output derivation. denoland/deno#19512 was the issue I ran into; it’s since been closed as fixed, but I remember when I looked at the fix I wasn’t confident it was enough to fully close up the issues. Do you think you’d be up for looking into whether that approach might work now? I think it’d be a much nicer solution if it turns out to be possible.

I can try to publish the incomplete draft Nix code I came up with while playing with that if it’d be of any use.

[ehmry]

I think it would be ideal to support the buildDenoApplication (finalAttrs: { pattern. It's not the expected behavior yet but nice to have. Here is an example of a conversion of buildGoModule: https://github.com/NixOS/nixpkgs/pull/321791/files

[ehmry]

It might be possible to reduce this noise by using builtins.intersectAttrs (builtins.functionArgs fetchDenoDeps) args somewhere futher down.

[ehmry]

build-nim-package.nix is convoluted and hard to understand because it uses fixed-point style right off, it does lockfile processing, and it has a function for overriding the lockfile.

The solution might be too much of a hack to be a good example but think the Nim problem is conceptually similar to Dino in that the Nim package description format and lockfiles are not reproducible by Nix standards. The current Nim solution is to use a tool to create a Nix specific lockfile when manually updating packages and commit the lockfile into Nixpkgs. There isn't anything so show that this will scale well into the future.

The solution that I've been working on is to reuse an independent software-bill-of-materials standard and add some Nix annotations but nothing has been merged to Nixpkgs. This comes with an alternative implementation of build-nim-package.nix - build-nim-sbom.nix.

[zebreus]

@emilazy Thanks for the pointer to that issue. As far as I can tell, the redirect problem has been fixed with deno.lock v3, which records redirects.

The second issue was about response headers, but I am not sure if they matter if we are fetching and building the dependencies independently from deno and are then supplying them as vendored dependencies. Deno recently gained a feature for vendoring dependencies, which is what I am currently using for building a fixed output derivation. However, I don't think it stores the received headers anywhere, so I don't think they matter when running with --cached-only --lock --vendor.

The bigger problem is that Deno sends its version number in the user-agent header of its http requests, and some sites change their response depending on the version. I put an example of this in the PR description.

I would love to have a look at your draft of the Nix code; it is much appreciated.

[emilazy]

Yeah, I handled the user agent thing like this:

      downloadPath = fetchurl {
        inherit url sha256;
        # "By default, esm.sh checks the User-Agent header to determine the build target." - https://esm.sh/#docs
        curlOptsList = ["--user-agent" "Deno/${deno.version}"];
      };

I don’t even have my old draft in version control right now, but I’ll try to get it very minimally cleaned up and published within the next few days; feel free to ping me here again if I don’t report back soon. It’s over a year old now and definitely not ready for prime‐time, but if Deno has evolved enough that we can actually use a pure Nix approach with fine‐grained dependencies like that it’d be a wonderful improvement on the huge fixed‐output derivation blobs approach most ecosystems are stuck with.

[emilazy]

FWIW the headers stuff mattered at least when interpreting URLs as JavaScript vs. JSON, and possibly for pointers to TypeScript type definitions; I had a proof‐of‐concept demonstrating the divergence, but I never got around to sharing it with the Deno people I don’t think. Maybe they changed behaviour enough by now that it no longer applies. It also might be marginal enough that we don’t have to care.

[zebreus]

@emilazy How is your progress with cleaning up your draft code? I would also be happy about a not cleaned up version.

[emilazy]

Here you go. I haven’t really done anything to update it but you can get the basic gist, at least. I think if you messed with the headers thing you could reproduce the issues I had with importing JavaScript vs. JSON files, but it’s possible Deno’s behaviour has changed since. If you can’t manage to make it break, I can try to write up a reproducer. It’s unclear what the ecosystem impact would be of just punting on that problem, anyway, but I think there are other headers that seemed relevant. I meant to spend more time pushing Deno upstream to define the list of headers they care about and include them in their lock file hashes, but I never got around to it.

Note that while I think the developer experience of a non‐IFD, non‐FOD based solution like this is second to none, and it’d be a great synergy between the Deno and Nix ecosystems if it could be made to work elegantly, there is some backlash to vendoring lock files in Nixpkgs currently, so it may be that a FOD‐based approach like yours is more palatable for Nixpkgs use. Ideally, both would be supported, and share as much interface as possible.

I hope the code is useful or at least interesting to you. Please let me know if you have any questions or want me to test something out.

[Eveeifyeve]

I mean this should be in a hook like the others (pnpm,yarn), I am planning to make a hook for deno but I think this should not have to be needed for a builder as hook basicly give you the overrides of what the hook does and it's just generally much better than a builder.