Proposal: deno vendor

[kitsonk]

Context

Deno aspires to provide a full featured set of development tools baked into the single deno CLI. At the same time, Deno provides an open platform where code can be imported just as easily as a browser can fetch a web page. Deno stores remote dependencies in a cache commonly referred to as the DENO_DIR. While Deno allows you "relocate" this directory at runtime, and the files in the cache are editable, it does not solve all the use cases in a well supported way.

Use cases

• As a developer, I want to be able to quickly edit a remote dependency to investigate a coding issue.
• As a developer, I want to be able to store all of my remote dependencies for a given project and check them into my source code to ensure that those dependencies don't change or can't be altered.
• As a developer, I want to be able to develop a workload with the Deno CLI and be able to test it and then deploy it on the edge with Deno Deploy with a fixed set of dependencies, not relying upon availability of other hosts or services for my application.

All of these use cases also are constrained by:

• Without needing to edit or change the code I have authored.
• I want to be able to quickly "revert" if needed.

Proposed Solution

Add a subcommand to the Deno CLI named deno vendor. The subcommand would work similar to the deno run subcommand, in that it would take a target script. For example:

> deno vendor main.ts

Would analyse main.ts and all of its dependencies. For those dependencies that are remote dependencies, it would write their source out locally to the _vendor path, along with _vendor/import_map.json, which would remap all the remote dependencies to local ones.

The concept would then be then that changes could be made to the local "vendored" code and be reflected in the application by simply using the output import map:

> deno run --import-map _vendor/import_map.json main.ts

(Note: having to be explicit about this import map isn't the ideal design, and suggestions below in other complimentary features would be able to make this just so then all the user would need to do is deno run main.ts)

This would also allow specific dependencies to be vendored, without having to vendor everything. For example if someone wanted to just vendor oak and all of it dependencies, but leave everything else to continue to be remote:

> deno vendor https://deno.land/x/[email protected]/mod.ts

Other flags that should be supported:

• --output - specify the output path, which defaults to _vendor of cwd.
• --import-map - specifies an upstream import map that should be used as the base when calculating the vendored import map
• --config - specifies a configuration file, which will be considered when vendoring.
• --info - output human readable information about the current state of the _vendor directory.

Other flags/features that might be supported, or might be added after the "MVP" feature:

• --ignore - a glob pattern of dependencies to exclude/ignore
• The ability to remove specific dependencies from _vendor (which can be accomplished from editing the _vendor/import_map.json directly, but might be unapproachable for some devs)

Semantics

• Once the remote dependencies are identified, they would be written out in a way to minimise the need of remapping the paths. For example https://example.com/pkg/mod.ts would get written out to ./_vendor/https/example.com/pkg/mod.ts. In cases where the remote dependency cannot be written to disk following the simple algorithm, it will be written out in as close to a human readable way as possible.

• Once the "manifest" of the _vendor is calculated, the import map will be generated with the smallest set of entries possible. For example a remote import of https://example.com/pkg/mod.ts would get an entry like the following:

  {
    "imports": {
      "https://example.com/": "_vendor/example.com/"
    }
  }

  But if there was something like https://example.com/pkg/mod.ts and https://example.com/pkg/lib?dev something like the following would be written out:

  {
    "imports": {
      "https://example.com/pkg/lib?dev": "_vendor/example.com/pkg/lib_a02vf0.ts",
      "https://example.com/": "_vendor/example.com/"
    }
  }

• If a _vendor or output path already exists, the _vendor/import_map.json and output dependencies would be updated.

• A design limitation would be that some dynamic imports that cannot be statically analysable would be missed. A consideration might be to warn on such dependencies being present.

• It is assumed that at this point in time, other state/meta data does not need to be stored, but might need to be. If this is the case, then the state should be stored in the _vendor/import-map.json under additional private keys (e.g. "__deno_vendor": {}). The import map specification specifically allows additional keys without invalidating the file, and this would help ensure that the output of deno vendor is contained, repeatable and portable.

• When an existing _vendor is updated, and for some reason a remote dependency has changed, import map "scopes" can be used to allow two different versions of the same dependency to exist in the vendor directory.

Other features which improve/compliment this feature

• Automatic detection of deno.json/deno.jsonc - This is already in progress.
• Ability to express an import map in deno.json/deno.jsonc. This would allow the feature a) identify an existing import map that should be amended with the vendor mappings as well as "point" the configuration at the vendor import map.
• Add the Deno: Vendor Dependencies to the vscode_deno extension.
• Ability to express and support import maps with Deno Deploy. This would allow someone who had vendored dependencies to deploy it to Deploy.

Specific advantages to this proposal

• It aligns to the web platform standards, providing a portable solution that is independent of Deno CLI.
• Integrates well into existing developer workflows.
• Because all the "magic" is done in import maps, it gives the end user fine grained control over the behaviour, allowing them to address edge cases or different workflows without "breaking" the feature or making the feature unusable.
• The feature is open, transparent, portable, easily reversible, non-destructive and totally optional.
• Easily integrate-able into editors, so that a dev would run the command and presto, all the remote dependencies are local and editable.

Other options considered

• Making access to DENO_DIR easier - currently, users can edit files in the DENO_DIR. There are significant risks to this, like depending on the dependency type and the state of the cache, files may or may not be invalidated in a way that causes the program to emit properly. Also, it is a reasonable expectation that we should consider DENO_DIR and its schema "private" and while accessible, the layout and format may need to change as Deno evolves, and if people are tightly coupled to this, it becomes problematic and could easily hold back the development of Deno for fear of breaking integrations to non-public behaviours. Another risk is that it would not be (ever) compatible with Deno Deploy.
• Adding specific features to the language server to edit dependencies - this would then require people to use an editor with a language server to take advantage of this feature, which is not realistic or desired.

[teleclimber]

If a _vendor or output path already exists, the _vendor/import_map.json and output dependencies would be updated.

I just want to point out that this is a good and important characteristic of this design. It means I can do this:

$ deno vendor main.ts
$ deno vendor dynamic.ts
$ deno run --no-remote main.ts

(Where dynamic.ts is imported asynchronously by main.ts.)

The cumulative nature of vendor means I can handle all deps including the dynamically loaded ones, something that bundle and compile I think don't let me do.

[tmikaeld]

Thank you, this suggestion is perfect!

I'm working on a quite large Deno App with my team that has a few Deno dependencies (Like Oak), but our security model won't allow us to pull in code from external links, this proposal will solve this issue once and for all.

[aaronhuggins]

@tmikaeld I'm working on a project that's probably relatively small, but where vendoring would allow me to "harden" the app against scenarios where there's no network after app distribution.

This proposal is a perfect fit; and it's inspiration to do vendor my modules like this by hand for my project in anticipation of the CLI command.

[justinmchase]

When you call deno cache you can pick which dir it caches into... how is this different?

[kitsonk]

When you call deno cache you can pick which dir it caches into... how is this different?

1. deno cache lays out files in a not very usable way.
2. You shouldn't really edit cache files.
3. It isn't portable to other runtimes (e.g. Deno Deploy).
4. It is all or nothing, this proposal (specifically with the import map) allows individual/partial choices of vendoring.
5. Plus everything stated in the other options considered above

• Making access to DENO_DIR easier - currently, users can edit files in the DENO_DIR. There are significant risks to this, like depending on the dependency type and the state of the cache, files may or may not be invalidated in a way that causes the program to emit properly. Also, it is a reasonable expectation that we should consider DENO_DIR and its schema "private" and while accessible, the layout and format may need to change as Deno evolves, and if people are tightly coupled to this, it becomes problematic and could easily hold back the development of Deno for fear of breaking integrations to non-public behaviours. Another risk is that it would not be (ever) compatible with Deno Deploy.

[justinmchase]

not very usable way

How do you intend to use them other than to just load them as cached files?

You shouldn't edit cached files

You shouldn't edit vendor files either, no? This would constitute a license violation most of the time and if you want to edit 3rd party code you should fork it and release as an independent package and just cache it as any other library.

What kind of use or modification are you thinking about here? It honestly sounds like cache is sufficient but you just need to adjust some patterns you may have previously relied upon in the past. The "vendor" use case here sounds very C++ish, where modern modularity is not easy or common.

[kitsonk]

How do you intend to use them other than to just load them as cached files?

From the use cases:

• As a developer, I want to be able to quickly edit a remote dependency to investigate a coding issue.

Specifically, users of Deno have come across several scenarios where this is a common pattern. Like for example you include a remote dependency and are getting some sort of type error and you want to investigate, or you are consuming a 3rd party library and something isn't working and you would like to be able to drop into that code in the editor and do a console log statement or insert a debugger; statement to investigate an issue.

This would constitute a license violation most of the time

Really? I am not a lawyer, but I am pretty sure you aren't either.

It honestly sounds like cache is sufficient but you just need to adjust some patterns you may have previously relied upon in the past

The good thing, is if you don't like it, you won't have to use it. 😄

[tmikaeld]

This would constitute a license violation most of the time

How would it do that? You're allowed to make changes to most open source license, the legality of it applies to distribution.

[justinmchase]

...to investigate a coding issue.

Ok got it, that does make sense. I thought you mean actually altering it permanently and then checking it in which is probably unadvisable.

[jsejcksn]

This seems like a candidate for the config file. (precedent refs: 1, 2, 3)