API - Document When a Plugin Should Call doResolve()

[webpack-bot]

Do you want to request a feature or report a bug?
Documentation request

What is the current behavior?

No documentation about how plugins are processed

What is the expected behavior?

Every webpack plugin follows the following structure:

resolver.plugin(this.source, function(request, callback) {
    if (something) {
         resolver.doResolve(target, obj, "aliased with mapping '" + name + "': '" + ...)
    } else {
         callback(...);
    }

Can anyone please explain when I should call doResolve(...) and when callback(...). I've found this phrase in the webpack docs:

To pass the request to other resolving plugins, use the this.doResolve(types: String|String[], request: Request, callback) method

However, I don't know what to make of it.

It seems that doResolve starts the process from the start - resolve types are repeated. Inside doResolve the callback.stack has the following entries:

[
    "resolve: (/d/...",    <------------- resolve here
    "new-resolve: (/d/...",
    "parsed-resolve: (/d/...",
    "described-resolve: (/d/...",
    "resolve: (/d/...",    <--------------- and it's repeated here
    "new-resolve: (/d/...",
    "parsed-resolve: (/d/...",
    ...
]

Here is the screenshot. Why does that happen?

Here is how the stack looks like in doResolve.

And also all plugins that hook into resolve events use the function getInnerRequest(resolver, request);. What is inner request?

If this is a feature request, what is motivation or use case for changing the behavior?

It's hard to write a plugin without understanding the processing flow. I'm not asking for a full-blown detailed documentation, maybe just a small passage explaining the matter in general terms so that I get correct mindset when exploring the code through the debugger.

---

This issue was moved from webpack/webpack#5380 by @sokra. Orginal issue was by @MaximusK.

[skipjack]

There is a fair amount of documentation here on the plugin interface and another page about how to write a plugin:

https://webpack.js.org/api/plugins
https://webpack.js.org/development/how-to-write-a-plugin

However I don't see anything on doResolve which should probably be mentioned here:

https://webpack.js.org/api/plugins/resolver

or maybe on the How to Write a Plugin page.

[skipjack]

@TheLarkInn can you take a look at this?

[maxkoretskyi]

@skipjack , yeah, thanks, I've seen those. They are pretty basic. There's nothing on doResolve and calling callback(). I've spent all day today debugging resolver and I think I now understand how the workflow goes for resolve plugins. I'm planning to write an article based on my findings so hearing anything from the core team is very welcomed. I've also seen this talk which is great but unfortunately doesn't go deeper into the details.

For example, each plugin takes two parameters:

function NextPlugin(source, target) {
	this.source = source;
	this.target = target;

What are these?

[skipjack]

@MaximusK sorry for the delay...

What are these?

Frankly I'm actually not too well versed in plugins (though I am starting to work more with them). I pinged @TheLarkInn who should be able to help clarify things a bit better than I can.

[TheLarkInn]

There is a really neat graphic that @sokra sent me one time. The resolver plugins use a waterfall/sequential try catch pattern in which once a path not find via first plugin, it then passed the info it did it did not collect and then call doResolve to the next resolve hook that another plugin will listen to. Let me find the graphic.

[TheLarkInn]

Yesssss I found it!!!!!!!!!

So this should help you understand the waterfall pattern. So you always start at A and then webpack resolver will search each resolution pattern, and it will cycle back to another point based on certain conditions (see the flow graph)

[TheLarkInn]

The labels that have dashed lines represent the resolve properties you can configure in webpack (and technically in enhanced-resolve itself if used outside of webpack).

[maxkoretskyi]

@TheLarkInn , hey Sean, thanks for the diagram. I'll study it. I'm willing to contribute to webpack's sources through PR's. Do you know of any bug in resolver functionality that I can start with?

[TheLarkInn]

I think for enhanced-resolve the most actionable changes are perf ones. I'd encourage you to take a look at enhanced-resolve issues list and see whats there. By all means you can ping me whenever.

[maxkoretskyi]

@TheLarkInn , ok, cool, I'll take a look. Do you respond to DM's on twitter? I tried to reach out to you on twitter with a tweet a while ago but got no response so I'm wondering if DM is an option. I'm also thinking that putting up some lower level docs on resolver will also be beneficial to the community.

[TheLarkInn]

I do most of the time, if you can't reach me, feel free to ping on here. Also if you are still interested in working on this we could add you to our documentation team slack etc. Let me know!

[mlcohen]

Hi all --

I'm someone who has also been spelunking through a lot of Webpack's code. Specifically Webpack's enhanced-resolve library which, from what I understand, is the meat and potatoes behind webpack's resource resolving logic. I was motivated to dig into the library because I was having issues trying to get the postcss-import-webpack-resolver module to work that makes use of the enhanced-resolve library. Specifically, module aliases weren't being resolved. Anyway, from all the digging around I did, perhaps I can help provide some information that may be useful to this thread.

I spoke with @TheLarkInn on Twitter and he was very helpful in getting some of my bearings straight since the code at first is admittedly difficult to navigate and debug; especially because of the way the plugins are all dynamically put together.

Moving along...

Looking at enhanced-resolve from a 30,000 foot view, the Resolver's doResolve is the primary glue the gets of all the plugins that assist in resolving a resource to work together. In a nutshell, the plugins that have been applied to a Resolver instance will call the resolver's doResolve method in order get other plugins to do some work. Kind of like this:

resolver.resolve -> doResolve -> pluginA -> doResolve -> pluginB -> doResolve -> ...

The order in which resolve plugins are invoked is based on a source type they are applied against and a target type they in turn supply to doResolve. In the enhance-resolve library, there are a couple of these types. One such type is "resolve" which the resolver supplies to doResolve to get the ball rolling (this.doResolve("resolve", ...)).

You know what source type each plugin is applied against based on the first argument given to them in the ResolverFactory. For instance:

new AliasPlugin("described-resolve", item, "resolve")

Here the AliasPlugin instance has a "describe-resolve" source type and a "resolve" target type. So when the resolver is instructed to execute plugins registered against the "describe-resolve" type, each plugin registered against that type will be executed in sequence. For the plugin that was able to satisfy the request, it will in turn call the resolver's doResolve method with its corresponding target type, which in the case of the alias plugin above is "resolve". (That plus a request object to further along the process.) This is what more or less drives the engine.

Of course there's starting and running the resolve process but you also need it to stop. This happens when the resolver is able to satisfy a request or not satisfy a request. Satisfying a request means that the resolver was able to find the resource in the file system (a resolver requires a file system to work). When the resource is found (i.e. it is determined that the resource does exist for a given path), the resolver invokes the original caller's callback with a result containing an absolute path to the resource in the file system. If the resolver was unable to find the resource in the file system, it will invoke the original caller's callback with an error.

Since the resolver is dependent on the various plugins to do a lot of the work, what does it really mean for a resource to be resolved? Well, it comes back to types and how plugins are associated with them. There are dynamic sequences the resolver will go through to try and satisfy a request. One such sequence may look like this:

resolve
parsed-resolve
described-resolve
after-described-resolve
relative
described-relative
raw-file
file
existing-file
resolved

Looking at the types, you can infer that the resolver: starts by kicking off the resolve process; parsing the request (breaking a request into pieces to help with the resolving process); going through a series of motions attempting to generate an absolute path to a file; determining that the file does indeed exist in the file system; and then finally confirming that the request has been resolved. There is at least one or more plugins associated with each type in the sequence, and each plugin does a bit of work to try and advance things along with the aid of the resolver's doResolve. For instance, the file -> existing-file step, it is handled by the FileExistsPlugin plugin:

new FileExistsPlugin("file", "existing-file")

And for the existing-file -> resolved step, that's handled by the NextPlugin plugin which is nothing more than a simple pass-through:

function NextPlugin(source, target) {
	this.source = source;
	this.target = target;
}
module.exports = NextPlugin;

NextPlugin.prototype.apply = function(resolver) {
	var target = this.target;
	resolver.plugin(this.source, function(request, callback) {
		resolver.doResolve(target, request, null, callback);
	});
};

In any case, as you break things down, the resolving process begins to look less intimidating.

If you take another look at the example sequence above, you may notice the "after-described-resolve" type. This is actually a derived type that doResolve creates. Basically, each base type like "relative", doResolve will generate a corresponding before and after type. So, again, for "relative" there will be a "before-relative" and "after-relative" type which plugins can be assigned to as a source type. For instance:

new NextPlugin("after-relative", "described-relative")

Finally, the last type, the "resolved" type, is handled by one plugin: The ResultPlugin which does nothing more than tell the resolver to run plugins against a "result" type and invoke the callback with the result:

function ResultPlugin(source) {
	this.source = source;
}
module.exports = ResultPlugin;

ResultPlugin.prototype.apply = function(resolver) {
	resolver.plugin(this.source, function(request, callback) {
		var obj = Object.assign({}, request);
		resolver.applyPluginsAsyncSeries1("result", obj, function(err) {
			if(err) return callback(err);
			callback(null, obj);
		});
	});
};

This will ultimately cause the resolver's onResolve callback in resolve() to be invoked that will then give the initial caller the result.

So, yeah, we went down the rabbit hole, but after the journey down it becomes easier to follow along and to debug.

Hopefully this helps provide some useful information for those trying to make sense of the resolver's doResolve method and associated plugin architecture. There are certainly more details to get into, but I think this at least gives a good starting point about what's going on.

[maxkoretskyi]

@mlcohen , great info! Thanks!

[skipjack]

@mlcohen great write up, thanks! Is there any chance you'd be interested in submitting a PR to formalize some of those notes and resolve this issue?

[mlcohen]

@skipjack Yeah, I'd be happy to. I'll fit in time to work on it. It'd be helpful if @TheLarkInn and @sokra could go over what I wrote above just to make sure that it all makes sense 😊

[TheLarkInn]

I love what I see so far. This is really incredible work @mlcohen. Thank you for taking the time to document your experience learning so far. I think there maybe are a couple ways to split this up. I think I'm going to have some folks who have experience with webpack plugins, but not Resolver Plugins, leave some feedback and ask questions to clarify pieces that didn't make sense so that we can really have a progressive learning experience in terms of complexity and architecture of enhanced-resolve.

[mlcohen]

@TheLarkInn Thanks for the feedback. Input from people who have experience with webpack's plugin architecture would be great.

I've actually been digging further into the enhanced-resolve library to get a more complete picture of what's going on. Specifically what each resolve plugin is really doing. Most of the plugins make sense. The only ones that were throwing me for a loop were those related to "concord". The notes in https://github.com/webpack/concord were kind of challenging to make sense of, but after watching a video (https://github.com/webpack/concord) I think I now get the gist.

From what I understand, "concord" is an idea that tries to address the problem of how developers go about putting together their project's webpack config in order to properly build and bundle code with different libraries.

Let's say there is a JS project called webapp that makes use of two node modules libA and libB. Depending on how the libraries are included in the webapp project, a developer may need to update the project's webpack config to assure that those libraries are built and bundled correctly. Meaning that the developer has to manually figure out what webpack loaders and plugins libA and libB need to be processed. That's a pain especially as a project grows and makes use of more libraries made of different types of resources.

With "concord", each library would ideally have their own concord configuration expressed either as a standalone config or placed inside another configuration, like package.json. The resolver detects the concord configuration within libA and libB and collects all the necessary information about how to correctly resolve each library's resources. The concord config also associates each resource (module) with a kind of MIME type-like value that further instructs webpack which loaders and plugins to use to load and process those resources. Webpack could then download the required loaders and plugins from central, remote "concord" repository.

Is what I described above more-or-less capture the essence of what "concord" is supposed to do?

[TheLarkInn]

That is exactly what "concord" is. Important to note that the feature never fully was completed, (despite its plugins still being in enhanced-resolve).

[skipjack]

@mlcohen once #1612 gets merged we should have a pretty stable base for the API section of the site. If you're still interested in contributing, even if it's just to resolve this issue, I would be more than happy to discuss more.

@TheLarkInn you may want to take a look at #1612 as well. It tightens the content up a bit and allows a new group flag in the YAML frontmatter to group pages within the sidebar.

[mlcohen]

@skipjack Awesome. I was actually wondering where to put documentation that explains deep webpack internals for things like the enhanced-resolve library. In the current docs, the only section that seemed to make sense was under /development.

[skipjack]

@mlcohen #1612 is merged so if you wanted to start on a PR to resolve this issue that would be amazing.

I was actually wondering where to put documentation that explains deep webpack internals for things like the enhanced-resolve library. In the current docs, the only section that seemed to make sense was under /development.

Yeah the separation between our current sections should be a bit clearer now. For utilities like enhanced-resolve they could go into the Contribute section I guess but it may make more sense to add another section like Loaders and Plugins where we dynamically pull the READMEs for utilities like enhanced-resolve, webpack-dev-middleware, etc. I think there are a fair amount of packages that don't fit into the "Loader" or "Plugin" category.

Let me know if you have any questions re the site or tackling this issue in particular. I would really love to have someone take the lead on finishing/organizing the API section. But even if you only have time to tackle this issue it'd still be 🎉 🎉 .

[code4cake]

Hey guys, could I take this issue? Is it about writing documentation or?
Thanks.

[skipjack]

@dantesolis thanks for volunteering. You should discuss with @mlcohen as I think they were planning on picking this up. If not, we'd be more than happy if you took a stab at it! @mlcohen wrote a summary of how things work above.

Is it about writing documentation or?

Yep, it would just consist of some updates on one of the pages in the API section of the site.

[code4cake]

@skipjack what is the best way to get in touch with @mlcohen, email or just via here? Thanks.

[mlcohen]

@dantesolis Hi! Apologies for the late response. Yes, if you'd like to take on writing docs for the enhanced-resolved library, please feel free. I unfortunately haven't had a chance to get around to doing it myself :(

[code4cake]

@mlcohen thanks this would be my first contribution, where would be a good place to start? Thanks

[skipjack]

@dantesolis see @mlcohen's detailed comment above and my short comment on #1505. It's really the first and last points in that checklist that relate to this site (and I can help with the first). Those should give you some background and a good starting point. Feel free to ping me if you have any questions (either here or on gitter).

Once you start a PR, I and hopefully 🙏 @mlcohen 😁 , can review and help you push things along.

[skipjack]

See my comment on #1505. The page is rewritten on the next branch and a ticket was created to track this in enhanced-resolve.