Path mappings based module resolution

[vladima]

Proposed module resolution strategy

UPDATE: proposal below is updated based on the results of the design meeting.
Initial version can be found here.

Primary differences:

• instead of having baseUrl as a separate module resolution strategy we are introducing a set of properties
  that will allow to customize resoluton process in existing resolution strategies but base strategy still is used as a fallback.
• rootDirs are decoupled from the baseUrl and can be used without it.

Currently TypeScript supports two ways of resolving module names: classic (module name always resolves to a file, module are searched using a folder walk)
and node (uses rules similar to node module loader, was introduced in TypeScript 1.6).
These approaches worked reasonably well but they were not able to model baseUrl based mechanics used by
RequireJS or SystemJS.

We could introduce third type of module resolution that will fill this gap but this will mean that once user has started to use this new type then support to
discover typings embedded in node modules (and distributed via npm) is lost. Effectively user that wanted both to use baseUrl to refer to modules defined inside the project
and rely on npm to obtain modules with typings will have to choose what part of the system will be broken.

Instead of doing this we'll allow to declare a set of properties that will augment existing module resolution strategies. These properties are:
baseUrl, paths and rootDirs (paths can only be used if baseUrl is set). If at least one of these properties is defined then compiler will try to
use it to resolve module name and if it fail - will fallback to a default behavior for a current resolution strategy.

Also choice of resolution strategy determines what does it mean to load a module from a given path. To be more concrete given some module name /a/b/c:

• classic resolver will check for the presense of files /a/b/c.ts, /a/b/c.tsx and /a/b/c.d.ts.
• node resolver will first try to load module as file by probing the same files as classic and then try to load module from directory
  (check /a/b/c/index with supported extension, then peek into package.json etc. More details can be found in this issue)

Properties

• baseUrl
• path mappings
• rootDirs

BaseUrl

All non-rooted paths are computed relative to baseUrl.
Value of baseUrl is determined as either:

• value of baseUrl command line argument (if given path is relative it is computed based on current directory)
• value of baseUrl propery in 'tsconfig.json' (if given path is relative it is computed based on then location of 'tsconfig.json')

Path mappings

Sometimes modules are not directly located under baseUrl. It is possible to control how locations are computed in such cases
using path mappings. Path mappings are specified using the following JSON structure:

{
    "paths": {
        "pattern-1": ["list of substitutions"],
        "pattern-2": ["list of substitutions"],
        ...
        "pattern-N": ["list of substitutions"]
    }
}

Patterns and substitutions are strings that can have zero or one asteriks ('*').
Interpretation of both patterns and substitutions will be described in Resolution process section.

Resolution process

Non-relative module names are resolved slightly differently comparing
to relative (start with "./" or "../") and rooted module names (start with "/", drive name or schema).

Resolution of non-relative module names (mostly matches SystemJS)

// mimics path mappings in SystemJS
// NOTE: moduleExists checks if file with any supported extension exists on disk
function resolveNonRelativeModuleName(moduleName: string): string {
    // check if module name should be used as-is or it should be mapped to different value
    let longestMatchedPrefixLength = 0;
    let matchedPattern: string;
    let matchedWildcard: string;

    for (let pattern in config.paths) {
        assert(pattern.countOf('*') <= 1);
        let indexOfWildcard = pattern.indexOf('*'); 
        if (indexOfWildcard !== -1) {
            // if pattern contains asterisk then asterisk acts as a capture group with a greedy matching
            // i.e. for the string 'abbb' pattern 'a*b' will get 'bb' as '*'

            // check if module name starts with prefix, ends with suffix and these two don't overlap
            let prefix = pattern.substr(0, indexOfWildcard);
            let suffix = pattern.substr(indexOfWildcard + 1);
            if (moduleName.length >= prefix.length + suffix.length && 
                moduleName.startsWith(prefix) &&
                moduleName.endsWith(suffix)) {

                // use length of matched prefix as betterness criteria
                if (longestMatchedPrefixLength < prefix.length) {
                    // save length of the prefix
                    longestMatchedPrefixLength = prefix.length;
                    // save matched pattern
                    matchedPattern = pattern;
                    // save matched wildcard content 
                    matchedWildcard = moduleName.substr(prefix.length, moduleName.length - suffix.length);
                }
            }
        }
        else {
            // pattern does not contain asterisk - module name should exactly match pattern to succeed
            if (pattern === moduleName) {
                // save pattern
                matchedPattern = pattern;
                // drop saved wildcard match 
                matchedWildcard = undefined;
                // exact match is found - can exit early 
                break;
            }
        }
    }

    if (!matchedPattern) {
        // no pattern was matched so module name can be used as-is
        let path = combine(baseUrl, moduleName);
        return moduleExists(path) ? path : undefined;
    }

    // some pattern was matched - module name needs to be substituted
    let substitutions = config.paths[matchedPattern].asArray();
    for (let subst of substitutions) {
        assert(substs.countOf('*') <= 1);
        // replace * in substitution with matched wildcard
        let path = matchedWildcard ? subst.replace("*", matchedWildcard) : subst;
        // if substituion is a relative path - combine it with baseUrl
        path = isRelative(path) ? combine(baseUrl, path) : path;
        if (moduleExists(path)) {
            return path;
        }
    }

    return undefined;   
}

Resolution of relative module names

Default resolution logic (matches SystemJS)

Relative module names are computed treating location of source file that contains the import as base folder.
Path mappings are not applied.

function resolveRelativeModuleName(moduleName: string, containingFile: string): string {
    let path = combine(getDirectoryName(containingFile), moduleName);
    return moduleExists(path) ? path : undefined;
}

Using rootDirs

'rootDirs' allows the project to be spreaded across multiple locations and resolve modules with relative names as if multiple project roots were merged together in one folder. For example project contains source files that are located in different directories on then file system (not under the same root) but user still still prefers to use relative module names because in runtime such names can be successfully resolved due to bundling.

For example consider this project structure:

 shared
 └── projects
     └── project
         └── src
             ├── viewManager.ts (imports './views/view1')
             └── views
                 └── view2.ts (imports './view1')
 userFiles
 └── project
     └── src
         └── views
             └── view1.ts (imports './view2')

Logically files in userFiles/project and shared/projects/project belong to the same project and
after build they indeed will be bundled together.

In order to support this we'll add configuration property "rootDirs":

{
    "rootDirs": [
        "rootDir-1/",
        "rootDir-2/",
        ...
        "rootDir-n/"
    ]
}

This property stores list of base folders, every folder name can be either absolute or relative.
Elements in rootDirs that represent non-absolute paths will be converted to absolute using location of tsconfig.json as a base folder - this is the common approach for all paths defined in tsconfig.json

///Algorithm for resolving relative module name
function resolveRelativeModuleName(moduleName: string, containingFile: string): string {
    // convert relative module name to absolute using location of containing file
    // this step is exactly the same as when doing resolution without path mapping
    let path = combine(getDirectoryName(containingFile), moduleName);

    // convert absolute module name to non-relative
    // try to find element in 'rootDirs' that is the longest prefix for "path' and return path.substr(prefix.length) as non-relative name
    let { matchingRootDir, nonRelativeName } = tryFindLongestPrefixAndReturnSuffix(rootDirs, path);
    if (!matchingRootDir) {
        // cannot extract non relative name
        return undefined;
    }
    // first try to load module from initial location
    if (moduleExists(path)) {
        return path;
    }
    // then try other entries in rootDirs
    for (const rootDir of rootDirs) {
        if (rootDir === matchingRootDir) {
            continue;
        }
        const candidate = combine(rootDir, nonRelativeName);
        if (moduleExists(candidate)) {
            return candidate;
        }
    }
    // failure case
    return undefined;
}

Configuration for the example above:

{
    "rootDirs": [
        "userFiles/project/",
        "/shared/projects/project/"
    ]
}

Example 1

projectRoot
├── folder1
│   └── file1.ts (imports 'folder2/file2')
├── folder2
│   ├── file2.ts (imports './file3')
│   └── file3.ts
└── tsconfig.json

// configuration in tsconfig.json
{
    "baseUrl": "."
}

• import 'folder2/file2'
  1. baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
     the location of tsconfig.json -> projectRoot
  2. path mappings are not available -> path = moduleName
  3. resolved module file name = combine(baseUrl, path) -> projectRoot/folder2/file2.ts
• import './file3'
  1. moduleName is relative and rootDirs are not specified in configuration - compute module name
     relative to the location of containing file: resolved module file name = projectRoot/folder2/file3.ts

Example 2

projectRoot
├── folder1
│   ├── file1.ts (imports 'folder1/file2' and 'folder2/file3')
│   └── file2.ts
├── generated
│   ├── folder1
│   └── folder2
│       └── file3.ts
└── tsconfig.json

// configuration in tsconfig.json
{
    "baseUrl": ".",
    "paths": {
    "*": [
            "*",
            "generated/*" 
        ]
    }
}

• import 'folder1/file2'
  1. baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
     the location of tsconfig.json -> projectRoot
  2. configuration contains path mappings.
  3. pattern '*' is matched and wildcard captures the whole module name
  4. try first substitution in the list: '*' -> folder1/file2
  5. result of substitution is relative name - combine it with baseUrl -> projectRoot/folder1/file2.ts.
     This file exists.
• import 'folder2/file2'
  1. baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
     the location of tsconfig.json and will be folder that contains tsconfig.json
  2. configuration contains path mappings.
  3. pattern '*' is matched and wildcard captures the whole module name
  4. try first substitution in the list: '*' -> folder2/file3
  5. result of substitution is relative name - combine it with baseUrl -> projectRoot/folder2/file3.ts.
     File does not exists, move to the second substitution
  6. second substitution 'generated/*' -> generated/folder2/file3
  7. result of substitution is relative name - combine it with baseUrl -> projectRoot/generated/folder2/file3.ts.
     File exists

Example 3

rootDir
├── folder1
│   └── file1.ts (imports './file2')
├── generated
│   ├── folder1
│   │   ├── file2.ts
│   │   └── file3.ts (imports '../folder1/file1')
│   └── folder2
└── tsconfig.json

// configuration in tsconfig.json
{
    "rootDirs": [
        "./",
        "./generated/" 
    ],
}

All non-rooted entries in rootDirs are expanded using location of tsconfig.json as base location so after expansion rootDirs will
look like this:

    "rootDirs": [
        "rootDir/",
        "rootDir/generated/" 
    ],

• import './file2'
  1. name is relative, first make it absolute using location of containing file as base location - rootDir/folder1/file2
  2. for this string find the longest prefix in rootDirs - rootDir/ and for this prefix compute as suffix - folder1/file2
  3. since matching entry in rootDirs was found try to resolve module using rootDir - first check if rootDir/folder1/file2
     can be resolved as module - such module does not exist
  4. try remaining entries in rootDirs - check if module rootDir/generated/folder1/file2 exists - yes.
• import '../folder1/file1'
  1. name is relative, first make it absolute using location of containing file as base location - rootDir/generated/folder1/file1
  2. for this string find the longest prefix in rootDirs - rootDir/generated and for this prefix compute as suffix - folder1/file1
  3. since matching entry in rootDirs was found try to resolve module using rootDir - first check if rootDir/generated/folder1/file1
     can be resolved as module - such module does not exist
  4. try remaining entries in rootDirs - check if module rootDir/folder1/file1 exists - yes.

[mmv]

The proposal seems sound 👍

I think the references to data should not be considered for now: their added value is marginal but the added configuration complexity would not be (eg: either refs are mandatory or there would have to be a good way to distinguish them from directory names). They seem like a nice improvement to consider after this is implemented and usage data about the mappings becomes available.

[kitsonk]

The biggest concern with this is that it sounds like a lot of complexity. I can understand the desire to align to SystemJS in hopes that it aligns to whatwg and eventually the loaders that are implemented in the runtime environments. Maybe it is the me being "myopic" but having lived with AMD for nearly a decade now, there are few use cases where my loader configuration is overly complex. For the most part "it just works". Up to this point with TypeScript, largely as well "it just worked", now I fear that I will have to spend a day trying to get TypeScript to understand where my modules are.

To be more specific, relative module resolution... Is there not a 95% happy path for relative module resolution? The AMD spec specifies:

• Relative identifiers are resolved relative to the identifier of the module in which "require" is written and called.

That is about as straight forward of a resolution logic as you can get.

And when that use case does not work, there is baseUrl, path, packages and the sledge hammer map. Each of those are simple and straight forward, without a significant amount of options. All the tools are there and the thing is 95% of the time, there is minimal configuration to get going, often with a single built layer for production, no configuration.

I am likely tilting at windmills, but sometimes it does seem like we are creating 32 varieties of Colgate here...

[vladima]

we have a request for this specific scenario, but I do agree that in majority of cases it is not necessary. That is why it should be opt-in thing: if rootDirs are not specified then all relative module names are resolved using location of containing module which is the behavior most of people would expect. I've tried to emphasize the intent by increasing complexity in examples :

• simplest scenario - possible - baseUrl is enough (and sometimes it can be inferred)
• more complicated case that needs complicated path manipulations (but no specific behavior for relative names ) - possible - path mappings and baseUrl should be enough
• specific case than requires applying path mappings to relative names as well - possible - use path mappings, baseUrl and rootDirs.

[kitsonk]

I guess I am just missing how the last two points can't be addressed by a simple paths that only takes a path, either or absolute to the baseUrl, and then some recursion that keeps relative MIDs relative to the importer. Even with the other scenarios, I suspect you would also still need some sort of sledgehammer map. The specific use case I run into now that makes this challenging (and therefore map would work) is when I am using MIDs that contain AMD plugins. Right now I have to create an ambient module that imports and then exports what I am trying to do.

[mprobst]

@kitsonk I have the use case @vladima is talking about. For me, just a set of --includeDirs would be sufficient (but I do need the canonicalization-of-relative-paths!). Once you add that, you might as well try and be compatible with what SystemJS does, I don't think it substantially increases complexity over that.

[bryanforbes]

@mprobst Could you provide a concrete example of this need and how you would solve it with SystemJS? I'm having a hard time coming up with a real-world example of merging two code trees together that couldn't be solved by creating a package for one tree and importing it in the other.

[mprobst]

@bryanforbes I think @vladima's example is good. Imagine you have something that generates TypeScript source. E.g. Angular 2 will have a build tool that transforms templates to TypeScript code that you can directly call in your app, for various reasons (mostly app startup time).

So you have your Angular controllers together with your templates, src/some/widget/foo_controller.ts and src/some/widget/foo_template.html. But you don't want to generate files in your regular source folder, as that creates a mess with version control, so you follow best practice and have a src folder and a build folder. The Angular template compiler generates build/some/widget/foo_template.ts. In your foo_controller.ts, you import FooTemplate from './foo_template';.

This works if you pass as rootDirs src and build, as ./foo_template would get canonicalized to some/widget/foo_template, then looked up in src and build in order, and found in build/some/widget/foo_template.ts. In SystemJS, I believe you would have a mapping of src/* and build/*.

[bryanforbes]

@mprobst Thanks for clarifying! I didn't understand the generated directory containing TypeScript files, but your explanation helps. What still confuses me is why merging two trees together necessitates another configuration flag with potentially duplicated settings (rootDirs) instead of just using path mapping to solve everything.

[vladima]

My concern about using path mapping for both canonicalization of relative module names and remapping non-relative module names is that this configuration settings become overloaded:

• it becomes more difficult for the end-user in understanding of the concept since it is no longer just path mapping
• it may lead to interesting bugs when some value of path mapping that was not intended to be used for canonicalization will be picked for this purpose.

To deal with duplication I'd rather prefer to have something for data sharing (i.e. references) instead of re-purposing field whose meaning is a already well-defined

[rolandjitsu]

👍

[alexdresko]

Can this go in 1.7? Pretty please with a cherry on top? And is there some way I can play with this now?

[bryanforbes]

@vladima Sorry for the delay in replying. I think my concern now is that you're giving new names to already defined concepts. As I see it (and aside from the obvious difference that these property names take arrays), paths is basically SystemJS's map, and rootDirs is basically like SystemJS's paths. Why not just use those names instead? The ideas are similar to what we already know from AMD and SystemJS, so why invent new names?

[mprobst]

@bryanforbes I'm not sure how well these map with SystemJS, but for what it's worth, rootDirs is a much more specific and understandable name than just paths. I'd take rootDirs over paths any time.

[opichals]

It looks like a lot of projects in the wild have already been using babel for ES6 in combination with webpack as a pretty standard configuration. It might be worth looking at how the module path resolution works in webpack.

There is no need to introduce other concepts/naming, it could be taken from https://webpack.github.io/docs/resolving.html and its configuration https://webpack.github.io/docs/configuration.html#resolve.

[mprobst]

@opichals I might be missing something, but by my reading webpack's resolve does not meet the requirements above for resolving files relative to a set of directories.

[opichals]

@mprobst The resolve.root configuration variable makes it possible for a module to be searched for in folders similarly to the rootDirs from the proposal.

The webpack's resolve.root can be an array of absolute paths (could not directly confirm this just by reading the docs, so I checked the sources.

[mprobst]

@opichals yes, but relative imports are not canonicalized against the list of roots, are they? I read the docs as saying if I load ./foo from a file physically located at /bar/baz, I'll always end up at /bar/foo instead of searching my rootDirs.

[opichals]

@mprobst True. I would find that extremely confusing if any loader attempted to resolve relative path against anything else than just the folder it is physically located at (something require.js tries to do and became a nuisance to configure because of). As stated above such resolution logic seems to add complexity which reflects in complicated use and debugging.

[mprobst]

I think we collected a couple of compelling use cases above.

Standa Opichal [email protected] schrieb am Mi., 21. Okt. 2015,
16:15:

@mprobst https://github.com/mprobst True. I would find that extremely
confusing if any loader attempted to resolve relative path against anything
else than just the folder it is physically located at (something require.js
tries to do and became a nuisance to configure because of). As stated above
such resolution logic seems to add complexity which reflects in complicated
use and debugging.

—
Reply to this email directly or view it on GitHub
#5039 (comment)
.

[csnover]

I would find that extremely confusing if any loader attempted to resolve relative path against anything else than just the folder it is physically located at (something require.js tries to do and became a nuisance to configure because of).

When does RequireJS do this?

[opichals]

You can achieve this with RequireJS because it applies the path mappings to any path segment of any require no matter whether the require module path is absolute or not. e.g. for { path: { 'pkg': '../folder1' } }:

• require('pkg/file1') resolves to ../folder1/file1,
• require('./local/folder2/pkg/folder1') resolves to ./local/folder2/../folder1/file1.

Also it when two require() calls resolve to a single file using different arguments the module gets loaded twice.

For the proposal I would rather like to see an API-based extensibility approach to support the likes of 'Example 3' (let the user replace the resolution function somehow through a configuration or a plugin).

[bezreyhan]

Is there a way to recreate this directory structure using TS: https://gist.github.com/ryanflorence/daafb1e3cb8ad740b346

In short:
Webpack allows you to do this:

  resolve: {
    extensions: ['', '.js', '.json', '.ts', '.tsx'],
    modulesDirectories: ['shared', 'node_modules']
  },

Which means that Webpack will recursively look up the directory tree for the shared dir (the same way it does with the node_modules dir).

This allows us to import shared components even from deeply nested directories without referencing the relative path:

import Button from 'Button';

This will work as long as Button lives in any shared folder that is higher up in the directory tree.

Is there a way to tell tsc to look for modules in this fashion?

[mattyork]

Is it intentional that the "paths" values are computed relative to "baseUrl" when baseUrl is set?

[mhegazy]

Is it intentional that the "paths" values are computed relative to "baseUrl" when baseUrl is set?

yes. this is the same behavior other module loaders like require follow as well.

[asfernandes]

Should outDir prevent some usage of paths?

This was working:

{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": false,
        "rootDir": ".",
        "baseUrl": ".",
        "paths": {
            "util/*": [
                "../../../../Util/src/main/ts/*"
            ]
        }
    }
}

Then when I add outDir:

{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": false,
        "rootDir": ".",
        "baseUrl": ".",
        "outDir": "../../target/ts",
        "paths": {
            "util/*": [
                "../../../../Util/src/main/ts/*"
            ]
        }
    }
}

The compiler says: "error TS6059: File 'C:/tmp/ts/paths/Util/src/main/ts/Util.ts' is not under 'rootDir' 'C:/tmp/ts/paths/SisModules/src/main/ts'. 'rootDir' is expected to contain all source files."

[mhegazy]

if you are using outDir, the compiler needs to mirror the input folder structure to the output directory. the root of the input is defined by rootDir. if there are files in the input, regardless if you are using paths or not, are not under rootDir, the compiler does not know how to emit them, and thus the error.

[asfernandes]

@mhegazy but "the compiler needs to mirror the input folder structure to the output directory" unfortunately creates the original paths in module names instead of the ones mapped to the virtual paths. It's weird at least.

[mhegazy]

unfortunately creates the original paths in module names instead of the ones mapped to the virtual paths. It's weird at least.

that is how it is meant to work. the compiler needs the paths to find the declaration of your module. module names are resource identifiers and should be emitted as is and not altered. please see #5039 (comment).

[asfernandes]

I'm not experienced in TypeScript nor modules, but I think the vpaths names would also be good for resource identifiers. In the browser they make much more sense than the real paths.

[mhegazy]

This is not meant to be a substitute of your require.config files, or system.config. this is meant to be a mirror of them to tell the compiler what your loader already knows.

[sapjax]

@mhegazy Why not add a compile option, so people can decide whether to rewrite module names or not?
I'm using react-native with Typescript, it's hard to resolve this problem.

[rochapablo]

+1

@sapjax, did you find a solution?

This is how I make it work (for now)

[sapjax]

@rochapablo

I'm using https://github.com/ds300/react-native-typescript-transformer now,
Use absolute paths is works for me, it works with tsconfig paths very well.

And it gives another advantage, we no need to use tsc -watch to compile our code before react-native package.

[rochapablo]

Damn! it worked!

Thank you @sapjax!

[rochapablo]

@fforres, follow the instructions it will work.