-
Notifications
You must be signed in to change notification settings - Fork 12.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support "medium-sized" projects #3469
Comments
Oh my goodness! |
👍 |
Yep! This makes perfect sense for the use-cases you've provided. Providing the tools we use with a better view on how our code is intended to be consumed is definitely the right thing to do. Every time I F12 into a .d.ts file by mistake I feel like strangling a kitten! |
Jonathan, Thank you so much for the kind feedback and for deciding to take this on. TypeScript is such a great tool and this functionality will help many people who want to componentize their medium-sized codebases that can't justify the inefficiency mandated by enormous projects that rely on strict division of concerns (for example the Azure portals or project Monacos of the world with > 100kloc and many independent teams). In other words, this will really help "regular people". Also, others have already proposed stuff for this, for example @NoelAbrahams (#2180), and others so I can't claim originality here. This is just something I've been needing for a while. I think that your proposal is excellent. The only shortcoming that I see versus my proposal (#3394), which I have now closed, is the lack of a fallback mechanism for references. Consider the following real-world scenario which I detailed here: #3394 (comment) I have a TypeScript project grunt-ts that depends on a different project csproj2ts. Hardly anyone who works on grunt-ts will also want to work on csproj2ts as it has a very limited scope of functionality. However, for someone like me - It'd be great to be able to work on both projects simultaneously and do refactoring/go to definition/find all references across them. When I made my proposal, I suggested that the dependencies object be an object literal with named fallbacks. A version more in keeping with your proposal would be: "dependencies": {
"csproj2ts": ["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
"SomeRequiredLibrary": "../SomeRequiredLibraryWithNoFallback"
} To simplify it to still be an array, I suggest the following alternate implementation of a future hypothetical "dependencies": [
["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
"../SomeRequiredLibraryWithNoFallback"
] The resolution rule for each array-type item in This is a very slightly more complicated-to-implement solution, however it gives the developer (and library authors) far greater flexibility. For developers who do not need to develop on csproj2ts (and therefore do not have a In the above example, the Thank you so much for considering this. |
There are two probelms here that we need to break apart, there is build, and there is language service support. For Build, I do not think tsconfig is the right place for this. this is clearly a build system issue. with this proposal the typescript compiler needs to be in the businesses of:
These are all clearly the responsibility of build systems; these are hard problems and there are already tools that do that, e.g. MSBuild, grunt, gulp, etc.. For Language Service, "files" : [
"file1.ts",
{
"path": "../projectB/out/projectB.d.ts",
"sourceProject": "../projectB/"
}
] Where tsc will only look at "path" but tools can look at other information and try to be as helpful as they can. I acknowledge the existence of the problem but do not think lumping build and tooling is the correct solution. tsconfig.json should remain a configuration bag (i.e. a json alternative to response files) an not become a build system. one tsconfig.json represents a single tsc invocation. tsc should remain as a single project compiler. MSBuild projects in VS are an example of using a build system to build IDE features and now ppl are not happy with it because it is too big. |
Thanks for your reply, Mohamed. Let me restate to see if I understand:
Do I have that right? If so, I think this is very fair. In my original proposal (#3394), I did say that my idea was incomplete because it didn't include the step of copying the emitted results from where they'd be output in the referenced library to where the referencing library would expect them at runtime. I think you're saying "why go halfway down a road for building when what is really needed is the language service support". To change the data a bit in your example, would you be willing to support something like this? "files" : [
"file1.ts",
{
"path": "externalLibraries/projectB.d.ts",
"sourceProject": "../projectB/"
}
] The assumption is that the current project would ship with a definition for projectB that would be used by default, but if the actual source for projectB is available, the actual source would be used instead. |
@nycdotnet you summed it up right; i would like to create a system that is loosely coupled and allows mixing and matching diffrent build tools with diffrent IDEs but still get a great design time experience. |
Sounds great! |
I agree with @mhegazy and actually I think it is important for TypeScript to stop thinking of itself as a 'compiler' and start thinking of itself as a 'type-checker' and 'transpiler'. Now there is single-file transpilation support I don't see any reason for compiled JavaScript files to be created except for at runtime/bundling. I also don't see why it is necessary to generate the external reference definitions during type-checking when the actual typescript source is available. File and package resolution is the responsibility of the build system you are using (browserify, systemjs, webpack etc), so for the tooling to work the language service needs to be able to resolve files in the same way as whatever build system/platform you are using. This either means implementing a custom LanguageServicesHost for each build system or providing a tool for each one which generates the correct mapping entries in tsconfig.json. Either of these is acceptable. @nycdotnet I think your use case for multiple fallback paths would be better handled using |
I agree with @mhegazy that we should keep build separate from the compiler/transpiler. I do like to include the tsconfig -> tsconfig dependency in the files section assuming that if the section doesn't list any *.ts files it still scans for them. E.g. "files" : [ Will still include all ts files in the directory and subdirectory containing the tsconfig.json file. |
@dbaeumer you then want to have it in a different property, correct? currently, if files is defined, it is always used and we ignore the include *.ts part. |
@mhegazy not necessarily a different section although it would make things clearer at the end. All I want to avoid is to be forced to list all files if I use a tsconfig -> tsconfig dependencies. In the above example I would still like to not list any *.ts files to feed them into the compiler. |
I think this is much needed. I don't think we can avoid the build question however. That doesn't mean we need to implement a build system along with this proposal, but we should have thought through some good guidance that works in a configuration such as that proposed. It will be non-trivial to get right (and we do need to solve it for Visual Studio). In the above proposal (where both the .d.ts and the source is referenced), does it detect if the .d.ts is out of date (i.e. needs to be rebuilt)? Do operations like Refactor/Rename work across projects (i.e. updates the name in the referenced project source, not just its .d.ts file which will get overwritten next build)? Does GoToDef take me to the original code in the referenced project (not the middle of a giant .d.ts file for the whole project)? These would seem it imply needing to resolve, and in some cases analyze, the source of the referenced projects, in which case is the .d.ts that useful? |
The general solution, which we have today, you have a .d.ts as a build output of one project, and then referenced as input in the other project. This works fine for build, so no need to change that. The problem is the editing scenario. you do not want to go through a generated file while editing. My proposed solution is to provide a "hint" of where the generated .d.ts came from. The language service will then not load the .d.ts, and instead load a "project" from the hint path. this way goto def will take you to the implementation file instead of the .d.ts and similarly errors would work without the need of a compilation. operations like rename, will "propagate" from one project to another, similarly find references, will do. |
Today it is completely up to the host (the IDE, whatever) to find the tsconfig.json file, although TS then provides APIs to read and parse it. How would you envision this working if there were multiple tsconfig.json located in a hierarchical fashion? Would the host still be responsible for resolving the initial file but not the others or would the host be responsible for resolving all of the tsconfigs? Seems like there is a tradeoff there between convenience/convention and flexibility. |
Wouldn't this start with the ability to build d.ts files as described in #2568 (or at least have relative imports)? |
@spion i am not sure i see the dependency here. you can have multiple outputs of a project, it does not have yo be a single delectation file. the build system should be able to know that and wire them as inputs to dependent projects. |
@mhegazy Oops, sorry. Looking at the issue again, it appears that this is more related to the language service. I read the following
and automatically assumed its related to better support for npm/browserify (or webpack) build worfklows where parts of the project are external modules. AFAIK there is no way to generate .d.ts file(s) for external modules yet? If so, the only way a language service could link projects that import external modules would be to have something like this in tsconfig.json : {
"provides": "external-module-name"
} which would inform the LS when the projects is referenced in another |
I do not think this is true. calling |
@spion TypeScript can generate d.ts files for external modules as @mhegazy said, but this results in a 1:1 ratio of definitions to source files which is different from how a library's TypeScript definitions are typically consumed. One way to work around this is this TypeStrong library: https://github.com/TypeStrong/dts-bundle |
@mhegazy sorry, I meant for "ambient external modules" i.e. if I write import {MyClass} from 'external-module-name' there is no way to make @nycdotnet I'm aware of dts-bundle and dts-generator but still if the language service is to know about the source of my other project, it should also know what module name it provides to be able to track the imports correctly |
What's the status of this feature? it seems this is an important option for medium size projects. How do you configure a project that have source in different folders with a specific "requirejs" config? |
Btw there is a downside to this model, that it leads to "phantom dependencies" where a project can import a dependency that was not explicitly declared in its package.json file. When you publish your library, this can cause trouble such as (1) a different version of the dependency getting installed than what was tested/expected, or (2) the dependency missing completely because it was hoisted from an unrelated project that is not installed in this context. PNPM and Rush both have architectural choices intended to protect against phantom dependencies. |
I have a general question about
Normally a system such as Gulp or Webpack manages this pipeline, and the compiler is just one step in the middle of the chain. Sometimes a secondary tool also runs the build in another way, e.g. Jest+ts-jest for Is Or, if the design is to process an entire monorepo in a single pass (whereas today we build each project in a separate NodeJs process), I'm also curious how the other build tasks will participate: For example, will we run webpack on all projects at once? (In the past that led to out-of-memory issues.) Will we lose the ability to exploit multi-process concurrency? These aren't criticisms BTW. I'm just trying to understand the big picture and intended usage. |
@pgonzal right, there are many non-tsc parts to building a real world monorepo. For our lerna monorepo I took the following approach:
|
How do you handle |
Does this have any implications on |
Reported another issue with build mode at #25355. |
Thanks for all the great feedback and investigations so far. I really appreciate everyone who took time to try things out and kick the tires. Great write-up, thanks again for providing this. Your issues in order -
@rosskevin 🎉 for the PR on the
Great question; I want to answer this one very clearly: definitely not. If you're happy today using The broader context is that there was fairly robust internal debate over whether
For single-file scripts, which implicitly cannot have project references, there is zero impact. @EisenbergEffect had some questions in the If we don't think cross-project rename is acceptably stable+complete for 3.0, we'll likely block rename operations only when the renamed symbol is in the .d.ts output file of another project - allowing you to do this would be very confusing because the .d.ts file would get updated on a subsequent build of the upstream project after the upstream project had been modified, which means it could easily be days between when you make a local rename and when you realize that the declaring code hadn't actually been updated. For features like Go to Definition, these are working today in VS Code and will work out-of-the-box in future versions of Visual Studio. These features all require .d.ts.map files to be enabled (turn on Open problems I'm tracking at this point:
Open questions
|
@RyanCavanaugh to add to
We've also mentioned having an incremental output cache, separate from the real project output location, to handle things like updating declarations in the background in the LS (today, changes don't propagate across project boundaries in the editor until you build), |
sorry for a dumb question, since it's checked in the roadmap, how do i get this feature enabled? |
@Aleksey-Bykov you can use it in typescript@next. I just tried this out on our yarn workspace powered monorepo and it works well. One thing I did notice was that |
i have a folder full of *.d.ts and nothing else what am i supposed to do about it:
|
@timfish that feedback matches some other I've heard; logged #25562 @Aleksey-Bykov #3469 (comment) should help explain some concepts |
@RyanCavanaugh it looks like project referecing only works for commonjs and node module resolution, doesn't it? in you example:
i have 2 simple projects |
If you have a sample repo or something I can diagnose why you're getting that error |
@RyanCavanaugh please do |
@RyanCavanaugh, it also looks like |
Thread too long (in both time and space); let's pick up the discussion at lucky issue number 100 * 2^8 : #25600 |
Listening to @nycdotnet has me fired up to tackle this one. Thanks, Steve. (btw, you can check out his good interview here: http://www.dotnetrocks.com/default.aspx?showNum=1149)
The proposal here was first started in times prehistoric (even before #11), when dinosaurs walked the scorched earth. While nothing in this proposal is novel, per se, I believe it's high time we tackled the issue. Steve's own proposal is #3394.
The Problem
Currently, in TypeScript, it's rather easy to start and get going, and we're making it easier with each day (with the help of things like #2338 and the work on System.js). This is wonderful. But there is a bit of a hurdle as project size grows. We currently have a mental model that goes something like this:
For small-sized projects, tsconfig.json gives you an easy-to-setup way of getting going with any of the editors in a cross platform way. For large-scale projects, you will likely end up switching to build systems because of the varied requirements of large-scale projects, and the end result will be something that works for your scenarios but is difficult to tool because it's far too difficult to tool the variety of build systems and options.
Steve, in his interview, points out that this isn't quite the right model of the world, and I tend to agree with him. Instead, there are three sizes of project:
As you scale in size of project, Steve argues, you need to be able to scale through the medium step, or tool support falls off too quickly.
Proposal
To solve this, I propose we support "medium-sized" projects. These projects have standard build steps that could be described in tsconfig.json today, with the exception that the project is built from multiple components. The hypothesis here is that there are quite a number of projects at this level that could be well-served by this support.
Goals
Provide an easy-to-use experience for developers creating "medium-sized" projects for both command-line compilation and when working with these projects in an IDE.
Non-goals
This proposal does not include optional compilation, or any steps outside of what the compiler handles today. This proposal also does not cover bundling or packaging, which will be handled in a separate proposal. In short, as mentioned in the name, this proposal covers only the 'medium-sized' projects and not the needs of those at large scales.
Design
To support medium-sized projects, we focus on the use case of one tsconfig.json referencing another.
Example tsconfig.json of today:
Proposed tsconfig.json 'dependencies' section:
Dependencies point to either:
Dependencies are hierarchical. To edit the full project, you need to open the correct directory that contains the root tsconfig.json. This implies that dependencies can't be cyclic. While it may be possible to handle cyclic dependencies in some cases, other cases, namely those with types that have circular dependencies, it may not be possible to do a full resolution.
How it works
Dependencies are built first, in the order they are listed in the 'dependencies' section. If a dependency fails to build, the compiler will exit with an error and not continue to build the rest of the project.
As each dependency completes, a '.d.ts' file representing the outputs is made available to the current build. Once all dependencies complete, the current project is built.
If the user specifies a subdirectory as a dependency, and also implies its compilation by not providing a 'files' section, the dependency is compiled during dependency compilation and is also removed from the compilation of the current project.
The language service can see into each dependency. Because each dependency will be driven off its own tsconfig.json, this may mean that multiple language service instances would need to be created. The end result would be a coordinated language service that was capable of refactoring, code navigation, find all references, etc across dependencies.
Limitations
Adding a directory as a dependency that has no tsconfig.json is considered an error.
Outputs of dependencies are assumed to be self-contained and separate from the current project. This implies that you can't concatenate the output .js of a dependency with the current project via tsconfig.json. External tools, of course, can provide this functionality.
As mentioned earlier, circular dependencies are considered an error. In the simple case:
A - B
\ C
A is the 'current project' and depends on two dependencies: B and C. If B and C do not themselves have dependencies, this case is trivial. If C depends on B, B is made available to C. This is not considered to be circular. If, however, B depends on A, this is considered circular and would be an error.
If, in the example, B depends on C and C is self-contained, this would not be considered a cycle. In this case, the compilation order would be C, B, A, which follows the logic we have for ///ref.
Optional optimizations/improvements
If a dependency does not to be rebuilt, then its build step is skipped and the '.d.ts' representation from the previous build is reused. This could be extended to handle if the compilation of a dependencies has built dependencies that will show up later in the 'dependencies' list of the current project (as happened in the example given in the Limitations section).
Rather than treating directories passed as dependencies that do not have a tsconfig.json as error cases, we could optionally apply default 'files' and the settings of the current project to that dependency.
The text was updated successfully, but these errors were encountered: