-
Notifications
You must be signed in to change notification settings - Fork 12.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document --incremental and composite project APIs #31849
Document --incremental and composite project APIs #31849
Comments
Thank-you for the new incremental APIs in the upcoming Typescript 3.6! I've read through the code, but cannot yet see how to plug in my own "compiler" to the solution builder so that I can make my own call to emit. My call to emit is needed to pass in transformers to the build. I can see how the "Watch" compiler is used from the builder, but that is not quite what I need. @sheetalkamat If you could direct me down the right path to save me a bit of time I would be appreciative. Thanks in advance! Edit: Using getNextInvalidatedProject() looks promising and/or emitNextAffectedFile... |
@ToddThomson The provision already exists and as you found out, If you need to pass custom transformers etc for emit you need to call |
@sheetalkamat Thank-you. The use of It was my assumption that overriding |
That's because in build is for all the projects and its not clear if the transformers are suppose to be for one project or for all.. |
@sheetalkamat That makes sense. Perhaps transformers should be part of the project configuration file then. Thanks for your help and have a great day! |
@sheetalkamat I'm having trouble getting my custom compiler to actually use the buildInfo files to shorten the builds const host = ts.createIncrementalCompilerHost(allOptions, ts.sys);
const program = ts.createIncrementalProgram({
host,
options: allOptions,
projectReferences,
rootNames: fileNames
});
const { diagnostics: emitDiagnostics } = program.emit(); my options look like { target: 1,
module: 1,
strict: true,
jsx: 1,
allowSyntheticDefaultImports: true,
sourceMap: true,
declaration: true,
lib:
[ 'lib.es2015.d.ts',
'lib.es2017.d.ts',
'lib.dom.d.ts',
'lib.es2019.d.ts' ],
esModuleInterop: true,
experimentalDecorators: true,
resolveJsonModule: true,
downlevelIteration: true,
outDir: 'dist',
rootDir: '/Users/alisowski/Documents/cgds/components/Card/src',
composite: true,
configFilePath:
'/Users/alisowski/Documents/cgds/components/Card/tsconfig.json',
incremental: true,
emitDeclarationOnly: true } any guidance? when I run tsc with the same setup it uses the buildinfo files |
@sheetalkamat I would also need some hint. I tried this: const host = ts.createIncrementalCompilerHost(compilerOptions);
const options: ts.CompilerOptions = {
...compilerOptions,
incremental: true,
tsBuildInfoFile: "tsBuildInfo.json"
};
const program = ts.createIncrementalProgram({ rootNames: filesToCompile, options, host });
let nextAffected;
while ((nextAffected = program.emitNextAffectedFile()) !== undefined) {
const result = nextAffected.result;
console.log(result.emitSkipped); // Always false :(
console.log(result.diagnostics);
}
return [
...program.getOptionsDiagnostics(), ...program.getGlobalDiagnostics(), ...program.getSyntacticDiagnostics(),
...program.getDeclarationDiagnostics(), ...program.getSemanticDiagnostics(),
]; ... and I also get a written |
I am not sure what could be going wrong.. Try debugging and see if tsbuildInfo that is being read. If it is check if version written there is same as ts.version and go from there? |
You are creating host with wrong options so those options are getting used to determine the tsbuild info path or something else in the compiler.
|
Okay, so how can I determine which files have been really skipped in the incremental (follow-up) build? I'm only asking for debugging purposes. What I really want is better cold-start performance 😃. And atm. I don't see a difference; so I assume that the incremental build info isn't used at all (although written). Another observation: In ProcMon, I saw "WriteFile" events for all BTW: Using TS 3.6.1 RC |
You will need to track of what files are printed when doing clean build and from there deduce which are not built, because
You would need to see if buildInfo is being read or not? and if it is then check the version if it is matched. (add host.readFile to override that lets you check if buildinfo is read or not. You would want to ) (i noticed that in earlier response o had commented on your options being not set correctly but its not reflected correctly and went into block scope. I have updated it so make sure you look into that as well. |
I have made an example repo showcasing the issue. If I am reading correctly it seems like we have to manage the "short circuit" ourselves by watching output? I would assume that this is handled https://github.com/hipstersmoothie/typescript-incremental-node-compiler-example |
I added the following to my repo and got some interesting results const host = ts.createIncrementalCompilerHost(allOptions, {
...ts.sys,
readFile: (...args) => {
return ts.sys.readFile(...args);
}
}); It seems to be reading the READ FILE tsconfig.tsbuildinfo
READ FILE /Users/alisowski/Documents/incremental-example/src/index.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es5.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.core.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.collection.d.ts |
Even if i pass an absolute path to my buildinfo file it still reads the other files const allOptions = {
...options,
outDir: 'dist',
incremental: true,
emitDeclarationOnly: true,
listEmittedFiles: true,
tsBuildInfoFile: path.join(__dirname, 'tsconfig.tsbuildinfo')
}; READ FILE /Users/alisowski/Documents/incremental-example/tsconfig.tsbuildinfo
READ FILE /Users/alisowski/Documents/incremental-example/src/index.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es5.d.ts
READ FILE /Users/alisowski/Documents/incremental-example/node_modules/typescript/lib/lib.es2015.core.d.ts |
Also to help me debug: where exactly does this short circuit happen? I'm having trouble finding where in the code it actually cancels the build based on the build info |
I have updated the example with the above findings. Now my example will not rebuild the files (yay!) but my build times are still about the same. Comparing with
|
I also had no luck. I've added a
|
@sheetalkamat Do I need to manually call |
@sheetalkamat any pointers? I feel like i'm almost there but now my wheels are spinning |
I just upgraded my example repo to 3.6.2. My compile times now look like: To check that the builds are fast run
yarn build - (get initial emit) ~1.65s
yarn build - (second emit should be much shorter) ~1.65s
Compare this with tsc:
yarn build:tsc - (first time takes ~1.55s)
yarn build:tsc - (second time takes 0.27s) Is there a working example anywhere (other than tsc) where i can see incremental builds via the node API? |
@hipstersmoothie I am looking into your repro and seeing if there is bug. |
@hipstersmoothie it seems like you are patching |
@sheetalkamat I had also backslashes in my Could you maybe provide a minimal sample with a simple (single-project) incremental build using |
const diagnostics = [
...program.getConfigFileParsingDiagnostics(),
...program.getSyntacticDiagnostics(),
...program.getOptionsDiagnostics(),
...program.getOptionsDiagnostics(),
...program.getSemanticDiagnostics()
];
const result = program.emit();
const allDiagnostics = diagnostics.concat(result.diagnostics); Note that you would need build with #33170 for 3.6 to be able to actually do incremental semantic diagnostics and its not yet in build from that branch. You can in the mean while test with typescript@next |
2x |
Okay now I'm actually seeing a difference in compile times! yarn build - (get initial emit) ~2.3s
yarn build - (second emit should be much shorter) ~1.12s
Compare this with tsc:
yarn build:tsc - (first time takes ~1.80s)
yarn build:tsc - (second time takes 0.35s) However |
you aren't comparing correct times.. |
Oh interesting. Is there a way to use the |
@sheetalkamat Many thanks for the sample in the Wiki! I have got it now too. My problem was that I created the I have now Note that So @sheetalkamat, is this a bug (using |
@ulrichb Not sure if its configuration issue. Please provide complete steps to repro, probably a repo where this repros and we can take a look. |
@sheetalkamat Is there an API to replicate |
I see the new docs and it's working! (I think) One last question is: how to feed custom transformers to the solution builder? |
You can pass that to |
Perfect! Thanks for all the help |
I can't seem to get only emitting .d.ts files to work though. It goes into an infinite loop const host = ts.createSolutionBuilderHost(
undefined,
undefined,
reportDiagnostic,
reportSolutionBuilderStatus,
reportErrorSummary
);
const solution = ts.createSolutionBuilder(host, ['./tsconfig.json'], {
verbose: true,
emitDeclarationOnly: true
});
while (true) {
const project = solution.getNextInvalidatedProject();
if (!project) {
break;
}
project.emit(undefined, undefined, undefined, true);
} |
@sheetalkamat While trying to create a repro sample, I mentioned that it also works using just const compilerOptions = {
// ...
outDir: outDirAbsolute.replace(/\\/g, "/"),
incremental: true,
tsBuildInfoFile: path.resolve(outDirAbsolute, "tsconfig.tsbuildinfo").replace(/\\/g, "/"),
}; So, I'm happy now. Many thanks again! BTW: Regarding cross platform compat, the "path normalization requirement" is a bug waiting to happen when starting with Linux/macOS and later running on Windows. |
@sheetalkamat seems like on this line https://github.com/microsoft/TypeScript/blob/master/src/compiler/tsbuild.ts#L433 it filters out any extra CompilerOptions (in my case EDIT: I can also do this, but it feels very, very dirty. ts.commonOptionsWithBuild.push({ name: 'emitDeclarationOnly' })
const solution = ts.createSolutionBuilder(host, ['./tsconfig.json'], {
verbose: true,
incremental: true,
listEmittedFiles: true,
emitDeclarationOnly: true
}); |
That is intended behavior, BuildOptions are different from CompilerOptions and only subset of those are allowed. |
So there is no way to pass compiler options and no future support? |
Is it intended that |
It seems i could use EDIT: Fixed that. The .d.ts ast has a reference to the source Now i'm just trying to get my custom transformer to abort the emit |
I have the same issue. Have you managed to make it work ? |
I've had some success following all your indications here, and the new documentation (thank you!) However, I find that incremental compilation does not work when using the Here is an example of a {
"bundle": {
"commonSourceDirectory": "../../src",
"sourceFiles": [
"../../src/main.ts"
],
"js": {
"sections": [
{
"pos": 0,
"end": 13,
"kind": "prologue",
"data": "use strict"
},
{
"pos": 14,
"end": 44,
"kind": "text"
}
],
"sources": {
"prologues": [
{
"file": 0,
"text": "",
"directives": [
{
"pos": -1,
"end": -1,
"expression": {
"pos": -1,
"end": -1,
"text": "use strict"
}
}
]
}
]
}
}
},
"version": "3.8.2"
} When removing the |
How does one implement |
Here's how I ended up doing it: https://github.com/intuit/design-systems-cli/blob/master/plugins/build/src/typescript.ts#L173 |
@hipstersmoothie that's pretty similar to what I was doing out of spite... Figured there should be a better way; but it juts looks like the necessary invalidation logic is presently burried. |
Hi. I am building a jest transformer that use solution builder. And I need some help on how can I get the emitted file for the jest input sourcePath. 🙏 import type { Transformer } from "@jest/transform";
import * as ts from "typescript";
const transformer: Transformer = {
process: (sourceText, sourcePath, config, options) => {
console.log({ sourcePath, rootDir: config.rootDir });
const host = ts.createSolutionBuilderHost({
...ts.sys,
readFile(path: string, encoding?: string) {
console.log("readFile", { path, encoding });
return ts.sys.readFile(path, encoding);
},
});
const solutionBuilder = ts.createSolutionBuilder(
host,
[config.rootDir],
{}
);
while (true) {
const next = solutionBuilder.getNextInvalidatedProject();
if (next === undefined) {
break;
}
switch (next.kind) {
case ts.InvalidatedProjectKind.Build:
next.done();
break;
case ts.InvalidatedProjectKind.UpdateBundle:
next.done();
break;
case ts.InvalidatedProjectKind.UpdateOutputFileStamps:
next.done();
break;
}
}
// TODO: Get the emitted files for the sourcePath
return {
code: "",
map: "",
};
},
};
export const process = transformer.process; |
Follow-up from #29978, suggested by @MLoughry at #29978 (comment)
The text was updated successfully, but these errors were encountered: