Support Typescript incremental build

[glebmachine]

🚀 Feature request

- [ ] new
- [x] build
- [x] serve
- [x] test
- [x] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Description

Support of Typescript incremental build https://devblogs.microsoft.com/typescript/announcing-typescript-3-4-rc/

[jimbuck]

3.4 is officially out, which is great news! Incremental builds will hopefully save me 10+ minutes each build (I have a set of projects that take 16mins to build all of them).

Out of curiosity may I ask what is currently causing this to be marked as "blocked"?

[clydin]

Angular itself needs to support a TS version before it can be integrated into the tooling.

[RicardoVaranda]

@alan-agius4 @filipesilva Hi Guys! Congrats on the new release! Just wondering if there has been any development with this improvement since TS 3.4 is now supported.

[MartinJohns]

since TS 3.4 is now supported.

@RicardoVaranda Angular 8.0.0 supports TypeScript 3.3, but not 3.4 (which introduced the incremental flag).

https://github.com/angular/angular/blob/master/CHANGELOG.md

add support for TypeScript 3.3

edit: Apparently you can't trust the Angular changelog. Angular 8.0.0 comes with TypeScript 3.4, not with 3.3.

[tomardern]

@MartinJohns Interestingly, 3.4 looks like it was merged into angular-8-beta-12: ef85336

[RicardoVaranda]

Can confirm this, upon doing ng update it actually installed 3.4.2

[MarcosMeli]

Is important to note that 3.4 introduce a regression that make type checking in compilation pain slow in some scenarios.
Now 3.5 is out with this regression fixed: https://devblogs.microsoft.com/typescript/announcing-typescript-3-5/

Speed improvements
TypeScript 3.5 introduces several optimizations around type-checking and incremental builds.
Type-checking speed-ups
Much of the expressivity of our type system comes with a cost – any more work that we expect the compiler to do translates to longer compile times. Unfortunately, as part of a bug fix in TypeScript 3.4 we accidentally introduced a regression that could lead to an explosion in how much work the type-checker did, and in turn, type-checking time. The most-impacted set of users were those using the styled-components library. This regression was serious not just because it led to much higher build times for TypeScript code, but because editor operations for both TypeScript and JavaScript users became unbearably slow.

[filipesilva]

Angular version 8 supports (and requires) TypeScript 3.4.x. I haven't look at this again, but at the time the only problem I saw was #29872.

We have to try and figure out what else (if anything) is preventing the incremental cache from being used and written.

I'm not sure it can be used in the AOT compiler though, so keep that in mind. Maybe it works, maybe it doesn't.

If anyone is interested in having a look at this, here's a good way to get started:

git clone https://github.com/angular/angular-cli
cd angular-cli
yarn

Then open packages/angular_devkit/build_angular/test/browser/works_spec_large.ts and replace it( with fit(. Also remove the afterEach. Now whenever you run yarn test-large, only this test runs. The results for this test will be in a temporary folder in tests/angular_devkit/build_angular.

At this point you'd just have to hunker down and go figure out what the logic for incremental builds in TS is trying to do, and how to work with it. There's not a lot of docs around it. When I was looking into this, I just added a lot of logs to the TS source in node_modules/typescript.

We use a in-memory file system for TS in packages/ngtools/webpack. So even if the cache file is being written, if TS is writing it then it will stay in memory. It would have to be written there somewhere. Check the packages/ngtools/webpack/src/compiler_host.ts for the FS abstraction used by TS. Maybe log some stuff in the writeFile method.

[FDIM]

From what I understand for incremental build to work and actually save time, files have to be emitted to outDir. If the incremental flag is true, there will be an extra tsconfig.tsbuildinfo file that holds hashes of involved files.

When we build with the CLI, outDir does not seem to be used. @filipesilva is this part also using in-memory file system during build? If it is virtualized, do you know if differential build reuses the same virtual FS or a new one is created?

I am quite curious about this, for my little chrome extension (for e2e tests), if I run tsc command and pass project tsconfig, first build is ~15s, 2nd ~5s (at work we might win minutes of build time)

EDIT: Looks like something in public typescript API is blocking effective use of incremental build programmatically. Either way ts-loader needs to support this first: TypeStrong/ts-loader#913 .

EDIT2: at work with angular 8 build alone currently takes 10 minutes (~5min for es2105 and ~5min for es5)

[filipesilva]

@FDIM I think realistically what would happen is that we'd have to allow that file to be loaded between builds.

Maybe to make sure it's loaded it is enough to give it a custom path instead of the outDir default. But at the end of each build it would still need to be persisted to disk with custom logic.

I am not sure how much this would help for differential loading as each build does use different ES targets. At that point it depends on how much TS itself caches for those cases.

We don't use TS loader, mind you. We have our own loader tailored to the Angular AOT compiler. So I don't think ts-loader supporting it blocks us.

[filipesilva]

For reference, the incremental API issue on TS is microsoft/TypeScript#29978.

[FDIM]

My bad regarding ts-loader, I should have realized that when tinkering with the loader in CLI.

I am fairly sure that writing tsconfig.tsbuildinfo file to disk would not be enough, it also needs already built files from last time to avoid compilation. Anyway, I did manage to output build files to disk after running ng build, but tsbuildinfo was not part of the output - not sure why.

EDIT: I've been playing around quite a bit more with this and it seem the issue linked above needs to be handled first.
When compiler plugin invokes ts program to emit files, typescript does not even try to emit build info as it does not have any (see emitBuildInfo in emitSourceFileOrBundle in typescript.js file, program is undefined). If I do this change - empty build info gets written to src/tsconfig.app.tsbuildinfo file.

Anyway, in order to speed up initial build and make use of incremental flag, ng build would have to write compiled typescript files and buildinfo file somewhere outside, so I guess there will be a need for a cache option that would enable extra emits.

The other part that I haven't figured out is how will typescript will know that info file exists, it did not try to access (checking for existence or reading) it via host methods.

[kukjevov]

Typescript already finished their implementation, so it should be possible to try it here :)

[pppdns]

Seems like there are no more roadblocks for Angular to support incremental builds.

Some related changes that happened since then:

Provide --build/--incremental APIs for build tools:

microsoft/TypeScript#29978

Api for tsc --build and --incremental:

microsoft/TypeScript#31432

Docs are in progress:

microsoft/TypeScript#31849

TS 3.5 Release notes about speed improvements and fixed regression from 3.4:
https://devblogs.microsoft.com/typescript/announcing-typescript-3-5-rc/#speed-improvements

[wottpal]

Seems like there are no more roadblocks for Angular to support incremental builds.

@pppdns Does this mean I just have to set "compilerOptions": { "incremental": true } in my tsconfig.json and it works or is there still something to do on the Angular-side?

[snsakib]

@wottpal
hope it's simple as that

[FDIM]

Unfortunately to improve cold start/initial compile, additional work is needed.

Api wise, new cache dir option should be enough, but it still need to be implemented.

[jukibom]

@FDIM I may be reading this wrong, but doesn't this permit arbitrary compiler options?

If so, doesn't tsBuildInfoFile provide the ability to specify the location of build cache?

[kukjevov]

Anything new here ? Typescript 3.5 is now supported by Angular

[moohkooh]

would be nice to have support of incremental build for ngc

[kukjevov]

Also with incremental build it would be not "nice" but "awesome" to have support of composite projects. I think it would heavily help large projects to split their projects into smaller subprojects. As far as i know typescript created API for both incremental and "composite" build mode in 3.5.

Well i think it has to be supported in compiler-cli first so maybe this issue should go to angular project?
:)

[aruballo]

Any updates on this? Looking to implement this in a complex project that I recently upgraded to Angular 8.2. I added the incremental and tsBuildinfoFile properties but I don't see a difference (I dont even see any new files generated anywhere)

[FDIM]

I am not sure what happened, but the API for incremental option/program is not available in TS 3.5, only in 3.6. It will soon be released though (23rd of August according to road map).

So for now I'd say this is blocked until angular packages depend on 3.6. Afterwards 2 solutions will be needed:

• find a way to configure current build to emit compiled ts files and tsbuildinfo in cache/out dir (side note: Maybe ivy could use this to cache factories as well to further improve cold start time)
• find a way to create incremental program that uses existing tsbuildinfo and feed that to current build. As far as I understand this program could be used as oldProgram (not sure though...)

As for cache dir @filipesilva , I guess it would be best to reuse outDir from ts config instead of adding new to angular.json? Currently angular compiler plugin is resetting this value and if it is set, build fails with a bit unclear message.

[aruballo]

@FDIM Where did you hear that? The documentation for typescript seem to indicate this is available in 3.4:

https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-4.html

[clydin]

The functionality was available via tsc (the TypeScript command line interface) in 3.4. What is needed here for it to be considered for inclusion is programmatic access.

[kukjevov]

@FDIM yes it is interesting about that API and version of typescript, because i followed release of 3.5 and fix was not present at time of release 3.5, but still they put it in changelog of 3.5 even thou i read some guy complaint that there is no API for incremental build available in 3.5, so it looks like some mess in release log, but it will be in 3.6 definitely :)

[FDIM]

@aruballo I've tried to use the api to see how it works, only to realize that it is not there

[aruballo]

How weird, I guess I'll have to wait for 3.6 before using this then. Thanks @FDIM

[kukjevov]

Great thanks :)

[alxhub]

@filipesilva is correct that composite projects are not supported currently. There is no strong technical reason for this, it's just not been a priority since there are other ways Angular users split up compilations (e.g. nx workspaces, Bazel) and we need to ensure the compiler does a good job for these circumstances first. Once the general multi-project story is in a good place, it will make sense to implement the TS layer.

The same is true for the incremental build cache. One of the big requirements is having strong incremental rebuild semantics under --watch. We have a roadmap for this that we're part of the way through implementing.

It's very important that we get the semantics right when we move to an incremental cache. Right now under --watch we have a safety net: if the compiler has a bug and does not handle an incremental change correctly, the user will just kill the --watch compilation (e.g. by killing ng serve) and rerun it. This safety net doesn't exist with an incremental build cache - if we process a change incorrectly, the mistake will persist until you turn off incremental compilation. Therefore I want to move slowly and deliberately when implementing this feature. I'd rather err on the side of not supporting it until we're absolutely sure our semantics are perfect.

[jukibom]

Just dropping in a use-case here for composite project support.

We have a typescript monorepo of which one of the packages is the main front-end angular app and which consumes a number of other application-like packages in a yarn workspace (and those packages have other dependencies in the repo, such as a core ECS package, a general service interaction layer etc). Building any one of these applications will auto-build any composite dependencies it needs (and their dependencies and so on), so conceptually we only need to run a build on any one top-level application a developer's working on.

For the angular frontend we have to first build all packages it depends on and get a build watcher going on each application a developer is working on. These are better than nothing but pretty slow and sometimes unreliable. It's also just generally a bit of a pain for devs to get an environment set up. So in our case it's not so much the incremental build cache (though that would be nice!) but the ability to allow typescript to recognise other composite projects in the repo with the build flag that we're really interested in.

[andreialecu]

I have a different use case regarding composite projects where I only have non angular component related code (so, nothing that would need JITing or AOTing) in separate projects under a monorepo/yarn workspace.

I was expecting to be able to use @ngtools/webpack with a tsconfig that defines the reference and enables composite mode, but it looks like it doesn't work properly, and doesn't build referenced projects. The separate projects output their compiled js to directories such as /lib underneath them, and everything is resolved through package.json references.

As a workaround, I need to build the referenced projects manually before starting the main angular webpack build.

I'm wondering if @ngtools/webpack could be updated to support this minimal scenario of composite projects until full AOT support is added.

[glebmachine]

Any updates on this?

[alan-agius4]

Transfering to the FW repo as the compiler doesn't support TypeScript incremental API.

Example it uses ts.createProgram, instead of ts.createIncrementalProgram with the latter is require to use the tsbuild info.

I did experiment a bit using the NGTSC plugin API with allows for a program to be supplied and patching on a small project https://github.com/ngrx/platform/tree/master/modules/component/src I saw the compilation to go from 5s to 2.5s.

const host = ts.createIncrementalCompilerHost(compilerOptions);
const wrappedHost = ngtsc.wrapHost(host, config.rootNames, compilerOptions)

const program = ts.createIncrementalProgram({
  options: compilerOptions,
  rootNames: config.rootNames,
  host: wrappedHost,
  createProgram: ts.createEmitAndSemanticDiagnosticsBuilderProgram
})

const { ignoreForEmit } = ngtsc.setupCompilation(program.getProgram(), undefined);

NB: NGTSC would also need to set versions to typechecking source files.

[glebmachine]

@alan-agius4 awesome news!
Hope you will implement support soon

let me know if I can help you, that is very important because of slow rebuilds on large projects

[clydin]

Rebuilds for a running instance (ng build --watch/ng serve) are already incremental. The support referenced in this issue would only affect warm build scenarios (i.e., repeat execution of ng build/ng serve).

[glebmachine]

Ok, what we do then with 40-60s rebuild time on ‘ng serve’?

angular ignores microfrontends for several years, bazel initiative was closed before it goes to production.

Angular will die if we will keep ignored that.
That’s a real problem!

we don’t need angular designer and features that cover corner cases

But, what is community real needs:

• fast build rebuild
• Ssr support out of the box (like next/nuxt do)
• Reduce bundle size (big angular drama)

For now, angular only fit to cms control panels, but lost competition to all public projects with lighthouse metrics, search engine optimizations and other common needs

till we lose that competition - nobody cares about template type checking, di, rx, modularity and great dx.

Please, think about that. I’m almost lost my faith in angular as instrument and angular as great community

[clydin]

Please try to keep the issue on the original topic.

If anyone has cases of 40-60 second application rebuilds, information regarding the project would greatly be appreciated by the team. If possible please open a separate issue detailing the performance situation that is being encountered. Even without access to the application code itself, the team has had success troubleshooting performance concerns when performance profiles have been provided. If needed, a team member can also provide assistance with capturing the performance profiles for analysis once the issue has been opened.

[mlc-mlapis]

The most important time is the incremental rebuild during the development process that is repeated hundred times.

[alan-agius4]

The most important time is the incremental rebuild during the development process that is repeated hundred times.

That’s not really what this issue is about. Incremental TypeScript API only improves subsequent cold builds as retrieves the old program data from disk.

[mlc-mlapis]

Yep, that's why I wanted to stress where the importance lies because Incremental TypeScript API can help at some point, but it's not crucial.

[simeyla]

That’s not really what this issue is about. Incremental TypeScript API only improves subsequent cold builds as retrieves the old program data from disk.

Isn't that a little naïve? When you change something after running ng serve it does an incremental build, and sometimes that incremental build takes 1 seconds and sometimes it takes 10. It depends what you change and what kind of ripple through effect that has through the rest of your code. It's not like you change 2 single lines of code in your source and then the angular compiler gives you 2 lines of updated .js files to patch.

When you change something that a lot of other pieces of your application depend upon there'll be a lot of unnecessary recompilation throughout the 'recompilation tree'. All for the incremental change that looked trivial. Surely some of that compilation would benefit from the Typescript incremental cache.

Is this issue basically dead? It's 3 years old, and while I periodically find it and read through the updates it looks like it's effectively been abandoned. I just don't buy the excuse that 'ng serve already does incremental so it wouldn't have much effect`. I realize these things are always complex, but saving 1 second that adds up. Also I would love for a cold build to be faster!

Q: When am I most likely to get distracted and pick up my phone?
A: When waiting for an incremental build.

[maxisam]

@simeyla if you want, you can try nx. If a project is big enough, I think NX is the way to go. Angular CI is good until the size of the project is too big

[alxhub]

Next step: followup with our tooling team and determine the potential benefits of this (vs. the rather large implementation cost)

[varthc]

Hi! Is there any update on this? Is it planned to include it? Thanks!

[a-boertien]

Is this now supported for Angular 18?

I received the following error on ng build when upgrading from 17-> 18 and the new build system (esbuild)

Option 'tsBuildInfoFile' cannot be specified without specifying option 'incremental' or option 'composite'

I don't have the tsBuildInfoFile option set in my tsconfig (and tsc --showConfig also shows this),
but setting incremental: true seems to fix it ?

[JoostK]

Is this now supported for Angular 18?

I received the following error on ng build when upgrading from 17-> 18 and the new build system (esbuild)

Option 'tsBuildInfoFile' cannot be specified without specifying option 'incremental' or option 'composite'

I don't have the tsBuildInfoFile option set in my tsconfig (and tsc --showConfig also shows this), but setting incremental: true seems to fix it ?

The CLI does indeed use incremental TS builds, if caching is enabled. OTOH, ngc (the raw variant of tsc that includes the Angular compiler) does not.

The CLI does set incremental using a nullish-assignment, so if you used to have it set to false before then that would explain that error.

https://github.com/angular/angular-cli/blob/d257388fee658e06fd4e09e369f9624db22e72c1/packages/angular/build/src/tools/esbuild/angular/compiler-plugin.ts#L542-L547

Edit: the CLIs behavior is being changed in angular/angular-cli#28161