Strategies for more accurate code coverage report

[kwonoj]

Related with PR #557

Current code coverage report (i.e, build 823) is indicating coverage of source points to transpiled results from original typescript based sources. Due to this reason it does not reports accurate coverage percentage also does not allows navigation to coverage detail, makes bit hassle to triage when coverage number goes down.

Coverage report shows coverage against transpiled

Not able to navigate actual source detail

PR #557 partially resolves this issue by enabling coverage remapping via sourcemaps. Still it doesn't give right coverage to original source but es6 transpiled source because of current build mechanism,

1. tsc compiles into ES6
2. babel uses compiled ES6 and compiles into commonjs

since test cases are using commonjs output from 2, even after enabling sourcemap remapping coverage report shows result against ES6 output from 1 instead of original source doesn't give right accuracy of test coverage along with lacks of navigation. (example report can be found at here)

source map enabled

detail's still not available

Could think of few options to resolve this issues,

1. build commonjs module via typescript compiler
: Not considerable due to #248
2. leave current build mechanism as-is, build commonjs module once again via typescript compiler only for test coverage
: Though this can work, still bit hacky since it requires build couple, or triple for same build.
3. it's ok to live with current coverage status, let's leave as-is.
4. Maybe some other way I couldn't think of

I have tried option 2, could create test coverage shows quite different number than what we're currently having against transpiled results. (example report can be found at here)

Ideal coverage report, shows accurate coverage to source

detailed navigation is available

Personally though it's bit hacky way, would like to suggest 2. (rebuild cjs via tsc only for coverage), does not hurt actual output also get detailed coverage to actual code.

Update: disposed above options by looking composing sourcemap as hinted.

[benlesh]

It seems like there are ways to compose the sourcemaps. LIke sourcemap to ES6 source map again to Babel.

[kwonoj]

That seems quite feasible solution. I'll try to look for some references.

[kwonoj]

Spend some time and could able to list up some options.

using babel --inputSourceMap

: babel supports to have 'base' sourcemap to generate sourcemap of transpiled results to corresponds original source. This works, but....

• Cons : only supported for inline source map for now, inline source map is probably not acceptable. also options itself is sort of undocumented option though it is listed as available option

using gulp-sourcemaps

: gulp has plugin to stream staged transpilation and generate source maps from those. Most simple, straightforward way.

• Cons : need to introduce new build system other than npm script.

using multi-stage-sourcemap module

: multi-stage-sourcemap module supports staged conversion of sourcemap.

• Cons : module has known issue (File property is being left out of the resulting rawSourceMap azu/multi-stage-sourcemap#3), but code is not updated quite while(~1yr).

[kwonoj]

Order of personal preference is 1.gulp 2.multi-stage-sourcemap (inline source map with babel is out of option for me).

Of course using gulp will require additional effort to replace existing npm based build scripts, but it'll bring some additional side effect of better windows support (specifically dealing with cross-platform environment variable, file glob) along with streamlined source map support using ready made plugin. Build script for current repo is not yet very complex, so it won't require huge effort to create new scripts for build compare to create new script for source map conversion via multi-stage-sourcemap module (and I'm willing to take this effort if it's going to happen).

Thoughts? or any other suggestions for tooling support would be appreciated.

[kwonoj]

After give few more try, seems gulp-sourcemap also has some issues. Given aspect of coverage check failure affects PR, I'll add changes based on current build script with multi stage sourcemap first.