Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: refactor coverage logic #35767

Closed
wants to merge 4 commits into from
Closed

test: refactor coverage logic #35767

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5877a15
test: refactor coverage logic
bcoe Oct 23, 2020
31359a5
Apply suggestions from code review
bcoe Oct 23, 2020
025a363
chore: address code review
bcoe Oct 23, 2020
cd901bb
Update Makefile
bcoe Oct 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 18 additions & 14 deletions BUILDING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,28 +389,32 @@ $ make coverage
```

A detailed coverage report will be written to `coverage/index.html` for
JavaScript coverage and to `coverage/cxxcoverage.html` for C++ coverage
(if you only want to run the JavaScript tests then you do not need to run
the first command `./configure --coverage`).
JavaScript coverage and to `coverage/cxxcoverage.html` for C++ coverage.

_Generating a test coverage report can take several minutes._

To collect coverage for a subset of tests you can set the `CI_JS_SUITES` and
`CI_NATIVE_SUITES` variables (to run specific suites, e.g., `child-process`, in
isolation, unset the opposing `_SUITES` variable):
If you only want to run the JavaScript tests then you do not need to run
the first command `./configure --coverage`. You can run `make coverage-run-js`,
bcoe marked this conversation as resolved.
Show resolved Hide resolved
to execute JavaScript tests independently of the C++ test suite:

```text
$ CI_JS_SUITES=child-process CI_NATIVE_SUITES= make coverage
$ make coverage-run-js
```

The above command executes tests for the `child-process` subsystem and
outputs the resulting coverage report.
If you are updating tests and want to collect coverrage for a single test file
(e.g. `test/parallel/test-stream2-transform.js`):

```text
$ make coverage-clean
$ NODE_V8_COVERAGE=coverage/tmp python tools/test.py test/parallel/test-stream2-transform.js
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can just set NODE_V8_COVERAGE, and then run tests any which way.

$ make coverage-report-js
```

Alternatively, you can run `make coverage-run-js`, to execute JavaScript tests
independently of the C++ test suite:
You can collect coverage for the entire suite of tests for a given subsystem
by providing the name of a subsystem:

```text
$ CI_JS_SUITES=fs CI_NATIVE_SUITES= make coverage-run-js
$ make coverage-clean
$ NODE_V8_COVERAGE=coverage/tmp python tools/test.py -J --mode=release child-process
$ make coverage-report-js
```

The `make coverage` command downloads some tools to the project root directory.
Expand Down
35 changes: 9 additions & 26 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,20 +197,11 @@ check: test
# Remove files generated by running coverage, put the non-instrumented lib back
# in place
coverage-clean:
if [ -d lib_ ]; then $(RM) -r lib; mv lib_ lib; fi
$(RM) -r node_modules
$(RM) -r gcovr build
$(RM) -r out/$(BUILDTYPE)/.coverage
$(RM) out/$(BUILDTYPE)/obj.target/node/gen/*.gcda
$(RM) out/$(BUILDTYPE)/obj.target/node/src/*.gcda
$(RM) out/$(BUILDTYPE)/obj.target/node/src/tracing/*.gcda
$(RM) out/$(BUILDTYPE)/obj.target/node/gen/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/node/src/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/node/src/tracing/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/cctest/src/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/cctest/test/cctest/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/embedtest/src/*.gcno
$(RM) out/$(BUILDTYPE)/obj.target/embedtest/test/embedding/*.gcno
$(RM) -r coverage/tmp
$(FIND) out/$(BUILDTYPE)/obj.target -name "*.gcda" -type f -delete
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was having trouble on OSX with left over gcno and gcda files, I believe it's safe to simply remove all files generated by gcov.

$(FIND) out/$(BUILDTYPE)/obj.target -name "*.gcno" -type f -delete
bcoe marked this conversation as resolved.
Show resolved Hide resolved

.PHONY: coverage
# Build and test with code coverage reporting. Leave the lib directory
Expand Down Expand Up @@ -245,8 +236,8 @@ coverage-test: coverage-build
$(RM) out/$(BUILDTYPE)/obj.target/node/src/*/*.gcda
$(RM) out/$(BUILDTYPE)/obj.target/node_lib/src/*.gcda
$(RM) out/$(BUILDTYPE)/obj.target/node_lib/src/*/*.gcda
-NODE_V8_COVERAGE=out/$(BUILDTYPE)/.coverage \
TEST_CI_ARGS="$(TEST_CI_ARGS) --type=coverage" $(MAKE) $(COVTESTS)
-NODE_V8_COVERAGE=coverage/tmp \
TEST_CI_ARGS="$(TEST_CI_ARGS) --type=coverage" $(MAKE) $(COVTESTS)
$(MAKE) coverage-report-js
-(cd out && "../gcovr/scripts/gcovr" \
--gcov-exclude='.*\b(deps|usr|out|cctest|embedding)\b' -v \
Expand All @@ -259,17 +250,10 @@ coverage-test: coverage-build
@grep -A3 Lines coverage/cxxcoverage.html | grep style \
| sed 's/<[^>]*>//g'| sed 's/ //g'

COV_REPORT_OPTIONS = --reporter=html \
--temp-directory=out/$(BUILDTYPE)/.coverage --omit-relative=false \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these options are encapsulated in the .nycrc file.

--resolve=./lib --exclude="benchmark/" --exclude="deps/" --exclude="test/" --exclude="tools/" \
--wrapper-length=0
ifdef COV_ENFORCE_THRESHOLD
COV_REPORT_OPTIONS += --check-coverage --lines=$(COV_ENFORCE_THRESHOLD)
endif

.PHONY: coverage-report-js
coverage-report-js:
$(NODE) ./node_modules/.bin/c8 report $(COV_REPORT_OPTIONS)
-$(MAKE) coverage-build-js
$(NODE) ./node_modules/.bin/c8 report

.PHONY: cctest
# Runs the C++ tests using the built `cctest` executable.
Expand Down Expand Up @@ -305,9 +289,8 @@ tooltest:

.PHONY: coverage-run-js
coverage-run-js:
$(RM) -r out/$(BUILDTYPE)/.coverage
$(MAKE) coverage-build-js
-NODE_V8_COVERAGE=out/$(BUILDTYPE)/.coverage CI_SKIP_TESTS=$(COV_SKIP_TESTS) \
$(RM) -r coverage/tmp
-NODE_V8_COVERAGE=coverage/tmp CI_SKIP_TESTS=$(COV_SKIP_TESTS) \
TEST_CI_ARGS="$(TEST_CI_ARGS) --type=coverage" $(MAKE) jstest
$(MAKE) coverage-report-js

Expand Down