Chores/test-coverage-setup-#151817717

[chrisdolendo]

No description provided.

[akegan]

can we setup codeclimate independently of semaphore? as a separate github integration? It might make it easier to understand the results

Here's some documentation: https://docs.codeclimate.com/docs/github-pull-requests#section-pull-request-statuses

[akegan]

Looks like code climate costs money for >4 users committing code https://codeclimate.com/quality/pricing/. I wonder how that was handled in the past

We could just do chris, cat, me, and martin, and not analyze josh's code? See how it goes and if we find it super valuable we can potentially procure it.

alternatively, are there free test coverage reporting packages for ruby and js?

[chrisdolendo]

Great we have coverage now after configuring semaphore + installing simplecov (ruby) and istanbul (for lcov js reporting) -- but for some reason some JS tests fail even if no actual feature code was changed. 🤔

[akegan]

@chrisdolendo we had been getting some red text running tests locally. seemed like warning-type things. I wonder if changing the packages somehow affected the tolerance of those tests

[chrisdolendo]

@akegan good call I will look into the packages

[chrisdolendo]

I figured out the issue with the e2e test from research -- essentially using puppeteer (which e2e tests uses) and jest with code coverage enabled breaks the test.

• Code coverage argos-ci/jest-puppeteer#90:

You can't activate coverage for E2E. It is very complicated to do because you don't have direct access to the tested code.

• Can't get test coverage with jest + puppeteer jestjs/jest#6793
• Promise rejected with: 'cov_2itgu9hdrb is not defined istanbuljs/nyc#514

I tried implementing puppeteer-to-istanbul but it didn't work, because istanbul is already baked in to jest so it conflicted, and also needs the latest node version.

SO, I turned on coverage just for the unit tests and did not include e2e in it until there a solution for jest.

Code coverage went up to 76% after including JS tests.

[akegan]

with python tests we generally ran the tests and the coverage separately. Is there a reason why you're combining it into one line here?

[chrisdolendo]

@akegan to feed two birds with one seed - but I can separate them -- would you like me to do that?

[akegan]

My only concern is that it might be hard to distinguish on semaphore if the tests are failing due to tests failing or due to coverage not passing. would that be an issue here?

[chrisdolendo]

This just reports the coverage that gets passed to code climate, it doesn't keep it from passing or failing. Code climate will let us know in the PR if it doesn't pass. e.g. this PR shows that the new changes don't pass the Diff Test Coverage:

[chrisdolendo]

I DID IT!!! Had to update a sempahore command. We now are up 76% test coverage.

[akegan]

Looks great, thanks @chrisdolendo !