Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

Create tool to check skipped tests #837

Open
jodh-intel opened this issue Jan 11, 2018 · 0 comments
Open

Create tool to check skipped tests #837

jodh-intel opened this issue Jan 11, 2018 · 0 comments

Comments

@jodh-intel
Copy link
Contributor

Some of our tests employ "skips" to avoid running tests that are known to fail (generally due to open bugs or known limitations). These skips look like this:

  • ginkgo:

    Skip("Issue https://github.com/clearcontainers/agent/issues/181")
    
  • bats:

    skip "This is not working (https://github.com/clearcontainers/tests/issues/694)"
    

Since we are careful to include an issue number, we could write a tool (check-disabled-tests.{sh,go} ?) that would:

  1. Search for all the skips.
  2. Warn about any skips that do not contain a URL.
  3. Check if the referenced issue is closed.
  4. If the issue is closed, warn that the file needs an update to remove the skip.

This tool should be run regularly (ideally as part of every PR CI run, but weekly would probably suffice) to avoid test code that isn't currently being run from going stale, but also to maximise our test coverage.

mcastelino pushed a commit to mcastelino/tests that referenced this issue Jan 23, 2019
Update the static check script to ignore `file://` URLs which won't
exist (since they only appear in examples).

Fixes clearcontainers#837.

Signed-off-by: James O. D. Hunt <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant