Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce file driven tests. #125

Merged
merged 11 commits into from
Nov 5, 2015
Merged

Introduce file driven tests. #125

merged 11 commits into from
Nov 5, 2015

Conversation

ryaneberly
Copy link
Contributor

I'm looking for feedback on this experimental one.

  1. introduce file driven tests.
    If you run the TestFiles unit test. It will pick up all .cf files under src\test\resources\com\cflint\tests. It applies the cflint config from the folder it is looking at, and compares the lint results with the expected results file.
    Normally a sub folder would be dedicated to a particular checker, and the config only allows that particular rule to fire.

You can control the behavior of the run with test.properties. You normally care about only 1 test at a time when you are building a checker -- there is a option to run the test for only the most recently changed input file. There is also an option to overwrite the expected results instead of comparing against them.

  1. introduce a check for misuse of comparison when assignment is intended.
    https://cwe.mitre.org/data/definitions/482.html. This is revelant to CF. And was a good proof of concept to demonstrate the file test.

@justinmclean
Copy link
Contributor

Like the idea, it probably a little more work for small tests, but as they get more compile it would be easier to add more tests for a rule.

I tend to use -Dtest=*testfile*#*testcase* quite a bit so would be nice if we had a way to specify the test to run via wildcards.

@ryaneberly
Copy link
Contributor Author

Thanks. In other projects, I've found it encourages contributions. Folks seem to find it easier to just drop in a code sample, than to figure out the unit test nuances.

I'll have to try -Dtest=name#subset with it. I haven't used that before -- seems like it could work.

@justinmclean
Copy link
Contributor

Also made above clearer it's -Dtest=*testfile*#*testcase*

@ryaneberly
Copy link
Contributor Author

I don't think maven/surefire plays nice with running a single parameterized test.

ryaneberly added a commit that referenced this pull request Nov 5, 2015
Introduce file driven tests.
@ryaneberly ryaneberly merged commit fd72688 into dev Nov 5, 2015
@ryaneberly ryaneberly deleted the testing branch November 5, 2015 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants