Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare run_tests to run multiple tests #1355

Merged
merged 3 commits into from
Oct 19, 2016
Merged

prepare run_tests to run multiple tests #1355

merged 3 commits into from
Oct 19, 2016

Conversation

ozyx
Copy link
Contributor

@ozyx ozyx commented Oct 18, 2016

Hello!

In this PR I've made some minor adjustments to allow for run_tests to be able to actually run a list of tests, instead of just one-at-a-time.

There are some issues that have arisen/will arise moving forward and I'd like to open a dialogue to see your opinion on how I should go about this.

Firstly, about mocha.grep...
I see now the reason why mocha tests are being run with multiple calls to before() and after() hooks -- each call to run_tests also called mocha.grep, turning a singular test suite containing 10 tests into 10 test suites containing 1 test. As my ultimate goal is to make mocha tests run properly, I would have to either remove the grep call completely (this would mean ALL mocha tests in a file will always run, breaking "Run Selected Tests" functionality) or construct a long grep string containing all selected tests and pass that to the grep call.

Second problem -- recording start and end times:
The approach by NTVS currently is to call RecordStart on the given test case, send mocha the testName and testFile and have mocha grep the one testName out of the provided testFile. Then once that process exits, it's safe to call RecordEnd and record the end time for the test, then repeat.
However, I want to run a list of tests-- in order to do so, I have to provide mocha with the list up front so it can adjust its test suite accordingly. If I'm providing mocha with a list of tests to run up front, we can no longer record accurate Start and End times for those tests. I could try to record the times in mocha.js, but this adds another requirement for the run_tests interface, and might make changes more difficult down the road.

Any guidance is welcome and appreciated. Please let me know if you need me to elaborate on anything written here, hopefully I haven't rambled too much...

Thanks again

@msftclas
Copy link

Hi @ozyx, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by Microsoft and real humans are currently evaluating your PR.

TTYL, MSBOT;

@mjbvz
Copy link
Contributor

mjbvz commented Oct 18, 2016

@ozyx Thanks for the work. Good progress overall.

For Mocha, I took and look at their programmatic api and it unfortunately seems poorly designed for this use case. I think you are correct that you'll have to construct a grep string for mocha. Starting from a list of tests cases, the flow would probably look something like this:

  • Group tests cases by file.

  • For each file:

    • Create a new instance of mocha.
    • Create a grep string by joining together all the test names. This would be something like:
    // https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/Regular_Expressions
    function escapeRegExp(string){
       return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
    }
    
    var testGrepString = '^(' + testCases.map(function(testCase){
        return escapeRegExp(testCase.title)
    }).join('|') + ')$'; 
    • call Mocha.grep
    • call Mocha.addFile
    • call Mocha.run

This adds extra work because you now have to worry about managing a run of Mocha for each file, but this is the only way I can see to properly support this.


For the timing, you'll want to move the execution time calculation from the C# code into test runner JavaScript implementations themselves. This would be another property on the result object.

In the case of Mocha, in the test event, set a startTime property on the result using Date.now(). Then in the test end event, compute the taken time using startTime and Date.now(). A similar approach should work for the tape and the export runners.

Let me know if that makes sense.

@mjbvz mjbvz merged commit 3321602 into microsoft:test-adapter-improvements Oct 19, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants