🚀 Feature: API surface for developing an up-to-date VS Code Extension

[Danielku15]

Is your feature request related to a problem or a nice-to-have?? Please describe.
As of today there is no up-to-date integration of Mocha into Visual Studio Code. There is the outdated and not maintained Mocha Test Explorer using an own test explorer instead of the VSCode Integrated one.

I was thinking of making a new extension utilizing the VS Code Testing API based on the examples the provide.

Unfortuantely integrating with Mocha in the right way for scraping the tests and executing them is not easy.

Describe the solution you'd like

It would be great if Mocha would expose additional APIs to fill the missing gap between Mocha and VS Code. For this I would need:

1. Provide a programmatic way of initializing a Mocha instance from a workspace directory (loading any available configs).
2. Provide a programmatic way of accessing the test hierarchy including the related file location.
3. Provide a programmatic way of executing test given any "test hierarchy" objects (as exposed by 2.).
4. Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.

Describe alternatives you've considered

1. Adopting the code from vscode-mocha-test-adapter which does a lot of stuff manually.

Additional context

[JoshuaKGoldberg]

I'm curious how much this intersects with the other API requests, namely those filed by @nicojs for Stryker: #3882, #4572.

It makes sense why you'd want these APIs though. I think given #5027 we're not likely to get to this soon, but it's on our radar. @Danielku15 if you could post more details that'd be really helpful:

• What's the total list of important APIs you need? As in, for each, the root behavior they're implementing, and possible inputs & outputs?
• What APIs already exist that do what you want?
• What are the things vscode-mocha-test-adapter has done manually that you'd like to move to an API?

I think the OP mentions the "root behavior" points nicely - so getting more in-the-weeds on what proposed APIs would satisfy them would help us triage this. Thanks! 🤎

[Danielku15]

What's the total list of important APIs you need? As in, for each, the root behavior they're implementing, and possible inputs & outputs?

I think it is hard to tell directly how the API signatures could look like. Mainly because I am not yet aware how mocha internally already stores the whole test file > suite > case hierarchy. To teach VS Code about tests you create vscode.TestItem objects. Beside the attached custom infos (e.g. for later execution or syncing with Mocha) the properties are: https://vscode-api.js.org/interfaces/vscode.TestItem.html

Looking at the example extension from Microsoft you can get a bit the feeling how things would integrate:https://github.com/microsoft/vscode-extension-samples/tree/main/test-provider-sample/src

To adapt the example to Mocha roughly the points mentioned above would be needed. I tried to define an interface description that could serve the plugin:

class MochaTestExtensionFacade {
    // Goal: Provide a programmatic way of initializing a Mocha instance from a workspace directory (loading any available configs).
    // Usage: This would be the main "context" object within the VS code extension
    // Behavior: this would setup internally mocha with all the configs and details like during normal test execution
    // in a directory making everything ready for the other operations.
    // any transpilers, plugins and settings would have to be respected (e.g. to support typescript)
    constructor(workingDirectory: string);
    dispose(): void;
    
    // Goal: Provide a programmatic way of accessing the test hierarchy including the related file location.
    // Usage: This would be used to obtain all details to fill the test list/explorer.
    // Behavior: provides all test files with the contained details about the tests.
    getRootSuite(): Promise<mocha.Suite>;
           
    // Goal: Provide a programmatic way of executing test given any "test hierarchy" objects (as exposed by 2.).
    // Usage: This would be used when a user requests test execution on. we 
    // Behavior: would create a runner which can be used execute the test (suite). 
    createRunner(suite:mocha.Suite): Promise<mocha.Runner>; 
    createRunner(suite:mocha.Test): Promise<mocha.Runner>; 
    
    // Watching Variant A: using Mocha watcher
    
        // Goal: Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.
        // Behavior: Similar to the existing `mocha --watch` this would internally setup
        //           all file system watchers and components to detect any dynamic changes done by the user. 
        //           on the input code. 
        constructor(workingDirectory: string, watch: boolean);
        // Goal: get notified about the changes in the root suite
        // Usage: We would listen to those changes and then dynamically update the test list/explorer accordingly. 
        onRootSuiteChanged( callbackWithChanges: (added:(mocha.Suite|mocha.Test)[], removed:(mocha.Suite|mocha.Test)[], modified:(mocha.Suite|mocha.Test)[])=>void );

    // Watching Variant B: using VS code watchers
        // Goal: Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.
        
        // Usage: We use those patterns to setup vscode.workspace.createFileSystemWatcher(<pattern>)
        // Behavior: It returns the file path patterns to start listeners. If the mocha config is changed this is signaled. 
        getWatchPatterns(): Promise<string[]>
        onWatchPatternsChanged( callbackWithChanges: (added: string[], removed:string[]) => void ): void;
        
        // Usage: We would call this when VS code detects changes on files.
        // Behavior: It would add or reload the tests and suites from the specified file 
        addOrUpdateFile( uri: string ): Promise< { added:(mocha.Suite|mocha.Test)[], removed:(mocha.Suite|mocha.Test)[] } >;
        
        // Usage: We would call this when VS code detects deleted files.
        // Behavior: It would add or reload the tests and suites from the specified file 
        removeFile( uri: string ): Promise< { removed:(mocha.Suite|mocha.Test)[] } >;
}

There might be the need for extensions/adaptions on existing interfaces and types to hold all information.

What APIs already exist that do what you want?
There are some APIs on the main Mocha object but from a workflow they don't really fit:

• Initializing a Mocha object like the CLI does is not publicly available. You have to do the loading manually.
• There APIs for loading and unloading files. But the loaded test structure is not available.
• For exeuction most things seem available. But unfortunately no "incremental" status updates are there. A user would expect to see how tests are pending, executing and done (success/fail) including the related logs attached, execution times shown, code coveragte trackable etc.

What are the things vscode-mocha-test-adapter has done manually that you'd like to move to an API?

Basically the things described above. The old adapted patched out various things in mocha to obtain the list of tests with their locations. I think also for execution a similar pattern is used to hook into the execution chain. For options loading they require duplicated configuration or attempt loading some settings manually from configs.

https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/worker/patchMocha.ts
https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/configReader.ts
https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/worker/reporter.ts

[voxpelli]

Just wanted to chime in and say that I'm very 👍 to this and I personally would also be very open on having an official VSCode extension be maintained within the mochajs org, based on https://github.com/hbenl/vscode-mocha-test-adapter or written from scratch.

Is this something you would be interested in helping out with @Danielku15?

[Danielku15]

It would be certainly in my interest to have an adapter but I'm also new into the topic of writing such a test extension and also to the mocha codebase. Hence I'm not sure how well I can progress in making the required changes. I lately had to shift the focus in my project a bit more to the features than to the dev-experience but somehow I miss every time such a plugin when working on features.

[voxpelli]

@Danielku15 I wasn't meaning to suggest that you should be building it all, just thinking if you want to be included if we are taking on doing an official VSCode Extension ;)

[Danielku15]

@voxpelli Success 🥳🎉 I forked the VS Code Extension Test runner from Microsoft and rewrote it to work with plain Mocha. Here a demo from within my project. It activates with the mocharc files and then provides all features like running and debugging tests. It also works with TypeScript. Only code coverage I left out for now as this depends again on the framework used.

Here a sneak peek video:

Code_lQARMvDNpt.mp4

I cleaned my code and uploaded my current state to CoderLine/mocha-vscode#1
Still some things to figure out and discuss but I'm already happily running and debugging tests in my project using this extension 😁

Let me know what you folks think.

[Danielku15]

@voxpelli Do you think the Mocha project would like to take over this extension? Otherwise I'd change the copyright notices and branding and release it for now under my name.

[voxpelli]

@Danielku15 Sorry for late reply, I think we’re not ready to take it on yet, exciting that you are looking into it!