diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b2665a2..f866c97 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -8,7 +8,13 @@ QTap is built to be simple, lean, and fast; valued and prioritised in that order ### Simple -We value simplicity in installing and using QTap. This covers the CLI, the public Node.js API (module export), printed output, the overall npm package, and any of concepts required to understand these. +We value simplicity in installing and using QTap. This covers the CLI, the public Node.js API (module export), printed output, the overall npm package, and any concepts required to understand these. Importantly, it applies not just to the day 1 experience when everything is going right, but also to day 1000 when something goes wrong on a Friday night, and you're debugging a mysterious bug while offline at 30,000 ft. + +We strive for a level of simplicity that naturally brings ease-of-use, rather than making an internally complex tool with an "easy" layer on the outside. See also _Simple Made Easy_ (2011) by Rich Harris ([cliff notes](https://paulrcook.com/blog/simple-made-easy), [original talk and slides](https://www.infoq.com/presentations/Simple-Made-Easy/)) + +We favour a single way of doing something that works everywhere, over marginal gains that would introduce layers of indirection, abstraction, conditional branches, or additional dependencies. We believe this significantly increases the stability of our code, by reducing the cost of testing and increasing the value of tests we do have. When there are fewer permutations of how code may run, there are fewer ways a bug may hide. If it works once, it should work always. We believe this also decreases the likelihood that a change in the user's environment may cause our tool to stop working, and thus warrant frequent updates and releases just to keep working. + +We favour an explicit and efficient inline implementation (e.g. in the form of a single well-documented function with a clear purpose, which may be relatively long, but is readable and linear), over many local functions that are weaved together. We believe this makes the code easy to contribute to, and, when a bug does slip in, quick to debug and get you back on track. Examples: @@ -18,13 +24,29 @@ Examples: qtap test.html ``` - This argument relates directly to a concept we know the user naturally knows and is most concerned about (their test file). There are no other unnamed arguments. There are no required options or flags. The default browser is selected automatically. No awareness of test framework is required (in most cases). + This one argument relates directly to a concept we know the user naturally knows and is familiar with (their test file). There are no other unnamed arguments. There are no required options or flags. The default browser is selected automatically. No awareness of test framework is required in most cases. When invoking `qtap` without arguments out of curiosity, we generate not just an error message but also the `--help` usage documentation. -* We favour a single way of doing something that works everywhere, over marginal gains that would introduce layers of indirection, abstraction, conditional branches, or additional dependencies. +* QTap builds on the universal [Test Anything Protocol](https://testanything.org/). + + QTAP works out-of-the-box with QUnit, Mocha, and Jasmine. + + But, it's important to understand that this isn't because hardcode these or bundle some plugin-like adapters for these within the main page. Doing so would have likely have limited the versions they support, which in turn imposes an update churn for the end-user. It might also sublty alter behaviour in ways the user doesn't own and won't know about until things fail, at which point the illusion of "just work" quickly erodes. It would also incur extra maintenance and support issues. + + Instead, we build on an the framework-agnostic TAP protocol. The user is responsible for setting that up, but once they do, everything truly just works and is transparent. The output sent to QTap is also visible in the browser console. There are no surprise side-channels or secret sauce. + + See also the [Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy). + +* QTap does not modify your code, and does not limit your debugging capabilities. + + No bundler or transpiler is applied to your code. The code you supply is the code we run. Yes, that means if you write code in TypeScript, or use newer JavaScript syntax (via Babel, SWC, etc.), you will have to run your build first (or run your build in watch mode in a separate terminal tab). + + We believe this is ultimately a good thing, because this is something the project will know about and be familiar with regardless, since you would already need this for outside your tests (e.g. when publishing or deploying your code). + + By choosing not to "own" this space, we encourage you to test your real code or real bundle. If we provided our own way to bundle or compile your code for you, it would likely differ in subtle ways from how you do it outside tests. It would be another thing to learn about and setup, and another thing to debug when it inevitably breaks or limits you in some unforeseen way. Even when it works, it may cause your tests to become dependent on the test runner to work (i.e. cannot be debugged standalone), and may obscure bugs (i.e. something the test runner is doing may cause your code to pass tests in ways it wouldn't otherwise). -* We favour an explicit and efficient inline implementation (e.g. in the form of a single well-documented function with a clear purpose, which may be relatively long, but is readable and linear), over many local functions that are weaved together. + See [QTap Internal: Server](#qtap-internal-server) for how this is implemented. ### Lean @@ -34,7 +56,7 @@ Examples: * We prefer to write code once in a way that is long-term stable (low maintenance cost), feasible to understand by inexperienced contributors, and thus affordable to audit by secure or sensitive projects that audit their upstream sources. For this and other reasons, we only use dependencies that are similarly small and auditable. -* We maintain a small bundle size that is fast to download, and quick and easy to install. This includes ensuring our dependencies do not restrict or complicate installation or runtime portability in the form OS constrains or other environment requirements. +* We maintain a small bundle size that is fast to download, and quick and simple to install. This includes ensuring our dependencies do not restrict or complicate installation or runtime portability in the form OS constrains or other environment requirements. * We will directly depend on at most 5 npm packages. Requirements for dependencies: @@ -89,7 +111,7 @@ Set `--verbose` in the QTap CLI to enable verbose debug logging. To avoid: - Karma provided a way to [configure proxies](http://karma-runner.github.io/6.4/config/configuration-file.html#proxies) which would let you add custom paths like `/foo` to Karma's proxy server, and forward those to your application. I'd like this to placing this complexity on the end-user. Not by proxying things better or more automatically, but by not breaking these absolute references in the first place. - Karma recommended against full transparent proxing (e.g. `/*`) as this would interfere with its own internal files and base directories. It'd be great to avoid imposing such limitation. -* Invisible or non-portable HTML compromises easy debugging of your own code: +* Invisible or non-portable HTML compromises ease of debugging for your own code: - [Airtap](https://github.com/airtap/airtap) takes full control over the HTML by taking only a list of JS files. While generating the HTML automatically is valuable for large projects (e.g. support wildcards, avoid manually needing to list each JS file in the HTML), this makes debugging harder as you then need to work with your test runner to debug it (disable headless, disable shutdown after test completion, enable visual test reporter). While some projects invest in a high-quality debugging experience, it's always going to lag behind what the browser offers to "normal" web pages. - [Jest](https://jestjs.io/docs/api), and others that don't even run in a real browser by default, require you to hook up the Node.js/V8 inspector to Chrome to have a reasonable debugging experience. This is non-trivial for new developers, and comes with various limitations and confusing aspects that don't affect working with DevTools on regular web pages. - Karma offers [configurable customDebugFile](http://karma-runner.github.io/6.4/config/configuration-file.html#customdebugfile) to let you customize (most) of the generated HTML. This is great, but comes at the cost of learning a new template file, and an extra thing to setup and maintain. diff --git a/README.md b/README.md new file mode 100644 index 0000000..2aad5af --- /dev/null +++ b/README.md @@ -0,0 +1,69 @@ +
+ +[![Continuous integration](https://github.com/qunitjs/qtap/actions/workflows/CI.yaml/badge.svg)](https://github.com/qunitjs/qtap/actions/workflows/CI.yaml) +[![Tested with QUnit](https://qunitjs.com/testedwith.svg)](https://qunitjs.com/) +[![npm](https://img.shields.io/npm/v/qtap.svg?style=flat)](https://www.npmjs.com/package/qtap) + +# QTap + +_Run JavaScript unit tests in real browsers, real fast._ + +
+ + +## Getting started + +1. Install QTap: + ``` + npm install --save-dev qtap + ``` +2. Run your tests: + ``` + npx qtap test/index.html + ``` +3. There's no step 3! + +## Features + +* **Framework Agnostic** + Use any test framework, whether QUnit, Mocha, Jasmine, Jest, Tape, or [anything else via a simple TAP plugin](https://testanything.org/producers.html#javascript). + +* **Anywhere** + - Cross-platform on Linux, Mac, and Windows. + - Built-in support for headless and local browsers (including Firefox, Chromium, Chrome, Edge, and Safari). + - Run remotely in real browsers on real mobile devices and tablets. + - Run on Android or iOS via cloud providers like [BrowserStack](https://www.browserstack.com/) and [SauceLabs](https://saucelabs.com/). + +* **Simplicity** + - No configuration files. + - No adapters or changes to how you write your tests. + - No installation wizard. + - No plugins or other packages required for most uses. + +* **Speed** + TODO - benchmarks. + +* **Real Debugging** + - Retreive console errors, uncaught errors, and unhandled Promise rejections from the browser directly in your build output. + - Instantly debug your tests locally in a real browser of your choosing with full access to browser DevTools to set breakpoints, measure performance, step through function calls, measure code coverage, and more. + - No imposed bundling or transpilation. Only your unchanged source code or production bundler of choice, running as-is. + - No need to inspect Node.js or attach it to an incomplete version of Chrome DevTools. + +* **Real Browsers** + - No need to support yet another "browser" just for testing (jsdom emulation in Node.js). + - No Selenium or WebDriver to install or update (e.g. chromedriver or geckodriver). + - No downloading large binaries of Chrome (e.g. Puppeteer). + - No patched versions of browsers (e.g. Playwright). + - No Docker containers. + +* **Continuous Integration** + GitHub, Jenkins, Travis, Circle, you can run anywhere. + +* **Ecosystem** + Your test framework likely already supports TAP. + + When you enable TAP in your frontend unit tests or backend Node.js tests, a door opens to an ecosystem of test runners, output formatters, and other [tools that consume the TAP protocol](https://testanything.org/consumers.html). + +## Prior art + +QTap was inspired by [Airtap](https://github.com/airtap/airtap) and [testling](https://github.com/tape-testing/testling). It may also be an alternative to [Testem](https://github.com/testem/testem/), [Web Test Runner](https://modern-web.dev/docs/test-runner/overview/), [TestCafé](https://testcafe.io/), [Karma Runner](https://github.com/karma-runner/) (including Testacular, [karma-tap](https://github.com/bySabi/karma-tap), and [karma-qunit](https://github.com/karma-runner/karma-qunit/)), [grunt-contrib-qunit](https://github.com/gruntjs/grunt-contrib-qunit), [wdio-qunit-service](https://webdriver.io/docs/wdio-qunit-service/), and [node-qunit-puppeteer](https://github.com/ameshkov/node-qunit-puppeteer). diff --git a/bin/qtap.js b/bin/qtap.js index 82be580..d75a74d 100755 --- a/bin/qtap.js +++ b/bin/qtap.js @@ -51,7 +51,8 @@ program ) .option('-r, --reporter ', 'One of "minimal", "dynamic", or "none".', 'minimal') .option('-w, --watch', 'Watch files for changes and re-run the test suite.') - .option('-d, --debug', 'Enable debug mode (non-headless browser, and verbose logging).') + .option('-d, --debug', 'Enable debug mode. This keeps the browser open,\n' + + 'and for local browsers it will launch visibly instead of headless.') .option('-v, --verbose', 'Enable verbose logging.') .option('-V, --version', 'Display version number.') .helpOption('-h, --help', 'Display this usage information.') @@ -75,7 +76,7 @@ if (opts.version) { connectTimeout: opts.connectTimeout, reporter: opts.reporter, debugMode: opts.debug || (process.env.QTAP_DEBUG === '1'), - verbose: opts.debug || opts.verbose, + verbose: opts.verbose, }); process.exit(result.exitCode); } catch (e) { diff --git a/src/client.js b/src/client.js index f81285d..c9756ba 100644 --- a/src/client.js +++ b/src/client.js @@ -153,6 +153,8 @@ export function qtapClientHead () { } writeConsoleError(str); }); + + // TODO: Add window.addEventListener('unhandledrejection') } export function qtapClientBody () { diff --git a/src/reporters.js b/src/reporters.js index e5da04c..4e8b4ba 100644 --- a/src/reporters.js +++ b/src/reporters.js @@ -83,7 +83,7 @@ function plain (eventbus) { console.log(util.styleText('grey', `[${event.clientId}]`) + ' Running...'); }); eventbus.on('consoleerror', (event) => { - console.log(util.styleText('grey', `[${event.clientId}]`) + ' Console:\n' + event.message); + console.log(util.styleText('grey', `[${event.clientId}]`) + ' Console:\n' + util.styleText('yellow', event.message)); }); eventbus.on('bail', (event) => { console.log(util.styleText('grey', `[${event.clientId}]`) + ` Error! ${event.reason}`);