Better testing stack

[mjackson]

It's pretty clear to me at this point that jsdom isn't going to work. The support for the HTML5 history API appears to be incomplete, and, given the quirks of the various history implementations, I'd rather test in real browsers anyway.

Stuff we need to be able to do in a single integration test:

• Get reliable hashchange/popstate events when the URL changes
• Use window.history.pushState/window.history.replaceState
• Use window.location.replace
• Assign the value of window.location.href

Stuff we need to be able to do in a test suite:

• Run multiple integration tests without leaking state between them (see Better testing stack #5 (comment))

There are a few options out there, none of which I'm very familiar with. If anyone has any experience with any of these stacks, please chime in.

• karma
• webdriver.io (Selenium 2.0 bindings for node)
• intern (Doesn't seem very flexible, but could maybe work)
• zuul (Powered by electron)

We use karma to test React Router, so it would be great if we could stick with that. But it seems like I remember it complaining when we tried to change the URL (which is strange because I thought karma ran all your tests in an <iframe> ... ?)

[phated]

The library I was talking about on twitter - https://github.com/defunctzombie/zuul - it's my favorite runner, supporting mocha, jasmine, qunit frameworks, using the --electron flag will give you an electron instance to run your tests in and (I believe) gives you all of the stuff needed for your integration tests. Changing the location doesn't seem to affect the reporter, so that's a plus.

[phated]

I don't think you should use intern, I've been following it since release and it was promising but it is very inflexible and have heard nothing but complaints from anyone that tries to use it.

[phated]

Also, you can use your webpack config as part of zuul using https://www.npmjs.com/package/zuul-builder-webpack

[dignifiedquire]

I'm biased (current maintainer of Karma) but I will suggest to use Karma. There is an issue with changing urls when using Karma, which I'm currently investigating and hope to have a fix for soon (Ref: karma-runner/karma#1101) so after that you should be good to go

[mjackson]

Thanks for the input @dignifiedquire. The reason I'm questioning if karma would be a good fit here is because it seems like karma doesn't want me to ever refresh the page, which is exactly the kind of thing we need to do every so often in a thorough test suite for a history lib. :)

For example, we currently have a beforeEach hook that does window.location.href = '/'. This hook is designed to give each test a clean history slate to start with. It would be better if we had a brand new session (i.e. browser tab) to work with in each test, but I'm currently working within the constraints of jsdom's "global" mode.

I guess my question for you is, is this the kind of thing that is within karma's expectations? Or has karma loaded code in the browser that will screw up the test if we do a page refresh?

@phated Thanks for the link to zuul (and bonus points for the Ghostbusters reference!). I'll add it to the list of possible candidates and give it a whirl.

[ericclemmons]

@mjackson I started a similar search last week and Zuul made it to the top of my list as well, followed by Karma (mostly due to mindshare & my history with Angular).

Intern went through just the "R" of "R & D" before I decided to try out Zuul first.

I've subscribed to this thread so I can share my findings with the group as well. See you on the other side! :)

[christian-bromann]

As the maintainer of WebdriverIO I can confirm that all of the above mentioned requirements would work since you can execute any arbitrary JS using execute or executeAsync. However I think you should go with zuul as it runs your tests directly in the browser. This makes them execute a lot faster and you don't work in two different environments (browser & node context) which makes it less confusing.

[joeybaker]

I'm not sure if you've already settled on zuul, but https://github.com/hughsk/smokestack has served me well – I'd recommend taking a look.

[mjackson]

Thanks for the insight @christian-bromann. I still think that WebdriverIO could be a slightly better fit than zuul because it controls a client instance from the outside. It seems like this would be a huge benefit when testing a history lib because we can essentially start with a clean browser session with each test.

For example, if I pushState in an in-browser testing framework like karma/zuul I've already permanently modified certain aspects of window.history (like window.history.length) that cannot be undone unless I start a new browser session, so I'll end up with tests that leak state. For this reason, I'd like to do something like this (in mocha):

beforeEach(function (done) {
  client = webdriverio.remote(...); // Create a brand new client instance.
  client.init(done); // Start a new browser session.
});

afterEach(function (done) {
  client.end(done); // Destroy the browser session.
});

@christian-bromann Is there anything about creating and destroying multiple client instances in WebdriverIO that would prevent me from doing this? Or is this totally unnecessary for some reason?

[mjackson]

@joeybaker Thanks for the recommendation, but unless I'm mistaken smokestack follows more or less the same process model as karma/zuul (runs everything in the same tab), right?

[joeybaker]

Yea, it does. :(

A possible solution would be to separate your tests into many files and launch a new smokestack instance for each file. That's basically what webdriver will do for you, but might make for a complicated test launch script.

[christian-bromann]

It seems like this would be a huge benefit when testing a history lib because we can essentially start with a clean browser session with each test.

True.

Is there anything about creating and destroying multiple client instances in WebdriverIO that would prevent me from doing this? Or is this totally unnecessary for some reason?

You should use the wdio runner. Like in protractor you define a set of capabilities + the test spec that the capability should test. You should also be able to recreate the Selenium session during a wdio test run.

[mjackson]

Man, spent most of today trying to put together a branch that uses WebdriverIO for testing, but it's a lot of work. Basically, the flow ended up like this:

1. Install & boot selenium
2. Bundle my library + tests to a test/bundle.js file
3. Boot an HTTP server that serves test/index.html. test/index.html has a <script> tag that includes test/bundle.js
4. From my test suite, use WebdriverIO's execute method to execute test methods on the client

But it feels clunky. Maybe a single process suite is the way to go...

[joeybaker]

Whatever works is good, but for what it's worth, the reason I suggested smokestack is because eveytime I touch selenium I get the same reaction.

You might look into saucelabs. It's free for OSS

[mjackson]

Well, as of 3dde5ce our testing story is to use zuul to run our tests locally, on sauce labs, and even on Travis. It was really easy to get everything setup, so that's what we'll go with for now.

If someone gets some time/ambition to dig into this further in the future, we can reopen this ticket. But I'm closing for now.