test runner for component

[matthewmueller]

i'm building out the testing pipeline for components and it'd be great to discuss and pull in matthewmueller/component-test.

Part of the reason angular became popular was because it came with an easy-to-use opinionated test runner (testacular/karma)

[jonathanong]

i'm +1. i know TJ's life long dream is to make a component for everything like golang, but until then, it would be great included (since that'll probably mean a rewrite). we might have to rebrand as component.js or component.web

pretty sure one of our goals is to move towards small, reusable, and high quality modules, and you need tests for that. an easy way to create and run tests is important part of that goal. i know i didn't bother to make tests for a lot of my stuff because it was too difficult before component test

[dominicbarnes]

+1

I didn't hear about component-test until today, and I'm likely to start using it right away. I already use mocha, and I would love a way to automate test envs like this. I think getting something like this into core would be a great idea.

[ianstormtaylor]

i do really like the idea of component making testing dead simple, so easy that it just makes the "right" way to do something (tests, deps, small modules, etc.) the easiest way to do it anyways

[tj]

it would be nice to unify but it seems like something that could be really annoying to maintain, especially compatibility-wise, but I guess seeing as we have a little army of people going now it could happen haha

[jonathanong]

plus, i think it's pretty good where it is now.

[jonathanong]

i'm not sure where this belongs in the spec, though. @matthewmueller maybe you want to add some rules or something? dunno.

[bmcmahen]

+1 for making adding tests as easy as possible.

[matthewmueller]

@jonathanong @visionmedia I could be missing part of the vision, but to me the registry is the big piece that can be unified across languages along with client actions like install, search, etc.

I don't think that the component should try to accommodate all languages, I think component should be the home for client-side javascript libraries. That's a big enough job.

The build, the testing, the manifests (ie. component.json or package.json) will all be language specific and should probably be under a different project.

[tj]

the thing with the registry is that every ecosystem re-invents the same thing over and over, usually in ways that don't scale too well, but they all effectively do the same thing - making it work for everything would just be a tiny bit more work than building one for component. but yup the client side manifest stuff is definitely completely separate, you won't have go using package.json or anything like that, just whatever it would be (go.json for example) and a little client-side shim that formats it for the global registry

[jonathanong]

yeah i'm cool with this as long as the repo is in the component org. i think it would make a great workflow. not sure how to add this in the spec though.

[ianstormtaylor]

+1 as part of component, testing seems super good to have in core, and then yeah i think it doesn't impact the registry vision i think cuz thats just for packages

but i think that the scope for component-test should be more limited, because @visionmedia's right that we don't want to be maintaining that kind of stuff really.

i don't quite get what it does for travis, or how that relates to component test at all. that sounds like a separate module. saucelabs sounds kinda cool, but i also dont get how that works with creds. having aliases for all the browsers seems like overkill. it should just test in the default browser imo. and having things like seamonkey seems way too edge case. i dont even know what maxthon is :)

honestly, component test would solve 99% of my testing by just running mocha with phantom on my test/ folder. (and if phantoms not installed fallback to browser)

[ianstormtaylor]

component test                  # test in phantom (or default browser)
component test --browser        # tests in default browser
component test --browser safari # tests in safari

and then component saucelabs and component travis can be separate plugins that dont need to be in core

[jonathanong]

i'm cool with that. this is what most people would be doing anyways.

i'd prefer if the api were still component test browser safari or just component test safari.

[ianstormtaylor]

im fine with component test safari adding browser in is weird cuz then its a flag without the normal flag syntax

[matthewmueller]

I'm fine with separating them out too. I think @yields has a better saucelabs integration with gravy anyway. Last I checked, there's still a few fixes I made that need to be integrated though (like JS errors and reporting on ZSH is broken). I'm also for removing the crazy browsers and the syntax change.

I do feel that getting some sort of multi-browser continuous integration (travis + sauce, or whatever) is the next big step for component. If we want anyone to take component seriously, we need to have more comprehensive testing - half the components we write don't even work on IE9.

The reason jQuery kills it is because it's a drop in script that works everywhere. They owe that to their crazy test suite + test swarm. Their testing is pretty complicated though and I think we can do better.

Also, anyone know how to speed up phantom's initialization? For me it's faster to just open the browser and test than to use phantom. If the default is going to be phantom i'd like it to be faster.

[yields]

tbh gravy was supposed to be a temporary hack until component test was more flexible, i wanted to control index.html which wasn't possible (unless i'm missing something ?)

i'm +1 because adding saucelabs tests and tests in general is kinda PITA, and making this easy will be better for component, it will improve all components, especially small utilities.

i don't think it should be in spec though, maybe add a wiki page about testing and add an install link ? maybe add a paragraph and a link to "building better components", "getting started" etc...

[bmcmahen]

I agree with @matthewmueller that multi-browser testing should be a bigger part of Component. The benefit of using libs like underscore, jquery, etc., is that you generally just know that they work with ie8+... whereas with Component, you kind of need to hope and pray.

[ianstormtaylor]

ah, i forgot one more thing that would solve a problem for me is if component test could do component test node in addition to component test browser or component test phantom for when you want to quickly create cross-environment components. now that component/assert is API compatible with node's the workflow is almost perfect

been trying out @matthewmueller's plugin tho and it is really nice to use, i really +1 a slightly stripped down version added to core.

[tj]

it still feels non-coreish to me (haven't actually tried the lib), but just conceptually it's weird, that's like advocating the use of Mocha only in node core or something. I'm totally for having a suite of well-documented and recommended tools, but it seems like a leaky thing to consider core IMO, more of a documentation issue than anything else

[jonathanong]

we could change it to component mocha since that's what it essentially is - a mocha command for the browser.

although it's opinionated, i think it'll be helpful for a lot of components, especially all the ones we create. it's not like everyone has to use it.

[johntron]

tbh, I probably won't use a test runner that can't easily run entire suites (I'm trying to build this with builder2.js) in multiple browsers. For a component, a single test is fine, but for web applications, you really need both.

I think right now component focuses on curators of high-quality components, but these two features would be huge bonuses for pro's looking for something to build an app with. I use Karma with Mocha, Chai, and Sinon (trying to get Yadda working) - this was a ton of work to setup, so it'd be nice to have some of this setup for me.

I'm trying to wrap my head around builder2.js, and I'd like to help with the above.

[tj]

no doubt that it'll be useful, I totally agree, but just because it's not in core doesn't make it not awesome/useful, especially when installing it is as easy as npm install -g component-test (playing devil's advocate to keep things clean haha)

[johntron]

Bonuses to including in core:

• Higher adoption of component
• No installation
• "Supported" (high-adoption) methodology

Downsides:

• Distraction
• Discourages third-party testing infrastructure
• "Supported" (obligations) methodology

IMO, not including in core vs. including is a pretty minor decision, because we could always just allude to a non-core component in core's README. This decision really seems like finding a balance between level of commitment and effort: do we officially support robust testing, or spend our energy on other things.

Is there a clear majority as far as testing frameworks? If we wanted to include basic, extensible testing in core, would choosing Mocha be the right decision?

[jonathanong]

Is there a clear majority as far as testing frameworks? If we wanted to include basic, extensible testing in core, would choosing Mocha be the right decision?

i think mocha is the right decision since it's what most of us use. but i don't want people to ask for more features or make suggestions of their own idea of a "testing" framework, which is why i want to call it component mocha. that would immediately eliminate any potential requests for supporting other testing frameworks.

an extensible testing framework for component should be a separate module from component. we should keep the one included as simple as possible. if we use component mocha then we won't have any conflicts with a bigger, better component test

[johntron]

i think mocha is the right decision since it's what most of us use. but i don't want people to ask for more features or make suggestions of their own idea of a "testing" framework, which is why i want to call it component mocha. that would immediately eliminate any potential requests for supporting other testing frameworks.

Good point, and I agree on Mocha. Maybe we could start with npm install -g component-mocha and consider graduating it to component-test and consider incorporating into core when/if the community gets behind it? Maybe I'm wrong, but choosing an assertion or mocking library seems to be a matter of preference, but choosing a test runner seems to be a matter of convenience.

On that note, here's another idea: include a test runner (only) in core, and recommend some components that implement assertions (Mocha), mocking (Sinon), etc. As for test runners, while I was impressed at how easy Karma was to setup, I think @matthewmueller's component-test would be a better investment. It's more easily extended and Matthew's a big part of component already, so we wouldn't have to deal with Karma's politics.