[enhancement] Enable writing test results to file

[jnfingerle]

I'm trying to publish the tap test results on Jenkins using the TAP plugin. For this, it would be great, to have a configuration option to have the output of the different runs of testall to be written into separate files. The rest could be handled by tap-merge and the Jenkins TAP plugin.

[jnfingerle]

Just added a pull request with my temporary solution. It would be great to have this - or something similar - added to the official version.

[jnfingerle]

Added a new pull request with a different, better solution, see discussion at #31.

[kategengler]

I wonder if a testem config file per scenario is the right approach. It seems like the only thing that would vary between scenarios would be the report_file, right?

Adding the ability to pass the report file as an option to ember test could be an interesting approach. The reporter option is already available and passed to testem. Then we could either have the report file configurable per scenario for ember try:testall or it could be used in a build matrix like so:

ember try beta test --report-file beta.tap (where 'beta' is the scenario name)

[jnfingerle]

Well, for my scenario this would suffice. But the more general approach means people could do unexpected things, e. g. different browsers for each scenario, maybe? Different test runners (I can't imagine a real use case, but who knows).

That said, any solution that lets me use different report files would solve the problem I'm facing right now.

[kategengler]

In the scenario where you run

ember try beta test --report-file beta.tap

You could also set any other options that test accepts. To make this work for you right now (and until we sort how we want to handle it in ember-try), you could add a testem.<scenario name>.json for each of your scenarios and use a build matrix:

ember try <scenario name> test --config-file testem.<scenario name>.json

[jnfingerle]

Using this kind of build matrix won't work for me: I've got only one build job for all branches with different scenarios in different branches. Therefore I have to use the scenarios checked into each branch. From my point of view, this is one of the basic promises of ember-try: To run scenarios checked in with your code and not having to resort to some buildbot magic. Anyway, until there's a solution in ember-try I can and will use the solution from #32, so I'm set for the moment. But I'd rather lose our fork in the long run and change to a solution in the official versions.

I understand that you want to add a configuration option report_file (or some other name) to the scenarios and provide this in case of ember try:testall as a command line parameter to ember test. Now, this would solve my problem, but we'd have to wait for changes in ember test to materialize. Wouldn't it be possible to use #32, strip the testem magic for ember try and change it for ember try:testall to recognize (only) that proposed new configuration option? Than we'd have a working solution now and could switch to the better solution when the new command line parameter in ember test becomes available? I'd volunteer to change my PR so that it conforms to this, but only if it's useful to you.

[jnfingerle]

I'd like to get rid of my fork, so I'm having another look at this one. At the moment (ember-cli 1.13.8) ember test has a switch --config-file. If we could add a configuration option config_file to ember-try scenarios, that would be provided as a command line parameter to ember test, if present, my problem would be solved.

If this is the way you want to go, I'd be volunteering to code this.

[kategengler]

I've been working on a refactor the last few days to prep for handling several issues, including this one. Here are a few ideas I've had around this issue:

• A config-file option in ember-try.js for each scenario
• A test-command option in ember-try.js that would be used to override the default test command (you can pass whatever you like, and possibly we provide a variable for scenario name
• I've been looking into whether tap, etc have standards for multiple test runs like ember-try is doing. I'd love to report the end results in a standard format.

Personally I'm leaning towards the test-command option in ember-try.js. It seems like this provides the most flexibility for any options that need to be used. I also want to looking to better reporting for the end results of testall but that can certainly be a different issue.

Thoughts?

[jnfingerle]

First things first: Any of those ideas would help me to solve my problem.
Having said that ...

• A config-file option seems a pretty obvious choice, since this maps directly to the ember test --config-file switch. You could generalize this with an additional-args option.
• A test-command option feels a bit too general to me. It's more flexible, but it may make it too easy to shoot yourself in the foot. Still, this is only a gut feeling.
• Combining taps from different runs out of the box would be really great.

I, for my part, would prefer the "combining taps", followed by the additional-args option. On the other hand you're right: test-command would be the most general option.

[jnfingerle]

Is there anything new, especially regarding things I could do to help?

[jnfingerle]

This is now supported, see #32, comment on closing by @rwjblue