Request: non-zero exit when running --no-web and errors found

[jpotter]

When running locust in a shell, i.e.:

./virtualenv/bin/locust --clients 50 --hatch-rate 5 --num-request 100 --no-web --host https://local.example.com:8443 --locustfile ./main_locust.py MainLocust

and the test finishes and there are are errors found:

[2014-05-09 19:12:50,293] dev/INFO/locust.runners: All locusts dead
[2014-05-09 19:12:50,293] dev/INFO/locust.main: Shutting down, bye..
...
Error report

occurences Error

--------------------------------------------------------------------------------------------...
37 GET api: /login/: "HTTPError('401 Client Error: UNAUTHORIZED',)"

It would be great if the exit code was non-zero, but currently:

% echo $?
0

This would allow us to add a command-line locust test as part of our build process -- we could halt a built if it failed to pass load testing.

Looking at the code, locust/main.py is presumably returning 0 on line 442:
main_greenlet.join()
shutdown(0)

I'm not sure if it's best to check global_stats.errors at this point, or if there's some other place that makes sense to do this?

Thanks!

-Jeff

[heyman]

Ah,

That's probably where you want to do it. The best way to reference the stats instance, would probably be using runners.locust_runner.stats, since the runner instance is likely to have a reference to the RequestStats instance, even if we get rid of the global_stats variable at some point.

I'd be happy to merge a pull request with this change.

[sanga]

Again chiming in with how we handle this in case it's of any use... It's very easy to write a new endpoint for the locust flask app. So what we did was just add a few new endpoints of the form /stats/requests/total_rps/is_greater_than/, have that handler calls the locust internal stats object, and the after the perf test run curl to that endpoint with whatever (configurable) limit and fail if curl returns HTTP 500 (i.e. if rps was < limit). In the same vein we added endpoints for error_rate and med_resp_time. It's a little more work than what you want, (but only a little :) and it's a lot more flexible I think.

Note: I still think your suggestion is a good one. Just mentioning our solution to the same problem as in case you'd find that of any use.

[jpotter]

Thanks, Tim -- seems like another useful thing to have (a web-based API for checking status). In our case, we'll want to fire up locust on the command line from Make, so I'll follow Jonatan's advice and do a pull request when I get to it.

[jpotter]

Pull request merged in, so closing this here...