Stats get corrupted when the number of swarm users reaches the objective

[jfacorro]

The following graph was generated with Rickshaw by making periodic requests to http://locust-url/stats/requests/csv. The spike happens when the total amount of users swarmed is reached, in this case 2000.

It seems that stats are reseted in an inconsistent way which causes the huge change in the average value.

[heyman]

This is likely due to the stats getting reset when all users has been spawned, in combination with the fact that the average response time is the total average of all requests made. So when the stats are reset the average response increases dramatically due to the fact that all the requests made during the ramp up phase - which probably got a response fast since there were less load on the system - are suddenly no longer accounted for in the average calculation.

For graphing the average response time over time, it would probably make more sense to use the average response time during the last X seconds (similar to Locust's "current reqs/s" which shows the RPS for the last 12 seconds). Currently this is not tracked in Locust, but could definitely be implemented, and it's probably a nice thing to have

[unbalancedparentheses]

@heyman Do you think it would be possible to resolve this inside locust? I do not think this should be resolved by the graphing code. Locust should provide correct stats. If the ramp up phase throws inadecuate response times, wouldn't it be better to reset all the stats, even the response times during the ramp up?

Thanks in advance and keep up the good work :)

[heyman]

@unbalancedparentheses The stats are not incorrect, and all the stats are getting reset when the ramp up is complete (that's why the average suddenly jumps up). It's rather a matter of total average response time not being very suitable to be graphed over time.

I would however be open to adding the current average response time to Locust, which would show the average response time during the last X seconds.

[jfacorro]

@heyman Thanks for the reply. What would be a good metric to graph over time? Median response time presents the same problem. I guess the only data that could offer a qualitative graph of how good the server is behaving is the percentile one...