Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure percentage being reported incorrectly #1006

Closed
DanielRussellkt opened this issue Apr 29, 2019 · 6 comments
Closed

Failure percentage being reported incorrectly #1006

DanielRussellkt opened this issue Apr 29, 2019 · 6 comments

Comments

@DanielRussellkt
Copy link

Description of issue

When running tests in locust 0.11.0, fails increment the total number of requests in addition to the number of fails. The percentage fails in the top right of the UI seems to use the old formula of:
Failure rate = (Number of fails/(Number of requests + number of fails))*100

This means that, when running a test that completely fails all requests (say a test that gets 400s exclusively), the failure percentage is just calculated completely wrong (50%).

I have such a test, that worked in version 0.9.0, and now, due to this functionality in 0.11.0, it is reporting an incorrect failure rate.

Expected behavior

The failure rate is calculated from the number of requests and number of fails alone, and does not add them (because failures are included in the request metrics now)

Actual behavior

Every fail is effectively treated as a successful request in the Failure rate in the top right, as well as a failure, making the failure rate seem much lower than it is

Environment settings

  • OS: Mac OSX 10.14.4
  • Python version: 3.7
  • Locust version: 0.11.0

Steps to reproduce (for bug reports)

from locust import HttpLocust, TaskSet, task
import json, random, uuid, os, csv

headers = {"Authorization": os.environ['API_KEY']}

class MyTaskSet(TaskSet):

    @task()
    def clearscoreLegacyRequest(self):
        with self.client.post("/url-endpoint", json={"someField": "someInvalidValue"}, headers=headers, catch_response=True) as response:
            if response.status_code != 200:
                print(response.status_code)
                response.failure("Got unexpected response: " + str(response.status_code) + " Error: " + str(response.text))

class MyLocust(HttpLocust):
    task_set = MyTaskSet
    min_wait = 0
    max_wait = 2500
    host="https://url-that-fails.io"

The above pseudocode is exactly what we are doing (with business-related endpoints etc replaced), and provides an accurate failure rate on 0.9.0, but an inaccurate one on 0.11.0. It aims to load test endpoints returning 400s.

The below screenshot shows what I mean; 116 fails come through as 116 fails and 116 requests, but the top right figure displays 50% failure rate rather than the correct 100%
Screenshot 2019-04-29 at 16 24 07

I'm hoping this is simply a calculation fix for that bit of UI, and this erroneous calculation doesn't extend to other things, like counting fails twice in the RPS calculations or graphs.

@afreepenguin
Copy link

I can also confirm this issue.

@aldenpeterson-wf
Copy link
Contributor

I think this will fix this? #994

@cgoldberg we should probably cut a release of that, it's an annoying UI bug that has bothered me a while too.

@rahim04a4
Copy link

I verified latest code, still issue not resolved. It showing 50%

@grganzaroli
Copy link

This issue also happens on no-web mode, but only for each individual endpoint. The total error % shows the correct value.

@cyberw
Copy link
Collaborator

cyberw commented Sep 5, 2019

This will probably be fixed by #1075, once it gets merged.

@heyman
Copy link
Member

heyman commented Oct 21, 2019

This should be fixed now.

@heyman heyman closed this as completed Oct 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants