Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QA Automation test task - mass user registration load test #5199

Closed
wants to merge 20 commits into from
Closed
Changes from 10 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
fc49bfb
Add mass register load test
inpv Oct 31, 2022
efe1741
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Oct 31, 2022
86241a5
- Refactor code for the nightly cvat-sdk build's models
inpv Oct 31, 2022
5f3371d
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Nov 1, 2022
60b9bcc
- Recreate fixture data to get an admin user (fixtures unsupported by…
inpv Nov 1, 2022
1353e0d
Merge remote-tracking branch 'origin/inpv_test_task_mass_user_registe…
inpv Nov 1, 2022
eaefeab
Restructure the import section according to isort
inpv Nov 1, 2022
f38de4b
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Nov 2, 2022
fc399e8
- Improve username generation (more randomness)
inpv Nov 2, 2022
8794bc2
- Disable the Pylint unnecessary lambda warning
inpv Nov 2, 2022
558902c
- Simplify config generation
inpv Nov 4, 2022
f0277c3
- Add faker to testing requirements.txt
inpv Nov 4, 2022
4153ce8
- Remove the response status check suppression
inpv Nov 4, 2022
3efa9a9
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Nov 4, 2022
e467394
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Nov 4, 2022
0f20e51
Add new PyAV dependencies with FFMPEG5
inpv Nov 4, 2022
a526267
Add explicit ApiClient exception raise
inpv Nov 4, 2022
9ac5af1
Update Werkzeug and click dependencies for locust
inpv Nov 4, 2022
e810aa9
- Revert Werkzeug and click dependencies for locust
inpv Nov 5, 2022
60c3438
Merge branch 'opencv:develop' into inpv_test_task_mass_user_register
inpv Nov 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions tests/python/rest_api/test_load_register.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# pylint: disable=unnecessary-lambda
import json
import os
import random
import secrets
from pprint import pprint

import gevent
import locust
from cvat_sdk.api_client import ApiClient, Configuration, exceptions
from cvat_sdk.api_client.models import RegisterSerializerExRequest
from faker import Faker
from locust.env import Environment
from locust.stats import stats_history, stats_printer

from shared.utils.config import ASSETS_DIR, BASE_URL, USER_PASS


class AdminUserNotFoundInDbException(Exception):
def __init__(self):
print("Can't find any admin user in the test DB")


class Container:
"""
We have to recreate the fixtures' data generation,
since locust doesn't support using pytest tests as tasks yet
"""

def __init__(self, data, key="id"):
self.raw_data = data
self.map_data = {obj[key]: obj for obj in data}

@property
def raw(self):
return self.raw_data

@property
def map(self):
return self.map_data

def __iter__(self):
return iter(self.raw_data)

def __len__(self):
return len(self.raw_data)

def __getitem__(self, key):
if isinstance(key, slice):
return self.raw_data[key]
return self.map_data[key]


class GetUser:
"""
We have to recreate the fixtures' data generation,
since locust doesn't support using pytest tests as tasks yet
"""

@staticmethod
def get_admin_user():
# getting the user from local db
with open(os.path.join(ASSETS_DIR, "users.json")) as f:
container = Container(json.load(f)["results"])

for user in container:
if user["is_superuser"] and user["is_active"]: # getting the admin user
return user["username"]

raise AdminUserNotFoundInDbException()


class PerformTask(locust.SequentialTaskSet):
@locust.task
def perform_random_user_register_task(self):
# Initializing Faker for data generation
fake = Faker()
username = None
first_name = fake.first_name()
last_name = fake.last_name()
username_type = bool(random.getrandbits(1))

if username_type == 0:
username = first_name + last_name
elif username_type == 1:
username = last_name + first_name

email = fake.ascii_email()

# Setting a random passwd
password_length = random.randrange(13, 24)
passwd = secrets.token_urlsafe(password_length)

# Setting the config for client auth
# Needs further dehardcoding
config = Configuration(
host=LoadUser.host, username=GetUser.get_admin_user(), password=USER_PASS
sizov-kirill marked this conversation as resolved.
Show resolved Hide resolved
)

# Launching the client with randomized data
with ApiClient(configuration=config) as api_client:
try:
# forming the request body
register_request = RegisterSerializerExRequest(
username=username,
password1=passwd,
password2=passwd,
email=email,
first_name=first_name,
last_name=last_name,
)
Comment on lines +46 to +53
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I run the test locally, I see some requests fail with 400, but the test doesn't fail. I think we should implement this test in such a way that we don't have this error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to catch the request exceptions and make the test fail upon parsing a non-200 response, or to suppress errors when sending the request altogether?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ended up doing both

Copy link
Contributor

@sizov-kirill sizov-kirill Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion the test should fail if server return non-200 responses.
We are trying to check the average user registration time, but how can we trust this number if some (or all) of our registrations fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, thought so too. Corrected as needed.

# sending the request
(data) = api_client.auth_api.create_register(register_request)
pprint(data)
sizov-kirill marked this conversation as resolved.
Show resolved Hide resolved
except exceptions.ApiException as e:
print("Exception when calling AuthApi.create_register: %s\n" % e)


class LoadUser(locust.HttpUser):

tasks = [PerformTask] # a list of tasks to perform
wait_time = locust.between(1, 3) # how long to wait before a single task is performed
host = BASE_URL


def test_load_register():
# setup Environment and Runner
env = Environment(user_classes=[LoadUser])
env.create_local_runner()

# start a greenlet that periodically outputs the current stats
gevent.spawn(stats_printer(env.stats))

# start a greenlet that save current stats to history
gevent.spawn(stats_history, env.runner)

# start the test with params
env.runner.start(5000, spawn_rate=20)

# in 10 seconds stop the runner
gevent.spawn_later(10, lambda: env.runner.quit())
sizov-kirill marked this conversation as resolved.
Show resolved Hide resolved

# wait for the greenlets
env.runner.greenlet.join()

assert env.stats.total.avg_response_time < 5 # testing average response time
assert env.stats.total.num_failures == 0 # testing for 0 failures

# stop the web server for good measures
env.runner.quit()