Simulating real browser behaviour

[krychu]

Hey,

A real browser will download not only the content of the page but also all js scripts, css, images etc. I was wondering how this is supported in locust. All those extra resources can easily amount for most of the web traffic but also server's ability to handle simultaneous users. It seems to me, however, that locust's (or requests') get gets only the main content. Is this the case? If so, how accurate are the locust measures when predicting real-life performance of your setup?

Thanks,

[heyman]

Hi!

Scaling up the serving of static files is very simple compared to scaling up a dynamic website. Also, high traffic websites often use some kind of CDN for static files. If not a CDN, a load balancer with a couple of nginx machines should be able to handle A LOT of traffic.

Locust focuses on simulating the traffic from dynamic requests. You could make Locust load static files by parsing responses and then issuing new calls to client.get(), but since the serving of static content should be totally separated from dynamic requests, I would recommend to have a separate load test that tests static files.

[krychu]

Hey,

Techniques of scaling up the serving of static files or dynamic components have nothing to do with the task of load testing. More, some of the linked resources in HTML might require computation on the server side so could be considered dynamic. More, serving static content will consume CPU and bandwidth which can have significant effect on the server's efficiency in relation to dynamic content, so it would appear that you can't simply focus on dynamic parts without considering static parts. Also, resources such as CDN or a couple of nginx machines don't come for free so their availability should not be taken by granted. I think these are some strong assumptions you make about how people should build their apps or what they can afford.

Perhaps this is something you will want to consider. It would be fantastic to see such functionality in locust. Thanks for taking time to respond, I appreciate it.

[mthurlin]

More, some of the linked resources in HTML might require computation on the server side so could be considered dynamic

Then you should write locust tasks for them.

There are a multitude of other tools that are suitable for loadtesting static content, and you can run them at the same time as you run a locust test. The strength of Locust is to fetch dynamic content and act on the response.

However, using BeautifulSoup to fetch all images/css/js from every HTML-response in your Locust test and download those should be fairly trivial to implement yourself.

[ghost]

@mthurlin We also want to load test something that will be hit by real browsers, so we need this functionality too. I just researched on this, and it would require implementing all a, img, link and other tags that can potentially reference things that a browser would download. While it isn't hard to do, it is certainly quite a bit of work and not something that can be done in 5 minutes and it's also easy to miss something. Most advice places like stackoverflow recommend simply using selenium instead, but of course that has probably way too much overhead for use in a load test.

Therefore, this seems to be a real gap of locust and I recommend reopening this ticket and reconsidering putting something in like this eventually, even if it's just a beautifulsoup script as some sort of contrib resource which people can reuse.

[ReallyNotARussianSpy]

This page has a lot of helpful info for doing this and more
https://tag1consulting.com/blog/drupal-loadtest-locust

[mektebi]

Hi all,

Any update?