Run locust programmatically #805

ps-george · 2018-05-28T10:09:32Z

Fixes #222, fewer changes than #799

ps-george · 2018-05-28T10:15:06Z

CI will pass after #804 has been merged.

…rammatic

codecov · 2018-05-28T16:23:21Z

Codecov Report

Merging #805 into master will increase coverage by 2.46%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #805      +/-   ##
==========================================
+ Coverage   66.55%   69.01%   +2.46%     
==========================================
  Files          14       14              
  Lines        1438     1488      +50     
  Branches      226      236      +10     
==========================================
+ Hits          957     1027      +70     
+ Misses        430      394      -36     
- Partials       51       67      +16

Impacted Files	Coverage Δ
locust/main.py	`45.83% <66.66%> (+19.66%)`	⬆️
locust/log.py	`81.25% <0%> (+15.62%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a155ed...6082fde. Read the comment docs.

ps-george · 2018-08-01T09:48:12Z

Hi, could someone tell me what is delaying this PR or give a detailed breakdown of exactly what I need to do to get this merged? @heyman

vamega · 2018-08-01T19:07:09Z

So I merged this into my own branch, and I think there's a few things that need to be changed before this should be merged.

This PR does a great job in providing an alternative interface to launching locust, but I think there is still some work to be done in accessing the results programatically.

Inovkust shows a way of allowing access to the results in process rather than having that be controlled through an option in the command line.

I'll look into it more to see if there's a good way to combine the benefits of invokust with the benefits of this PR.

ps-george · 2018-08-09T09:02:43Z

@vamega Ok let me know how you get on and if you need any help. Keen to have this functionality merged.

This allows programmatic usage to collect stats from objects after the locust run has completed.

joaonc · 2018-12-13T00:41:24Z

Would love to see this get merged! @ps-george would you mind fixing the conflicts and contact the project owner(s) to have it merged?

ps-george · 2018-12-17T12:10:58Z

@joaonc Me too, conflicts resolved.

It's an important but also quite big change, I'm not sure the project owners have the time to properly review.

I'm sure if we get some thumbs up and support for this PR, they will be more willing to look into it and help us merge this.

aldenpeterson-wf · 2018-12-17T15:50:52Z

locust/main.py

+
+
+def create_options(
+        locustfile='locustfile',


I wonder if there's a more reader friendly way to do this - or if the general use case will be ok with a massive list of kwargs here.

Unsure, open to any suggestions.

Essentially, with the CLI tool, you're basically doing this same thing - loads of default arguments but the ability to change all of them via CLI arguments. But have put defaults in so unless the user who's calling this code only needs to put in the ones they're changing.

aldenpeterson-wf · 2018-12-17T15:51:08Z

locust/main.py

@@ -210,7 +210,7 @@ def parse_options():
       action='store_true',
       dest='only_summary',
       default=False,
-       help='Only print the summary stats'
+       help="Only print the summary stats"


super nitpicky but why this change?

Looking back, I have no idea. I think it was to be consist with some code elsewhere.

aldenpeterson-wf · 2018-12-17T15:52:46Z

examples/run_programmatically.py

+    max_wait = 5000
+    task_set = UserTasks
+
+options = create_options(locust_classes = [WebsiteUser])


I'd add a comment here saying that all command line options are also available as kwargs to this function

aldenpeterson-wf · 2018-12-17T15:55:00Z

locust/main.py

-        print("Locust %s" % (version,))
+        version_text = "Locust %s" % (version,)
+        print(version_text)
+        if not cli_mode:


I'm trying to think of a valid use case for this type of function call from a non-cli mode - is there a reason you'd ever intentionally hit this codepath programmatically?

The way I was finding use from this was being able to import locust in Python and being able to interact with all the statistics directly on the class, rather than reading from the CLI interface.

aldenpeterson-wf · 2018-12-17T16:42:30Z

docs/running-locust-programmatically.rst

+.. _running-locust-programmatically:
+
+===============================
+Running Locust programmatically


Nit: you could link to the example here

aldenpeterson-wf · 2018-12-17T16:53:40Z

This looks pretty good to me, hopefully a few other folks can chime in - most of my comments are pretty minor and documentation related.

I think this is a great add to Locust. There's still likely opportunities to ensure that the results can be reported meaningfully but this is a good start (I think you could just access them directly at runners.locust_runner.request_stats right now though).

SpencerPinegar · 2018-12-17T17:40:36Z

locust/main.py

+        list_commands=False):
+    """Create options objects for passing to `run_locust` when running Locust programmatically.
+
+    Keyword Arguments:


I like how all of the options are listed here, but could the most important ones be isolated out and set as kwargs in run_locust call with appropriate defaults. Other options could be grouped by their functionality like no-web options, distributed options, etc..

Good comment; can probably apply some functional programming concepts here to reduce the complexity.

Ok, will group these up with comments explaining the grouping.

SpencerPinegar

Looks good, I want the options to be grouped by functionality so each parameters purpose is more explicit. Functionality groupings could look something like:
Undistributed options,
distributed options,
no-web options,
etc.

SpencerPinegar · 2018-12-17T17:47:58Z

One other thing to note, if locust is run pragmatically by the multiprocessing module the response times are not accurate (they are largely inflated) - when you run them in the subprocess module the response times appear to be accurate - if this could be noted explicitly that would help avoid a lot of confusion.
I think this slowdown is related to the process management in the multiprocessing module.

joaonc · 2018-12-17T19:10:26Z

@SpencerPinegar RE response times, you think issue #930 would take care of it?

SpencerPinegar · 2018-12-17T19:31:14Z

While I cannot say definitively, I think that this fix works great for all clients using http requests (and it seems it would be the most accurate way to benchmark) but this wont work for non http-clients so using it as the way to monitor all response times isn't the options (unless I am missing something - kinda hope I am because it seems accurate).

That being said for most http clients it would probably fix the multiprocessing issue.

cyberw · 2019-11-01T10:21:36Z

@ps-george Hi! sorry for the lack of "official" response on this PR. @heyman what is your opinion on this? The diff is reasonably small at least...

heyman · 2019-11-01T12:07:36Z

I think it's a good idea to be able to run Locust programatically (and be able to use it more as a a lib). I'm thinking that the best way to do this would be to expose a larger part of Locust's internal code and classes (runners, stats, etc.) as part of the public API. However, this would definitely require some refactoring/re-work of some of Locust's code. Both to make sure we expose a good API, and also because once we add something to the public API, we want to try our best to not do breaking changes. I've been thinking about this quite a bit, but haven't taken the time to concretize the ideas into a Github issue yet.

For this reason, I'm a bit hesitant to merge this pull request. I understand that this PR aims to introduce this feature with the least obtrusive changes as possible. That is usually a great goal for PRs to have, but in this case I think it might be better to take a step back and do some re-factoring of the existing code instead.

cyberw · 2019-11-18T08:22:40Z

@heyman Ok to close/decline?

g.punter added 6 commits May 28, 2018 10:53

Adding docs for programmatic locust.

9587957

Allow locust to be run programmatically

ba47245

Adding example of programmatic functionality

c4334f5

Adding tests for programmatic locust

159b870

move version to top of file to fix cannot import name version error

5ddff28

Placing locustfile for testing in test/

54944f4

Merge branch 'master' of https://github.com/locustio/locust into prog…

f06d833

…rammatic

g.punter added 6 commits May 29, 2018 14:23

Remove parser split

bc7d517

Remove parser, fix typos in option list

d391fa9

Fixing : instead of -- typo

85420c5

import core before main

a66b7e1

Remove refactoring and is_taskset/load_taskset functions

9e920b7

Remove import TaskSet

28f3a82

SpencerPinegar mentioned this pull request Jul 19, 2018

Assign Locust Behavior Programmatically -- Code Example #847

Closed

vamega added a commit to vamega/locust that referenced this pull request Jul 27, 2018

Add programmatic way to run locust PR locustio#805

edd9223

vamega added a commit to vamega/locust that referenced this pull request Jul 31, 2018

Add programmatic way to run locust PR locustio#805

f5da190

Do not terminate program after main greenlet exits. (#18)

0e0affd

This allows programmatic usage to collect stats from objects after the locust run has completed.

Merge branch 'master' into programmatic

6082fde

aldenpeterson-wf reviewed Dec 17, 2018

View reviewed changes

SpencerPinegar reviewed Dec 17, 2018

View reviewed changes

SpencerPinegar approved these changes Dec 17, 2018

View reviewed changes

cyberw mentioned this pull request Nov 10, 2019

How to run test programatically #222

Closed

cyberw closed this Nov 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run locust programmatically #805

Run locust programmatically #805

ps-george commented May 28, 2018

ps-george commented May 28, 2018

codecov bot commented May 28, 2018 •

edited

Loading

ps-george commented Aug 1, 2018 •

edited

Loading

vamega commented Aug 1, 2018

ps-george commented Aug 9, 2018

joaonc commented Dec 13, 2018

ps-george commented Dec 17, 2018

aldenpeterson-wf Dec 17, 2018

ps-george Oct 12, 2019

aldenpeterson-wf Dec 17, 2018

ps-george Oct 12, 2019

aldenpeterson-wf Dec 17, 2018

aldenpeterson-wf Dec 17, 2018

ps-george Oct 12, 2019

aldenpeterson-wf Dec 17, 2018

aldenpeterson-wf commented Dec 17, 2018

SpencerPinegar Dec 17, 2018

ps-george Oct 12, 2019

ps-george Oct 12, 2019

SpencerPinegar left a comment •

edited

Loading

SpencerPinegar commented Dec 17, 2018 •

edited

Loading

joaonc commented Dec 17, 2018

SpencerPinegar commented Dec 17, 2018

cyberw commented Nov 1, 2019

heyman commented Nov 1, 2019

cyberw commented Nov 18, 2019

Run locust programmatically #805

Run locust programmatically #805

Conversation

ps-george commented May 28, 2018

ps-george commented May 28, 2018

codecov bot commented May 28, 2018 • edited Loading

Codecov Report

ps-george commented Aug 1, 2018 • edited Loading

vamega commented Aug 1, 2018

ps-george commented Aug 9, 2018

joaonc commented Dec 13, 2018

ps-george commented Dec 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aldenpeterson-wf commented Dec 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SpencerPinegar left a comment • edited Loading

Choose a reason for hiding this comment

SpencerPinegar commented Dec 17, 2018 • edited Loading

joaonc commented Dec 17, 2018

SpencerPinegar commented Dec 17, 2018

cyberw commented Nov 1, 2019

heyman commented Nov 1, 2019

cyberw commented Nov 18, 2019

codecov bot commented May 28, 2018 •

edited

Loading

ps-george commented Aug 1, 2018 •

edited

Loading

SpencerPinegar left a comment •

edited

Loading

SpencerPinegar commented Dec 17, 2018 •

edited

Loading