Ability to run test_start on workers.

[Zooce]

Is your feature request related to a problem? Please describe.

I have a load test where some setup is needed, but I really only need to do it once to set some global variables, that all spawned users to use.

Previously this was done with the HttpLocust.setup method. Unfortunately, now there is the test_start functionality which only runs on the master.

Describe the solution you'd like

Allow the test_start functionality to run on workers.

Describe alternatives you've considered

I can use the HttpUser.on_start method, but it results in ever single user that is spawned making a request (part of the work that I have to do in the setup), and it's not ideal.

[Zooce]

I seem to have found a workaround for this -- bypassing the logic in HttpUser.on_start if the global variables I'm trying to set have already been set by another spawned user.

class MyUser(HttpUser):
    def on_start(self):
        global SOME_GLOBAL
        if not SOME_GLOBAL:
            # do the setup work

[cyberw]

The setup method was removed because it was running in a bit of a strange place - people ended up getting confused because it was only being run on the first user, so if you set an instance property it would only be available on the FIRST instance of the User. Sometimes it was also run before the instance was even constructed (when there was no .client or anything). Maybe we should reintroduce it in some form.

Another way to work around it is like this:

class MyUser(User):
    _first_instance = True

    def __init__(self, parent):
        super().__init__(parent)
        if MyUser._first_instance:
            MyUser._first_instance = False
            # do the setup work

Tbh, I'm not sure what the difference between on_start and __init__ really is, since on_start is called immediately after construction. We do catch any InterruptTaskSet thrown in on_start, but I dont really see any use case for throwing that there anyway..

[cyberw]

Also, if what you want it just to do some general one-off initialization (not in the context of a User), you can do it using the init event, which IS triggered on all locust processes:

from locust import events
from locust.runners import MasterRunner
...
@events.init.add_listener
def on_locust_init(environment, **kwargs):
    if not isinstance(environment.runner, MasterRunner):
        # do the setup work

[heyman]

We could consider adding an additional event that is triggered on the worker nodes when they start spawning users. We would have to consider if it should trigger for new worker nodes that connects while a test is running (I guess it should?). Though I'm not sure it's necessary, since there are work-arounds like the one by @Zooce above.

The setup method was removed for a good reason, and I don't think it makes sense to re-add it.

[Zooce]

Removing the setup method was probably a good idea, I agree.

@cyberw -- I didn't realize there was an init even listener I could add. That's essentially what I'm looking for. I just looked at the documentation and found https://docs.locust.io/en/stable/extending-locust.html -- however, from that description/use case it wasn't clear to me that it's fired before everything else (and for every process). Note that I'm very new to locust and not a web developer.

Maybe a simple documentation update is all that is needed?

[cyberw]

Removing the setup method was probably a good idea, I agree.

@cyberw -- I didn't realize there was an init even listener I could add. That's essentially what I'm looking for. I just looked at the documentation and found https://docs.locust.io/en/stable/extending-locust.html -- however, from that description/use case it wasn't clear to me that it's fired before everything else (and for every process). Note that I'm very new to locust and not a web developer.

Maybe a simple documentation update is all that is needed?

I think you are right, the documentation is not clear and doesnt indicate that the init event is very useful for other things than adding web routes. Could I bother you to write something and make a PR?

[Zooce]

Certainly! I'm going to use that code you showed a couple of comments ago (if that's okay with you) since I immediately understood the usage after reading it -- it's a good example of how to use the init event.

I'll have the PR up soon.

[robbrit]

Is there a way to access the HTTP client from a listener? I want to be able to make a few HTTP requests to the target during the test_start event, but since that's not within the User context I can't do it.

The global flag setup works fine for single-node setup but that's not ideal long term if we want to scale up.

To elaborate on the use-case: I'm stress testing an API and before hammering it, I request a JWT from the auth server. I can hard-code a JWT into the Locust test, but that's a bit brittle if the JWT secret or the contents ever change.

[cyberw]

Is there a way to access the HTTP client from a listener? I want to be able to make a few HTTP requests to the target during the test_start event, but since that's not within the User context I can't do it.

In short, no, not from the test_start/stop events. But there is nothing stopping you from making an HTTP request ”manually”, using requests directly. It just wont be logged.

[kamalk-groww]

@cyberw If I use requests inside test_start and then spawn multiple workers then I get the following error and load test never starts

objc[9322]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[9323]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[9322]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[9323]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[2024-03-16 22:13:10,686] GM-C02FM31JMD6M/INFO/locust.runners: Spawning is complete and report waittime is expired, but not all reports received from workers: {"LoadTest": 0} (0 total users)
[2024-03-16 22:13:13,710] GM-C02FM31JMD6M/INFO/locust.runners: Worker GM-C02FM31JMD6M_f8b0d490d5204fd5bb73e49e82f57ecb failed to send heartbeat, setting state to missing.
[2024-03-16 22:13:13,710] GM-C02FM31JMD6M/INFO/locust.runners: Worker GM-C02FM31JMD6M_7b327b6b1ba34651add3e0b12d75bb56 failed to send heartbeat, setting state to missing.
[2024-03-16 22:13:13,710] GM-C02FM31JMD6M/INFO/locust.runners: The last worker went missing, stopping test.

[cyberw]

Interesting. Can you share the full locustfile? In a separate ticket, because your issue doesnt really relate to the original issue.