locust stuck in hatching state

[sanga]

we're not quite sure how we got into this state but in any case, our locust is currently stuck in hatching state. I've clicked stop and it's still hatching (with statistics updating). Also starting a new test seems to not have any effect.

I'll try to see if I can figure a minimal set of steps to repro this, but in the mean time any ideas on how to debug this? Wild stab in the dark, but perhaps it's something like:

• hit stop
• locust starts stopping
• start a new run whilst locust is still telling it's slaves to shut down?

[heyman]

Hmm, is the problem consistent (does it happen again if you kill alla locust nodes and try again)? How many slave nodes are you running?

[sanga]

killing the locust master and slaves and restarting them all fixed the problem.

[heyman]

Okay. I've seen similar stuff happen on at least one occasion.

I would like to add some kind of heartbeat messages that are sent periodically from the slaves to the master, with the slave's current status. That could be used to display the status of slave nodes, and could probably help in debugging similar issues.

[sanga]

Sounds reasonable. I'll in any case have a crack at reproing this to see if
I can figure anything out

On Tue, Apr 29, 2014 at 5:50 PM, Jonatan Heyman [email protected]:

Okay. I've seen similar stuff happen on at least one occasion.

I would like to add some kind of heartbeat messages that are sent
periodically from the slaves to the master, with the slave's current
status. That could be used to display the status of slave nodes, and could
probably help in debugging similar issues.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/146#issuecomment-41685439
.

[sanga]

Actually seems like just clicking stop whilst in hatching is enough to trigger this. Somehow stopping whilst in the hatching state is not working for us.

[sanga]

Seems like the stop event doesn't get through somehow, as:
a) I click stop, the stop button disappears, but stats continue to update
b) I refresh the browser, the stop button reappears
c) goto a

[sanga]

also, I happen to be running only with a single master (no slaves) at the moment, so it's not related to master-slave comms

[xuedm2016]

I met the same issue, have any plan to fix it?

[DmitryMarkovnikov]

The same here on master - slave config

[max-rocket-internet]

Any update on this? Or how to debug it? Or how to work around it?

Many times I see the load test is STOPPED:

But all slaves are still hatching:

[cyberw]

@max-rocket-internet Does this happen all the time? Could it be something related to your test plan? Or maybe it is related to the high number of slaves?

[heyman]

@max-rocket-internet Do you see any exceptions (not originating from your test code) in any of the slave processes or the master process? Any client side javascript errors?

[max-rocket-internet]

@heyman, sorry I just deleted a message where I was mistakingly using an old locust version.

For me, this issue is resolved. I think we can close it 🙂