Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

在 locust 发送 stop 消息时,boomer 偶现 panic 的情况 #187

Closed
alexshopee opened this issue Mar 1, 2023 · 3 comments
Closed

Comments

@alexshopee
Copy link

相关日志如下:
2023/02/27 19:15:21 Recv stop message from master, all the goroutines are stopped
panic: sync: negative WaitGroup counter

goroutine 158192 [running]:
sync.(*WaitGroup).Add(0x1?, 0x1?)
/usr/local/go/src/sync/waitgroup.go:80 +0xda
sync.(*WaitGroup).Done(...)
/usr/local/go/src/sync/waitgroup.go:105
github.com/myzhan/boomer.(*slaveRunner).onAckMessage(0xc001b36fc0?, 0xc0032ac2a0?)
/root/go/pkg/mod/github.com/myzhan/[email protected]/runner.go:385 +0x28
github.com/myzhan/boomer.(*slaveRunner).onMessage(0xc001d0e000, {0x11d9420?, 0xc003224db0?})
/root/go/pkg/mod/github.com/myzhan/[email protected]/runner.go:414 +0xa5
github.com/myzhan/boomer.(*slaveRunner).startListener.func1()
/root/go/pkg/mod/github.com/myzhan/[email protected]/runner.go:461 +0x37
created by github.com/myzhan/boomer.(*slaveRunner).startListener
/root/go/pkg/mod/github.com/myzhan/[email protected]/runner.go:457 +0x56

@alexshopee
Copy link
Author

alexshopee commented Mar 6, 2023

这里补充一下,并不一定是 boomer 接收 stop 消息,然后发送给 locust client_ready 消息之后,等待 ack 消息时会出现上述问题,如果出现了接收两次 ack 消息都会触发这个 bug
感觉这个 bug 很不好复现,可能是在网络不是很稳定的情况下才会出现,我在 onAckMessage 函数内加了一个 log:

func (r *slaveRunner) onAckMessage(msg *genericMessage) {
	log.Println("receive the ack message")
	r.waitForAck.Done()
	Events.Publish(EVENT_CONNECTED)
}

然后我再执行很多次之尝试后,出现了下述问题:
2023/03/05 19:34:04 Boomer is built with gomq support.
2023/03/05 19:34:05 Boomer is connected to master(tcp://10.129.118.77:15557) press Ctrl+c to quit.
2023/03/05 19:34:05 receive the ack message
2023/03/05 19:34:05 receive the ack message
panic: sync: negative WaitGroup counter

goroutine 264775 [running]:
sync.(*WaitGroup).Add(0xc001e46eb8?, 0x1?)
/usr/local/go/src/sync/waitgroup.go:80 +0xda
sync.(*WaitGroup).Done(...)
可见接收了两次 ack 消息

master 相关日志如下:

image

@myzhan
Copy link
Owner

myzhan commented Mar 7, 2023

OK,我做一下容错。不过目前 boomer 从实现上,对弱网环境的支持不好,考虑到弱网也不适合做性能测试,就没在这里花很多力气。

@alexshopee
Copy link
Author

嗯嗯理解的哈

myzhan added a commit that referenced this issue Mar 21, 2023
@myzhan myzhan closed this as completed Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants