Race conditions in the herd package

[vj396]

Summary
We we have a large mdb.json file, say 100+, the dominator binary throws off a data race, when ran under the race detector.

this race condition seems to originate in dom/herd package, where a lot of read/writes are happening in a concurrency unsafe manner in the sub.go file, under methods connectAndPoll.

this originates from the usage of the derived datatype subStatus.

Is there any plan on addressing these data races, and leverage atomic operations for read/writes?

[rgooch]

To reduce lock contention, there are indeed some accesses for this variable which are not locked. The reasoning behind this is that since a status variable is inherently out of date once obtained, it need not be protected by a lock for simply presenting the value on a dashboard. The variable is protected by a lock when updating or if a decision needs to be made when reading it.

Are there any observed bugs due to this, or is this strictly being raised by the race detector?

[vj396]

we have hit this one time and it did cause dominator to crash

[rgooch]

Can you post a full panic log? I'd like to see how the race leads to a crash.

[vj396]

https://gist.github.com/vj396/b72e92b46fbc883752bfdf285860da09