Multiple processes running when limit is set to 1

[skorper]

We have a singleton process (only enabled on 1 worker, and limit is 1), but we are occasionally seeing 2 concurrent runs of that process.

It seems like this problem is more likely to occur when the process is failing.

[galenatjpl]

One theory is that there are various increment and decrement counter points in the code, and it's probably some race condition where the counter is somehow 0 when it really should be 1 for a very short time, and the worker thinks it's okay to grab and launch another process.

[galenatjpl]

This could be complicated and time-consuming to fix. Doesn't happen all that often..
Rough time estimate: 1 week

[kgrimes2]

Related to #61