Orleans 7.x does not support multiple stateless worker grain types with different StatelessWorkerPlacement.MaxLocal values

[isaachili]

We have observed that under load, the value provided to the maxLocalWorkers parameter in the StatelessWorkerAttribute constructor does not get respected at times. In such situations, StatelessWorkerPlacement.MaxLocal is getting set to DefaultMaxStatelessWorkers.

This behaviour was reproduced on multi-node and single-node clusters alike, whereby the stateless worker is being constrained to a maximum of 1 stateless worker per silo as per the following code:

[StatelessWorker(1)]
public class FooGrain : Grain, IFooGrain
{	
}

Whilst debugging Orleans code, we noticed that the initial stateless worker grain gets created via StatelessWorkerGrainContext.cs#L185. StatelessWorkerGrainContext.CreateWorker(object) gets invoked a second time, this time by StatelessWorkerGrainContext.cs#L216 due to _workers.Count being less than _maxWorkers, which holds a value of 8 (number of logical cores) instead of 1.

Furthermore, when the GrainTypeSharedContext constructor is invoked, the PlacementStrategy property is being set to a StatelessWorkerPlacement instance with a MaxLocal of 1.

[isaachili]

I have noticed that this happens whenever there are different stateless worker grain types with different stateless worker count limits.

[StatelessWorker(1)]
public class SingleStatelessWorkerGrain : Grain, ISingleStatelessWorkerGrain
{	
}

[StatelessWorker]
public class MultipleStatelessWorkerGrain : Grain, IMultipleStatelessWorkerGrain
{	
}

It seems that the PlacementStrategy used by StatelessWorkerGrainContext is the same instance for both SingleStatelessWorkerGrain and MultipleStatelessWorkerGrain, even though the two grain types have different StatelessWorker attributes, with different MaxLocal values.