Invalidate CommunicationClientCache Through Failover Notifications

[edwardgarza]

Is your feature request related to a problem? Please describe.
I use a servicepartitionclient to talk to my stateful services and use a factory that changes very little from the communicationclientfactorybase.

When a stateful service fails over the communicationclient should be aborted, the communicationclientcache invalidated, and then a new communicationclient should be added to the communicationclientcache with the updated uri/ip+port mapping. The current flow looks like this:

existing calls to the now stale partition will hopefully timeout or fail after a period of time
Once a non-transient error is encountered the client will be aborted and removed from the cache

Instead I'd like to fail the current request instantly, remove the stale client from the cache, and create a new one with doresolve = true.

It looks like this is a feature that is understood to be needed but not implemented in here under ClearClientCacheEntries

Describe the solution you'd like
I would like the client cache to be invalidated and the client to be aborted whenever a primary or replica change notification occurs. I would be happy implementing this as this would greatly improve our service's tail end latency

Describe alternatives you've considered
In my team's service I have implemented a workaround for now that only works when talking to the primary.

1. Listen for primary change notifications on the client of the stateful service
2. When I receive the notification, I make a dummy call that throws a special type of exception that is handled by the communicationclientfactory as non-transient.
3. In the abortclient code I cancel all existing calls instantly

The above only works for primaries and won't extend to secondaries however.
Additional context
Add any other context or screenshots about the feature request here.