improve token refresh logic

[yunhaoling]

addressing issue: Azure/azure-sdk-for-python#20573

The improvements include:

• allow override refresh window
  • resolve the refresh windows gap between current uamqp setting (6 mins) and AAD service (5 mins)
    • the AAD service returns a cached token, the service side refresh the token until 5 mins before expiration
• not to put an old token onto the cbs link
  • avoid throttling SB/EH service
  • avoid potentially blocking handler work while token is still valid but about to be expired

TODO:

• step1: discuss with anna on the fix/design
  • should override_... be a public parameter?
• step2: add tests

There's one more improvement I could think of, introducing backoff when calling "update_token" if the newly-generated token is the same as old one to avoid throttling token generation module/service, but I'm afraid that goes too far for now.

[annatisch]

Is this log statement going to get very noisy? Or is it very rare that token == previous_token after calling update_token?

[yunhaoling]

yes, it would be very noisy in the case token == previous_token (which happens now when there's a gap between the client refresh window (uAMQP 6 mins to refresh) and service refresh window (AAD service 5 mins refresh))

So once we have fixed the issue in uamqp/eh/sb, changing default refresh window to 5 mins, it should be rare -- at least not worse than what we currently have.

---

I think what uamqp is doing technically is not wrong -- it's doing its best to refresh token, however it's doing a bit excessively, retry without backoff. So ultimately we might want to introduce kinda backoff to token refresh to let it not throttle.

with that being said, I think we could keep this "potential" noisy logging for now until user complains.