PCS has a lot of 403 and Rate Limit Exceeded exceptions

[dkurepa]

Context

Starting on December 20th, we started getting RateLimiting and 403 (Forbidden) exceptions from GitHub.
The running theory is that the 403s are also because of Rate Limiting, GitHub API documentation says it is a possibility.

The main fact supporting this theory is that looking at a single replica at a point in time, we can clearly see that in some cases the same API call fails with a 403 and sometimes it succeeds, even tho they're made with the same cached token.

Goals

Stop these exceptions from happening. A good start might be to add some logging so we can figure out how many calls we actually make per hour. Currently the only related metric is the number of token cache hits we have per hour (around 4.3k), but that number gives a lower limit, there's a number of places where we reuse the same HttpClient without going to the token cache again.

Also we might have to start respecting the backoff timer we get sent in these Rate Limiting responses

[premun]

We have figured out why these happen and logged an issue with next immediate steps. We will go from there so closing this.