retry clearml calls on 429 responses.

[johnml1135]

#524

---

This change is 

[ddaspit]

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @Enkidu93)

[Enkidu93]

Thank you for doing this, John! Have we figured out with ClearML if we're actually over-requesting because of an error in our code? Also, might it be wise to see if them upping the limit fixes things before introducing another change like this (i.e., this might increase the number of requests even while they're upping our limit)?

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @johnml1135)

[johnml1135]

They just removed the limit entirely. The issue may be that all of our servers present themselves as the same IP address. Also, each agent calls out to ClearML every 2 seconds. It's a number of things together.

[johnml1135]

Moreover, we were intending the code to handle the 429 case and it was a mistake that it didn't. That is why I am ok going forward with this change.