Use exponential backoff for DNS updates

[hakman]

The AWS Route53 API is severely rate-limited and makes periodic e2e tests fail. This may happen to users also, as they see DNS records not being updated quite often lately.
Not sure if AWS changed something or we're just noticing this, but there's not much that can be done except do this less often.

/cc @rifelpet @olemarkus

[olemarkus]

Shouldn't we use some exponential backoff or something for this operation?

If the ratelimit is caused by external factors, say that there are many more tests running in the same region in the same account (by other installers/projects?), increasing to 30 just makes matter worse. We would try less often, but still risk being pulled into a rate limit.

[hakman]

I agree that exponential backoff is a better idea, though I don't get how increasing to 30s makes the matter worse.
In any case, 5s between runs is totally unreasonable in real life.

[olemarkus]

If a user has some other system installed that triggers the ratelimit this change makes things worse because it will fail on just as many calls, but we make the calls less frequently. So it will need a lot more time to get a successful attempt through.

[hakman]

If a user has another system installed that triggers the rate limit somehow, nothing will help, not even exponential backoff will help. This way at least we reduce the chance of this happening because of too many requests.

[hakman]

/retest

[hakman]

@olemarkus Does this look more appealing now?

[olemarkus]

Certainly. Thanks for this, @hakman

[hakman]

Thanks @olemarkus. I want to fix one small issue before merging.
/hold

[rifelpet]

There should be random jitter added to the backoff to avoid the thundering herd problem

[hakman]

There should be random jitter added to the backoff to avoid the thundering herd problem

I don't think we have many updates in parallel from the same cluster. Other clusters should not matter that much as they are already quite random.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: olemarkus, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [olemarkus,rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment