Client checkin spinlocks on RPC error

[faec]

In client/client.go, (*client).startCheckin has a loop calling through to (*client).checkinRoundTrip. In normal operation the rate of these round trips is gated by the pace of the Agent's checkin responses, however if there is an error condition that prevents connecting to the Agent the loop will fall through immediately, spinlocking on the error. This manifests in live deployments as e.g. a gRPC PermissionDenied error occurring 5-10K times per second, which both worsens the effects of the error and floods the logs so that any clues about the original cause of the error are lost.

This error loop should have some minimum retry interval to prevent this symptom.

[cmacknz]

Related:

• [Elastic Agent] Failures connecting to the Elastic Agent can spam the logs with errors beats#35351
• Network failures (?) cause *excessive* logging elastic-agent#2628