Add a knob to allow Kong client creation retries

[mlavacca]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

In the controller-manager run function, KIC creates the Kong clients. In case Kong is not ready yet, the run function returns an error to the caller, and the process exits, making the pod fail and getting into a CrashLoopBackOff state. This is an expected error that gets automatically solved as soon as the Kong pod gets ready to accept connections. However, the failure of the KIC pod can create alerts in systems having monitoring enabled. We may want to introduce a mechanism to retry the client creation upon failure and introduce a new flag to customize such behavior.

Acceptance Criteria

• instead of returning an error, a retry mechanism is implemented in the run function
• a flag(s) to customize the above retry parameters are introduced