Add retry mechanism for neural search inference

[zane-neo]

Is your feature request related to a problem?

In some edge cases e.g. the instance down and the inference requests either to ingest or to query can fail, adding retry can relief this issue dramatically.

What solution would you like?

Add basic retry mechanism in neural search inference client.

What alternatives have you considered?

Add complicated retry like backoff policy, jitter etc in the retry mechanism, but for our system it's an internal retry which means we know the how the retry will behave e.g. round robin or least load. So a basic retry is enough for our system.

Do you have any additional context?

NA