[PE-1779] add retry in case of inserter failure

[avivl]

Add retry for k8s calls in inserter

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.97%. Comparing base (baa447d) to head (423b399).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #28   +/-   ##
=======================================
  Coverage   70.97%   70.97%           
=======================================
  Files          15       15           
  Lines        1664     1664           
=======================================
  Hits         1181     1181           
  Misses        346      346           
  Partials      137      137           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[smintz]

I am not familiar with this retry mechanism, so I might be wrong on my assumptions, but, from what I noticed:

• There is no limit on how many retries
• There is no indication or log something happened
• There is no validation for the type of error (should we retry if context canceled?)

This means if something is wrong, this will go to endless loop without any indication to the caller.
I think its better to move the retry mechanism to the inserter/agent and not to implement it on the KV level.
thoughts?
I am usually using this library: https://pkg.go.dev/github.com/avast/retry-go

[avivl]

I am not familiar with this retry mechanism, so I might be wrong on my assumptions, but, from what I noticed:

• There is no limit on how many retries
• There is no indication or log something happened
• There is no validation for the type of error (should we retry if context canceled?)

This means if something is wrong, this will go to endless loop without any indication to the caller. I think its better to move the retry mechanism to the inserter/agent and not to implement it on the KV level. thoughts? I am usually using this library: https://pkg.go.dev/github.com/avast/retry-go

• There is no limit on how many retries - var DefaultRetry = wait.Backoff{ Steps: 5, Duration: 10 * time.Millisecond, Factor: 1.0, Jitter: 0.1, }

• I'll add logs

• There is no validation for the type of error (should we retry if context canceled?) this is only for errors returned from k8s.

I prefer to use k8s retry and not an external library

[smintz]

few nits