Add retry mechanism for put call #18137

Juanadelacuesta · 2023-08-03T08:57:05Z

The SDK for the locking could benefit from a retry mechanism for calls made to the nomad server, this PR introduces this mechanism on top of the put call already present on the api.

jrasell

Looking good. Added some questions and think it would be good to have some more tests to cover functions such as the calculateDelay function.

jrasell · 2023-08-03T10:35:46Z

api/retry.go

+	for attempt := uint64(0); attempt < rc.MaxRetries; attempt++ {
+		iDelay = rc.calculateDelay(attempt)
+
+		t := time.NewTimer(iDelay)


I think we also will need a defer t.Stop() here, otherwise we could leak timers if the context is closed.

Another thing that might make sense is to instantiate the timer outside the loop and then call Reset on it when needed. Our helper.SafeTimer would be nice here but we don't want to pull that into this package.

Sounds good, reuse the time, will do!

jrasell · 2023-08-03T10:36:23Z

api/retry.go

+		return 0
+	}
+
+	new := rc.DelayBase << (attempt - 1)


Can we rename this variable as it collides with the builtin function.

jrasell · 2023-08-03T10:42:39Z

api/retry.go

+type retryOptions struct {
+	MaxRetries      uint64        // Optional, defaults to 3
+	MaxBetweenCalls time.Duration // Optional, defaults to 0, meaning no time cap
+	MaxToLastCall   time.Duration // Optional, defaults to 0, meaning no time cap
+	FixedDelay      time.Duration // Optional, defaults to 0, meaning Delay is exponential, starting at 1sec
+	DelayBase       time.Duration // Optional, defaults to 1sec
+}


The struct isn't exported but the parameters are, do we want to alter the parameters? This also makes me wonder if this is something we would want to make available to external users of the API package?

Could we also add some added descriptions on the parameters? I had to read the code to understand MaxBetweenCalls the first time.

jrasell · 2023-08-03T10:49:48Z

api/retry.go

+	return wm, err
+}
+
+func isCallRetriable(statusCode int) bool {


Seeing as this function is available to the whole API package, I wonder if it needs a more descriptive name?

That was the best I could come up with jejej any ideas?

jrasell · 2023-08-03T11:31:58Z

api/retry.go

+	var err error
+	var wm *WriteMeta
+
+	iDelay := time.Duration(0)


iDelay for initial delay? If so, we use it for all delays, so maybe delayDuration?

it was for iteration delay, but lets change it for attempt delay :)

tgross · 2023-08-03T12:37:04Z

api/retry.go

+func newRetryClient(c client, opts retryOptions) *retryClient {
+	rc := &retryClient{
+		c: c,
+		retryOptions: retryOptions{
+			MaxRetries: 3,
+			DelayBase:  time.Second,
+		},
+	}
+
+	if opts.DelayBase != 0 {
+		rc.DelayBase = opts.DelayBase
+	}
+
+	if opts.MaxRetries != 0 {
+		rc.MaxRetries = opts.MaxRetries
+	}
+
+	if opts.MaxBetweenCalls != 0 {
+		rc.MaxBetweenCalls = opts.MaxBetweenCalls
+	}
+
+	if opts.MaxToLastCall != 0 {
+		rc.MaxToLastCall = opts.MaxToLastCall
+	}
+
+	if opts.FixedDelay != 0 {
+		rc.FixedDelay = opts.FixedDelay
+	}
+
+	return rc
+}


retryClient isn't exported, so it's only available for Nomad-internal uses. I think that's probably fine for now (until we're happy with our own uses of it). But from the standpoint of API design, maybe it would make sense to have all this functionality in the standard Client by adding it to the Config struct as an unexported field?

Im not sure I know what you mean by adding as a Config

So if we added the retryOptions to api.Config then we wouldn't need to have a special retryClient struct, and the standard api.Client could use the retry options whenever it's been configured that way.

type Config struct { // ... there's a bunch of existing field here retry retryOptions }

And then later if we want to, we could make the retry options public for all api package consumers (ex. could the Terraform provider use this?)

Makes sense :)

I was thinking about this last night and there's no real reason to hold this PR up for this though. The new code you've added isn't exported, so it's not part of the public API and will be safe to update later on without breaking backwards compatibility. Let's ship and do any changes like this later if we want.

tgross · 2023-08-03T12:39:09Z

api/retry.go

+	for attempt := uint64(0); attempt < rc.MaxRetries; attempt++ {
+		iDelay = rc.calculateDelay(attempt)
+
+		t := time.NewTimer(iDelay)


Another thing that might make sense is to instantiate the timer outside the loop and then call Reset on it when needed. Our helper.SafeTimer would be nice here but we don't want to pull that into this package.

tgross · 2023-08-03T12:46:54Z

api/retry.go

+	return statusCode > http.StatusInternalServerError &&
+		statusCode < http.StatusNetworkAuthenticationRequired ||
+		statusCode == http.StatusTooManyRequests


Unfortunately there are still a bunch of places in the HTTP server where we return 500 when we should be returning 400. 😊 But I don't think we can fix all of those in this PR. So long as we're not exporting this for now I think that's fine.

tgross · 2023-08-03T12:52:30Z

api/retry.go

+type retryOptions struct {
+	MaxRetries      uint64        // Optional, defaults to 3
+	MaxBetweenCalls time.Duration // Optional, defaults to 0, meaning no time cap
+	MaxToLastCall   time.Duration // Optional, defaults to 0, meaning no time cap


I don't think we really need this if we're passing context.Context to retryPut. The caller can cancel the work however they see fit there, and it would be easy to misconfigure this to be shorter than any context timeout.

I was thinking of the lock lease on this, how it doesn't make sense to retry a call after the TTL passed, the servir will expire the lock by then anyways.

stswidwinski · 2023-08-15T14:10:02Z

api/retry.go

+		return 0
+	}
+
+	newDelay := c.config.retryOptions.delayBase << (attempt - 1)


I believe this will overflow quite quickly similarly to what is shown here: #18200

Good catch. @Juanadelacuesta note that we don't want to pull in the helper into this api package, so we'll need to duplicate the logic we added in #18200 here.

Should we set a limit to the max number of attempts?

tgross · 2023-08-16T13:08:13Z

api/retry.go

+	// Ensure that a big attempt number or a big delayBase number will not cause
+	// a negative delay by overflowing the delay increase.
+	if math.MaxInt64/c.config.retryOptions.delayBase.Nanoseconds() < (1 << attempt) {
+		return defaultMaxBackoffDelay
+	}


This doesn't work to protect against overflow in the bitshift, because it does the bitshift in order to do the check. Without a low cap on the number of attempts it'd be easy to cause this to overflow. See https://go.dev/play/p/DztXnyHL9hk for the results.

Instead, we should be tracking the current delay in the caller and returning early if that delay exceeds the max backoff (or default max backoff, if no max backoff is set).

…hat wont cause the delay to overflow

* func: add retry mechanism for put call * style: remove debg print * fix: move timer outside of main loop, add descriptions * func: make the retryPut a method of the API client directly * func: add verification to avoid negative delays * func: change the delay capping mechanism to checking for an attempt that wont cause the delay to overflow

Juanadelacuesta added 2 commits August 3, 2023 10:51

func: add retry mechanism for put call

e2316a4

style: remove debg print

2e65db3

Juanadelacuesta requested review from tgross and jrasell August 3, 2023 08:57

vercel bot deployed to Preview – nomad-storybook-and-ui August 3, 2023 09:02 View deployment

Juanadelacuesta requested a review from lgfa29 August 3, 2023 11:18

jrasell reviewed Aug 3, 2023

View reviewed changes

tgross reviewed Aug 3, 2023

View reviewed changes

Juanadelacuesta requested a review from tgross August 3, 2023 16:15

fix: move timer outside of main loop, add descriptions

ced2bd8

vercel bot deployed to Preview – nomad-storybook-and-ui August 3, 2023 16:18 View deployment

Juanadelacuesta requested a review from jrasell August 4, 2023 08:20

tgross approved these changes Aug 15, 2023

View reviewed changes

func: make the retryPut a method of the API client directly

e246e34

vercel bot deployed to Preview – nomad-storybook-and-ui August 15, 2023 13:30 View deployment

stswidwinski reviewed Aug 15, 2023

View reviewed changes

func: add verification to avoid negative delays

7515247

vercel bot deployed to Preview – nomad-storybook-and-ui August 16, 2023 10:07 View deployment

Juanadelacuesta requested review from stswidwinski and tgross August 16, 2023 10:07

tgross reviewed Aug 16, 2023

View reviewed changes

func: change the delay capping mechanism to checking for an attempt t…

b75d022

…hat wont cause the delay to overflow

vercel bot deployed to Preview – nomad-storybook-and-ui August 16, 2023 14:35 View deployment

Juanadelacuesta merged commit adfcb1b into f-gh-17449 Aug 17, 2023

Juanadelacuesta deleted the f-gh-17449-sdk branch August 17, 2023 11:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add retry mechanism for put call #18137

Add retry mechanism for put call #18137

Juanadelacuesta commented Aug 3, 2023

jrasell left a comment

jrasell Aug 3, 2023

tgross Aug 3, 2023

Juanadelacuesta Aug 3, 2023

jrasell Aug 3, 2023

Juanadelacuesta Aug 3, 2023

jrasell Aug 3, 2023

jrasell Aug 3, 2023

Juanadelacuesta Aug 3, 2023

jrasell Aug 3, 2023

Juanadelacuesta Aug 3, 2023

tgross Aug 3, 2023

Juanadelacuesta Aug 3, 2023

tgross Aug 14, 2023

Juanadelacuesta Aug 15, 2023

tgross Aug 15, 2023

tgross Aug 3, 2023

tgross Aug 3, 2023

tgross Aug 3, 2023

Juanadelacuesta Aug 3, 2023 •

edited

Loading

stswidwinski Aug 15, 2023

tgross Aug 15, 2023

Juanadelacuesta Aug 15, 2023

tgross Aug 16, 2023

Add retry mechanism for put call #18137

Add retry mechanism for put call #18137

Conversation

Juanadelacuesta commented Aug 3, 2023

jrasell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Juanadelacuesta Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Juanadelacuesta Aug 3, 2023 •

edited

Loading