Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decorrelatedJitterGenerator returns NaN #86

Closed
novemberborn opened this issue Jan 22, 2024 · 3 comments
Closed

decorrelatedJitterGenerator returns NaN #86

novemberborn opened this issue Jan 22, 2024 · 3 comments

Comments

@novemberborn
Copy link
Contributor

Perhaps I'm using it wrong, but after 1025 attempts the jitter in the state goes to Infinity and for the 1026th attempt the delay is NaN.

const { decorrelatedJitterGenerator: d }= require('cockatiel')

const loop = (limit, state = [0,0]) => {
  for(let i = 0; i < limit; i++) {
    let delay;
    ([delay, state] = d(state, { generator: d, exponent: 2, maxDelay: 15*60_000, initialDelay: 45_000 }));
    console.log(delay, state)
  }
}

loop(1026)

Output:

…
900000 [ 1023, 5.201197568175768e+307 ]
900000 [ 1024, 1.3382550381451475e+308 ]
900000 [ 1025, Infinity ]
NaN [ 1026, Infinity ]
@mpareja
Copy link

mpareja commented May 3, 2024

We have encountered a CRITICAL failure where-in retries are performed without delay. The issue is likely related to the problem outlined in this ticket since the issue occurs after 1025 retries.

The problem can be reproduced using a retry with new ExponentialBackoff({ initialDelay: 1_000, maxDelay: 100 }). After ~1025 attempts, the retries occur much more aggressively than every 100ms.

The issue is critical because the retry loop has the potential to overwhelm downstream systems.

@mpareja
Copy link

mpareja commented May 3, 2024

@novemberborn Maybe we should update the title in this ticket to reflect the criticality? Something like, CRITICAL: error state results in retries without delay

@connor4312
Copy link
Owner

Fixed in 3.1.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants