Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nighthawk may hang on exit #627

Closed
oschaaf opened this issue Feb 12, 2021 · 0 comments · Fixed by #628
Closed

Nighthawk may hang on exit #627

oschaaf opened this issue Feb 12, 2021 · 0 comments · Fixed by #628
Assignees
@oschaaf
Labels
bug Something isn't working

Comments

@oschaaf
Copy link
Member

oschaaf commented Feb 12, 2021

In tests I noticed that Nighthawk may sometimes wait for a long time, or even hang after reporting results. After reproducing it, I think there are two factors at play.

  • We set up a drain callback on the connection pool regardless of the number of active connections. But when passing, say, --request-header "connection: close", no active connections may exist, and therefore no drain callback will be fired.
  • When sending high volume traffic into a proxy, it may buffer up a lot of requests. In practice, this spooling may take a long time to unwind.
@oschaaf oschaaf added the bug Something isn't working label Feb 12, 2021
@oschaaf oschaaf self-assigned this Feb 12, 2021
oschaaf pushed a commit to oschaaf/nighthawk that referenced this issue Feb 12, 2021
- don't set up a drain callback when there are no active connections
4896d5f 
- set up a timer that will cap the amount of time we wait for the pool
  to drain.
- disable latency measurement when commencing the drain procedure, as
  by that time we are no longer interested in it, and in particular
  don't want to hear about any warnings issued by the Statistic
  implementation about recorded values being too large.

Fixes envoyproxy#627

Signed-off-by: Otto van der Schaaf <[email protected]>
@oschaaf oschaaf mentioned this issue Feb 12, 2021
Merged
oschaaf pushed a commit to oschaaf/nighthawk that referenced this issue Feb 12, 2021
Fix potential hangs on exit & cap drain wait time
8f1fdf2 
- don't set up a drain callback when there are no active connections
- set up a timer that will cap the amount of time we wait for the pool
  to drain.
- disable latency measurement when commencing the drain procedure, as
  by that time we are no longer interested in it, and in particular
  don't want to hear about any warnings issued by the Statistic
  implementation about recorded values being too large.

Fixes envoyproxy#627

Signed-off-by: Otto van der Schaaf <[email protected]>
mum4k pushed a commit that referenced this issue Feb 15, 2021
Fix potential hangs on exit & cap drain wait time (#628)
15b86be 
- don't set up a drain callback when there are no active connections
- set up a timer that will cap the amount of time we wait for the pool
  to drain.
- disable latency measurement when commencing the drain procedure, as
  by that time we are no longer interested in it, and in particular
  don't want to hear about any warnings issued by the Statistic
  implementation about recorded values being too large. 

Fixes #627

Signed-off-by: Otto van der Schaaf <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oschaaf oschaaf

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix potential hangs on exit & cap drain wait time oschaaf/nighthawk
1 participant
@oschaaf