Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Exessive logging #2211

Closed
4 tasks done
pittbull opened this issue Oct 24, 2024 · 6 comments · Fixed by #2227
Closed
4 tasks done

[Bug] Exessive logging #2211

pittbull opened this issue Oct 24, 2024 · 6 comments · Fixed by #2227
Labels
bug Something isn't working
Milestone

Comments

@pittbull
Copy link

Is this a support request?

  • This is not a support request

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Headscale successfully up and running behind Nginx Proxy Manager. Logs in NPM suggests a problem reaching DERP:

[24/Oct/2024:13:54:05 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 178.232.163.94] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"
[24/Oct/2024:13:54:05 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 178.232.163.94] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"
[24/Oct/2024:13:54:16 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 178.232.163.94] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"
[24/Oct/2024:13:54:16 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 178.232.163.94] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"
[24/Oct/2024:13:54:16 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 192.168.1.19] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"
[24/Oct/2024:13:54:16 +0200] - 404 404 - GET https <URL> "/derp/latency-check" [Client 192.168.1.19] [Length 0] [Gzip -] [Sent-to unraid.home] "Go-http-client/1.1" "-"

This is logged every other second.

Expected Behavior

If this indicates a problem I don't know how to solve it. If not could this logging be disabled?

Steps To Reproduce

Start headscale as normal.

Environment

- OS: Unraid
- Headscale version: 0.23.0
- Tailscale version: 1.76.1

Runtime environment

  • Headscale is behind a (reverse) proxy
  • Headscale runs in a container

Anything else?

No response

@pittbull pittbull added the bug Something isn't working label Oct 24, 2024
@pittbull
Copy link
Author

I temporarily reduced the logging by adding a custom location in nginx, and disabling access logging.

As I use these logfiles to analyze access, this events skewed the numbers.

image

@nblock
Copy link
Collaborator

nblock commented Oct 29, 2024

According to tailscale/tailscale@15fc6cd the routes /derp/probe and /derp/latency-check are the same and different versions of the tailscale client use one or the other endpoint.

Headscale currently only supports /derp/probe (see: https://github.com/juanfont/headscale/blob/main/hscontrol/app.go#L462). We should probably handle /derp/latency-check with the same handler.

@nblock
Copy link
Collaborator

nblock commented Oct 29, 2024

Since you are using Nginx Proxy Manager, you might as well add an internal redirect on the exact path /derp/latency-check until a fix is available in headscale. Something like:

location = /derp/latency-check {
  rewrite ^/derp/latency-check$ /derp/probe last;
}

@pittbull
Copy link
Author

I had to implement your suggestion like this:
image

This results in going from a 101 message to a 200.

@nblock
Copy link
Collaborator

nblock commented Oct 29, 2024

Right, I was actually on track to workaround http/404 and don't care about the logging as you wrote.

Headscale itself does not log requests to /derp/probe or /derp/latency-check, so the excessive logging is only due to the reverse proxy and should be handled there.

@pittbull
Copy link
Author

I see. Thanks for the feedback.

@kradalby kradalby reopened this Oct 30, 2024
nblock added a commit to nblock/headscale that referenced this issue Nov 6, 2024
According to tailscale/tailscale@15fc6cd
the routes `/derp/probe` and `/derp/latency-check` are the same and
different versions of the tailscale client use one or the other
endpoint.

Also handle /derp/latency-check

Fixes: juanfont#2211
@nblock nblock mentioned this issue Nov 6, 2024
6 tasks
@nblock nblock closed this as completed in 9a46c57 Nov 6, 2024
hopleus pushed a commit to hopleus/headscale that referenced this issue Nov 12, 2024
According to tailscale/tailscale@15fc6cd
the routes `/derp/probe` and `/derp/latency-check` are the same and
different versions of the tailscale client use one or the other
endpoint.

Also handle /derp/latency-check

Fixes: juanfont#2211
@nblock nblock added this to the v0.24.0 milestone Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants