Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling OpenTracing makes ingress controller unhealthy #3057

Closed
rafaeljesus opened this issue Sep 8, 2018 · 4 comments
Closed

Enabling OpenTracing makes ingress controller unhealthy #3057

rafaeljesus opened this issue Sep 8, 2018 · 4 comments

Comments

@rafaeljesus
Copy link

Is this a request for help? Yes I've asked for help in slack channel

What keywords did you search in NGINX Ingress controller issues before filing this one? OpenTracing, Jaeger


BUG REPORT

NGINX Ingress controller version: 0.17.1
Kubernetes version (use kubectl version): v1.10.6
Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS
  • Kernel (e.g. uname -a): 4.4.0-1065-aws

What happened:
I am trying to enable opentracing jaeger to my ingress controller, however the controller pod health check fails and it does not work. I am editing the controller configmap and then I delete one pod to test and tee latter health check fails.

configmap

enable-opentracing: "true"
jaeger-collector-host: jaeger-collector.tracing.svc.cluster.local
jaeger-collector-port: "14267"
jaeger-sampler-type: "probabilistic"
jaeger-sampler-param: "0.3"
jaeger-service-name: "ingress-nginx"

error on the logs

nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused

The pod works fine after I disable opentracing, when I check the opentracing.json I see:

"reporter": {
    "localAgentHostPort": "jaeger-collector.tracing.svc.cluster.local:14267"
  }

What you expected to happen:
At least some error message describe any error about the opentracing config is properly set.

Thank you

@aledbf
Copy link
Member

aledbf commented Sep 8, 2018

nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused

This is not an error. The reason for this message is the scraping of prometheus metrics start before the ingress controller. This is fixed in 0.19

@rafaeljesus
Copy link
Author

rafaeljesus commented Sep 8, 2018

thx for the quick reply! Interesting, it only happen when I enable the trace and the pod is never ready. I will update to the 0.19 version anyway

@rafaeljesus
Copy link
Author

@aledbf I was able to reproduce now and got Error: signal: segmentation fault (core dumped) the same as here #2856 so as you mentioned in that PR also it should be fixed in 0.19. Will reopen the issue in case the error persists, thank you

@rafaeljesus
Copy link
Author

As a reference, using zipkin config fields worked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants