Jaeger query shows error in log #112

mmshaikh88 · 2019-01-28T06:16:15Z

I am running jaeger with elasticsearch and elasticsearch is running on another VM.

But the jaeger query pod on kubernetes is showing following error in logs:

{"level":"error","ts":1548655854.701871,"caller":"zap/logger.go:33","msg":"error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap/logger.go:33\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:257\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:267"}

The text was updated successfully, but these errors were encountered:

mmshaikh88 · 2019-01-28T06:18:01Z

ubuntu@kube-master-1:~/jaeger-kubernetes$ kubectl logs jaeger-query-676d6658d6-cdqs9 -f  
{"level":"info","ts":1548655820.6596537,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"}
{"level":"info","ts":1548655820.7135236,"caller":"query/main.go:184","msg":"Archive storage not created","reason":"Archive storage not supported"}
{"level":"info","ts":1548655821.1784363,"caller":"query/main.go:132","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1548655821.1785743,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1548655821.1786165,"caller":"query/main.go:141","msg":"Starting HTTP server","port":16686}
{"level":"error","ts":1548655854.701871,"caller":"zap/logger.go:33","msg":"error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap/logger.go:33\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:257\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:267"}

jpkrohling · 2019-01-28T08:47:16Z

This is the relevant error message:

error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused

This happens because there's no agent at localhost available to the query component. This is usually done by adding a agent container to the query pod. Without it, you won't see spans that are created by the query component itself. If that's not a problem for you, you can ignore the error messages.

If you still need assistance, feel free to reopen this issue.

yurishkuro · 2019-01-28T14:19:49Z

@jpkrohling do you have that sidecar setup for the agent in our k8s templates?

jpkrohling · 2019-01-28T14:23:34Z

We should, but looks like we don't:

jaeger-kubernetes/jaeger-production-template.yml

Lines 129 to 139 in da22250

    
           containers: 
        
           - image: jaegertracing/jaeger-query:1.7.0 
        
             name: jaeger-query 
        
             args: ["--config-file=/conf/query.yaml"] 
        
             ports: 
        
             - containerPort: 16686 
        
               protocol: TCP 
        
             readinessProbe: 
        
               httpGet: 
        
                 path: "/" 
        
                 port: 16687

I think we only have it in the operator. I'll leave this open, to add the sidecar to this template.

mmshaikh88 · 2019-01-28T23:05:30Z

I don't have agent running along with query. Are you suggesting to setup agent along with query.

I do have jaeger-agent setup along with application sending trace to agent which further send trace to collector.

apiVersion: v1
kind: List
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: collector-pod
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "14268"
      spec:
        containers:
        - image: jaegertracing/jaeger-collector:1.7.0
          name: jaeger-collector
          args: ["--config-file=/conf/collector.yaml"]
          ports:
          - containerPort: 14267
            protocol: TCP
          - containerPort: 14268
            protocol: TCP
          - containerPort: 9411
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 14269
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: collector
                  path: collector.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-service
  spec:
    ports:
    - name: jaeger-collector-tchannel
      port: 14267
      protocol: TCP
      targetPort: 14267
    - name: jaeger-collector-http
      port: 14268
      protocol: TCP
      targetPort: 14268
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: ClusterIP
- apiVersion: v1
  kind: Service
  metadata:
    name: zipkin
    labels:
      app: jaeger
      jaeger-infra: zipkin-service
  spec:
    ports:
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: ClusterIP
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: query-pod
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "16686"
      spec:
        containers:
        - image: jaegertracing/jaeger-query:1.7.0
          name: jaeger-query
          args: ["--config-file=/conf/query.yaml"]
          ports:
          - containerPort: 16686
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 16687
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: query
                  path: query.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-service
  spec:
    ports:
    - name: jaeger-query
      port: 80
      protocol: TCP
      targetPort: 16686
    selector:
      jaeger-infra: query-pod
    type: LoadBalancer

mmshaikh88 · 2019-01-28T23:07:59Z

Configmap for jaeger to connect to elasticsearch running on separate VM as:

apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-configuration
  labels:
    app: jaeger
    jaeger-infra: configuration
data:
  span-storage-type: elasticsearch
  collector: |
    es:
      server-urls: http://10.145.70.97:9200
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://10.145.70.97:9200
  agent: |
    collector:
      host-port: "jaeger-collector:14267"

mmshaikh88 · 2019-01-28T23:12:14Z

Deployed hotrod with sidecar as jaeger-agent

apiVersion: v1
kind: Service
metadata:
  name: example-hotrod
spec:
  ports:
    - name: ui 
      port: 8080
      targetPort: 8080 
    - name: interface1 
      port: 8081
      targetPort: 8081
    - name: interface2 
      port: 8082
      targetPort: 8082
    - name: interface3 
      port: 8083
      targetPort: 8083
  selector:
    app: example-hotrod 
  #clusterIP: None
  type: LoadBalancer
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: example-hotrod 
spec:
  selector:
    matchLabels:
      app: example-hotrod 
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: example-hotrod 
    spec:
      containers:
      **- image: jaegertracing/example-hotrod:latest**
        name: example-hotrod 
        ports:
          - containerPort: 8080 
          - containerPort: 8081 
          - containerPort: 8082
          - containerPort: 8083 
        args:
          - "all"
      **- image: jaegertracing/jaeger-agent**
        name: jaeger-agent
        ports:
          - containerPort: 5775
            protocol: UDP
          - containerPort: 6831
            protocol: UDP
          - containerPort: 6832
            protocol: UDP
          - containerPort: 5778
            protocol: TCP
        args: ["--collector.host-port=jaeger-collector:14267"]

jpkrohling · 2019-01-29T10:46:58Z

Are you suggesting to setup agent along with query

Yes, if you want to get the Query traces reported to the collector, or if the error messages are bothering you. If you don't care about the spans generated by the query component, you can just ignore the error messages.

yurishkuro · 2019-01-29T13:59:00Z

You can also configure the sampling strategy in the collector for the query service to use 0 probability.

naseemkullah · 2019-05-10T20:04:51Z

Is it possible to configure query to talk to an agent elsewhere than localhost?

e.g. running agent as a daemonset with useHostPort: true and have query talk to the host directly...

mmshaikh88 · 2019-05-10T21:46:07Z

As long as the agent with specific IP/Port is reachable from the Pod then yes you can configure it.

naseemkullah · 2019-05-10T22:21:53Z

As long as the agent with specific IP/Port is reachable from the Pod then yes you can configure it.

Thanks, do you know what the ENV_VAR would be?

mmshaikh88 · 2019-05-10T23:32:46Z

Which ENV_VAR? Can you please clarify and elaborate a bit more?

naseemkullah · 2019-05-10T23:55:18Z

You mentioned we could configure it so I presumed there would be an environment variable for query to get query to talk to an agent that is not on local host.

jpkrohling · 2019-05-24T13:52:25Z

I don't think there's a way to customize the agent location for query. See jaegertracing/jaeger#1044.

naseemkullah · 2019-05-30T10:42:14Z

Hi @jpkrohling can you please confirm if we stil’ Need to add a agent sidecar to query in version 1.12 of jaeger? If so I will add it to the helm chart for jaeger via helm/charts#13693 (comment)

jpkrohling · 2019-06-03T08:58:07Z

@naseemkullah: yes, in order to have spans generated at the query component to reach the collector, it needs the agent as sidecar.

naseemkullah · 2019-06-03T12:17:50Z

Thank you for confirming @jpkrohling , an optional agent sidecar for query has been added to the jaeger helm chart.

jpkrohling closed this as completed Jan 28, 2019

jpkrohling added the question label Jan 28, 2019

jpkrohling reopened this Jan 28, 2019

naseemkullah mentioned this issue May 10, 2019

[incubator/jaeger] Adding agent sidecar for query and OWNERS file, update jaeger to 1.12 helm/charts#13693

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jaeger query shows error in log #112

Jaeger query shows error in log #112

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

jpkrohling commented Jan 28, 2019

yurishkuro commented Jan 28, 2019

jpkrohling commented Jan 28, 2019

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

jpkrohling commented Jan 29, 2019

yurishkuro commented Jan 29, 2019

naseemkullah commented May 10, 2019

mmshaikh88 commented May 10, 2019

naseemkullah commented May 10, 2019

mmshaikh88 commented May 10, 2019

naseemkullah commented May 10, 2019

jpkrohling commented May 24, 2019 •

edited

Loading

naseemkullah commented May 30, 2019

jpkrohling commented Jun 3, 2019

naseemkullah commented Jun 3, 2019

Jaeger query shows error in log #112

Jaeger query shows error in log #112

Comments

mmshaikh88 commented Jan 28, 2019 • edited by jpkrohling Loading

mmshaikh88 commented Jan 28, 2019 • edited by jpkrohling Loading

jpkrohling commented Jan 28, 2019

yurishkuro commented Jan 28, 2019

jpkrohling commented Jan 28, 2019

mmshaikh88 commented Jan 28, 2019 • edited by jpkrohling Loading

mmshaikh88 commented Jan 28, 2019 • edited by jpkrohling Loading

mmshaikh88 commented Jan 28, 2019 • edited by jpkrohling Loading

jpkrohling commented Jan 29, 2019

yurishkuro commented Jan 29, 2019

naseemkullah commented May 10, 2019

mmshaikh88 commented May 10, 2019

naseemkullah commented May 10, 2019

mmshaikh88 commented May 10, 2019

naseemkullah commented May 10, 2019

jpkrohling commented May 24, 2019 • edited Loading

naseemkullah commented May 30, 2019

jpkrohling commented Jun 3, 2019

naseemkullah commented Jun 3, 2019

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

mmshaikh88 commented Jan 28, 2019 •

edited by jpkrohling

Loading

jpkrohling commented May 24, 2019 •

edited

Loading