jaeger-query: "msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures" - wut?

[Dieterbe]

sometimes when i search from the UI I get:

There was an error querying for traces:
Bad JSON returned from the Jaeger Query Service.

according to network tab in devtools of the browser, the json response was:

{"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":500,"msg":"Operation failed - received 0 responses and 1 failures"}]}

and the pod logs show:

{"level":"error","ts":1506096484.0922747,"caller":"spanstore/reader.go:376","msg":"Failed to exec query","error":"Operation failed - received 0 responses and 1 failures","stacktrace":"github.com/uber/jaeger/vendor/go.uber.org/zap.Stack\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/field.go:209\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).check\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:273\ngithub.com/uber/jaeger/vendor/go.uber.org/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/go.uber.org/zap/logger.go:176\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).executeQuery\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:376\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).queryByService\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:362\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).findTraceIDs\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:280\ngithub.com/uber/jaeger/plugin/storage/cassandra/spanstore.(*SpanReader).FindTraces\n\t/home/travis/gopath/src/github.com/uber/jaeger/plugin/storage/cassandra/spanstore/reader.go:236\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).search\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/handler.go:212\ngithub.com/uber/jaeger/cmd/query/app.(*APIHandler).(github.com/uber/jaeger/cmd/query/app.search)-fm\n\t/home/travis/gopath/src/github.com/uber/jaeger/cmd/query/app/handler.go:124\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\nnet/http.(Handler).ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/h2_bundle.go:4084\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1726\ngithub.com/uber/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/uber/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/uber/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:53\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:2202\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.7.linux.amd64/src/net/http/server.go:1579"}

so the problem is:

1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)
2. the error in the log doesn't say much either. what operations failed, what's the failure? where do i look next? is there other services involved?

PS: sorry for just opening a ticket and not providing a PR. i would like to contribute more eventually but right now i'm quite busy with work.

[tiffon]

1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)

@Dieterbe I created jaegertracing/jaeger-ui#127 to address the first problem you mention.

Thanks for reporting this.

[tiffon]

1. error in UI is not really informational about what the problem is and where to look next (in particular, the returned json looks fine to me)

1 was fixed by jaegertracing/jaeger-ui#133.

[yurishkuro]

"error":"Operation failed - received 0 responses and 1 failures"

Not sure how we can do better, it seems to be the limitation of the gocql driver, because we do log the error returned from it:

	for i.Scan(&traceID) {
		retMe.Add(traceID)
	}
	err := i.Close()
	if err != nil {
		s.logger.Error("Failed to exec query", zap.Error(err))
		return nil, err
	}

[jpkrohling]

Closing this, as the UI part was done already and the backend part won't be done, as far as I understood.