jaeger integration substantially harms search performance

[emidoots]

I have evidence that enabling "useJaeger": true substantially harms search performance on single-node deployments with the Jaeger deployment described in https://github.com/sourcegraph/deploy-sourcegraph-docker

It is possible the performance implications are not as severe in deployments that:

1. Are spread across more machines / not ran on a single beefy machine.
2. Are connected to a more aggressively tuned (discarding more spans), or beefier, Jaeger deployment.

To confirm my findings, run a search performance benchmark before and after setting "useJaeger": true and measure the change. I will share my results here soon.

[beyang]

Underlying issue: search fan-out generates a lot of Jaeger spans for a given trace. Network I/O becomes flooded/bottleneck.

Subsampling may work? Need to try it, because downsampling may happen at the collector, rather than the agent. (< Need to answer this question.) Traffic is generated both from client to agent and from agent to collector. Other options if subsampling doesn't work:

• Modify client code to discard traces if we exceed a certain threshold per unit time
• Add some a URL parameter (and internally a context item) to turn traces on or off
  • https://github.com/jaegertracing/jaeger-client-go#debug-traces-forced-sampling
• Explore more Jaeger config (sampling strategies, Discuss post-trace (tail-based) sampling jaegertracing/jaeger#425)

Test plan: try it out with Jaeger on and off and see if there's an perf discrepancy

• Test with and without indexed search code path
• Test with commit search
• Test with multiple replicas of searcher
• Test on both Docker Compose (single node) and Kubernetes (multi-node)
  • Docker Compose: all the network traffic gets generated from a single node
  • Kubernetes: you'll get more network bandwidth with more nodes (and less traffic from each node), but more traffic will get generated with more replicas (e.g., of searcher)

[beyang]

Closing given we now have the selective tracing option.