RemoteControlledSampler fails when Agent is on a different host from the application

[hannahchan]

When the JAEGER_AGENT_HOST environment variable is set to a host other than the default localhost, the Jaeger.Samplers.RemoteControlledSampler fails with the following exception;

warn: Jaeger.Samplers.RemoteControlledSampler[0]
      Updating sampler failed
System.Net.Http.HttpRequestException: Cannot assign requested address
 ---> System.Net.Sockets.SocketException (99): Cannot assign requested address
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Jaeger.Util.DefaultHttpClient.MakeGetRequestAsync(String requestUri)
   at Jaeger.Samplers.HttpSamplingManager.GetSamplingStrategyAsync(String serviceName)
   at Jaeger.Samplers.RemoteControlledSampler.UpdateSampler()

I think it's trying to read the sampling configuration of the agent. When I change to the const sampler by setting the following environment variables, the agent starts receiving traces.

JAEGER_SAMPLER_TYPE: const
JAEGER_SAMPLER_PARAM: 1

This confirms that the network setup for the application and agent is correct and that the issue is in the RemoteControlledSampler class.

I tested this by writing a docker-compose file that spun up my application and the Jaeger agent as separate containers on the same virtual network.

[Falco20019]

Hi @hannahchan, thanks for your report.

I'm sadly pretty occupied this and next week with another project. But I will try to fix it ASAP afterwards. Feel free to prepare a pull request for it if you have an assumption what's causing it.

[Falco20019]

I just gave it a quick look because it sounded strange that it works with const sampler.

What sampler do you intend to use? If you have not defined any sampler yourself, the default sampler is used. That's an remote controlled sampler whose endpoint you can configure using JAEGER_SAMPLER_MANAGER_HOST_PORT and which defaults to localhost:5778.

So if you intend to use a const, probabilistic or rate limited sampler, make sure to also set the correct parameters for JAEGER_SAMPLER_TYPE and JAEGER_SAMPLER_PARAM.

Please see https://www.jaegertracing.io/docs/1.16/client-features/ & https://www.jaegertracing.io/docs/1.16/sampling/ & https://github.com/jaegertracing/jaeger-client-csharp/blob/master/src/Jaeger/Samplers/README.md for more information on samplers.

Depending on your docker-compose.yml, I assume you wanted to stick with the default in jaeger-agent and just need to set JAEGER_SAMPLER_MANAGER_HOST_PORT to jaeger-agent:5778.

[hannahchan]

I can confirm that setting the environment variable JAEGER_SAMPLER_MANAGER_HOST_PORT to localhost:5778 works. However I thought that just setting JAEGER_AGENT_HOST should work and the RemoteControlledSampler would know to read its configuration from port 5778 by default. It's confusing because switching out the RemoteControlledSampler for the const sampler does sent traces to the agent.

I'm not too fussed this and there's no rush for me to get this fixed. In production we actually deploy the agent as a side car to our app in K8s with no environment variables set other than JAEGER_SERVICE_NAME. That way it defaults to sending traces to the localhost of the pod and uses the default RemoteControlledSampler. The docker-compose file we have is just to help other developers run the environment locally. However when writing it is when we discovered the behaviour described above.

[Falco20019]

Hm, technically it's two different things at two different end points. But I assume you are right, that usually one would assume that just changing the agent would assume to also change the endpoint as it's mostly the same with just another port. I will look into it :)

[Falco20019]

It is related to jaegertracing/jaeger#1971 (comment)

The general consent is there to keep the variables decoupled, so it wouldn‘t be good to break that consent with this change. I will make another PR to resemble the consent here. But this means the default way is to specify the config endpoint in the separate variable as explained before.

I will make sure to explain that more in the documentation to avoid confusion in the future.