Use DaemonSet by default instead of sidecar

[tejaswiniVadlamudi]

Hello,

Just curious to know the reasons & advantages of using Jaeger agent as a side car instead Daemon set.

Many Thanks,
Teja

[ledor473]

It's important to note that using a Service with a DaemonSet does not send the traffic only to the local Pod:

Service: Create a service with the same Pod selector, and use the service to reach a daemon on a random node. (No way to reach specific node.)
Reference

Considering the Jaeger Agent receives UDP traffic, it's important to consider that and use the NodeIP and a known NodePort.
Something like what is described here

[yurishkuro]

Why mix Service and DaemonSet? Can't the latter be defined without being a Service?

[ledor473]

Yes DaemonSet can (and should in this case) be used without a Service!

I'm just pointing out that the client library cannot send the traces to the Jaeger Agent using localhost. Each Pods spec needs to contain something like this:

env:
- name: JAEGER_AGENT_HOST
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP

One could easily be tempted to create a Service over the DaemonSet to have a consistent name like jaeger-agent.svc.cluster.local but that would result in sending UDP packet over the network.

[yurishkuro]

we should probably have docs pointing this out. E.g. this repo is for k8s templates for running Jaeger itself, but we could also have a section on setup of the applications.

[tejaswiniVadlamudi]

I belive its clear that we send UDP packet over network if we deploy "Jaeger Agent" as a daemonset. But at same time if we deploy it as a sidecar, we need to think about Compute Resources & cost affected based on this kind of implementation. Even now, I am not clear about correct approach apparently. We have a plan to use Jaeger on K8s, we would really appreciate if you guys can come up with final solution towards deployment.

[pavolloffay]

One reason to use sidecar might be scaling. I am not sure if deamonset can scale up pods.

[yurishkuro]

I cannot give an authoritative advice for kubernetes because we don't use it at Uber. We designed the agent to be run as a daemon (one per physical host). It was not meant to be run as a side car (one per pod), because there are no perf benefits to it (apparently there's a config benefit on k8s), but there are downsides, such as exploding the number of connections to collectors.

[pieterlange]

The solution proposed by @ledor473 looks good (workaround for not being able to reach localhost from kubernetes pods)

[tejaswiniVadlamudi]

Hi, it would not be good idea to scale daemonset on K8s. Do we really need to scale agent on a Node?

I believe if it is daemon set, every application running on that Node can use it. Only problem is "if we have a single daemon set, does it have any bandwidth overhead/performance impacts and also configuration impacts(This is really important when we have plenty of Apps which need different configuration on Agent, will this work)". Two advantages I see as a DaemonSet are "1. No additional Compute Resources saves me cost 2. Daemonset can be used for full stack tracing i.e Application + K8s"

@yurishkuro Could you please let us know your views. I just started with Jaeger, not an expert.
Could you quote a real time example of your K8s clients who deployed Agent as DaemonSet. Would love to know their feedback. In case if we loose a UDP packet, what kind of impact do we have? To me, I see Jaeger as some monitoring tool & its okay even if we loose a packet or soo, as it is not involving with application's real traffic.

Thanks!

[ledor473]

@tejaswiniVadlamudi To be clear, I don't think either solution are bad.

They both need a change to each and every Deployment to be working correctly (but it could be fixed by an Admission Controller):

• Sidecar: all Deployment need the Jaeger Agent container
• DaemonSet: all Deployment need the proper environment variable to keep UDP from going to a different node (could result in broken traces)

Depending on how you manage your Deployment, maybe one change is easier than the other?

Otherwise, you effectively just trade the isolation of a sidecar (a bad actor can only impact it's own Jaeger Agent) with the lower resources usage of a DaemonSet

[tejaswiniVadlamudi]

Also @pieterlange Is there any proposal document from your side to deploy agent as a sidecar?
It would be good to have that in place.

[jpkrohling]

The reasoning for this change was explained here:

https://medium.com/jaegertracing/deployment-strategies-for-the-jaeger-agent-1d6f91796d09

Basically, both scenarios (DaemonSet, SideCar) are supported, it only depends on what your requirements are. If you have a multi tenant setup, for instance, you might want to have each pod to have its agent, so that each agent knows where to send the collected spans to. On a daemon set, the agent will send all spans collected by all pods running on that node to a single backend.

[yurishkuro]

@jpkrohling wouldn't you agree that the prevailing use case for most orgs is a single tenancy deployment? I think the templates should reflect that. Ideally the template would be parameterized to support both the daemon set and the side car deployments, defaulting to the former.

It's also worth mentioning that for multi-tenant deployments the side car approach is not the golden solution either, more like a current workaround, so I'm quite sympathetic with the users who find it confusing that the template defaults to that mode.

[pieterlange]

I think the default usecase will be single-tenant clusters.. the sidecar scenario is fine too but should probably be configured using pod presets (https://kubernetes.io/docs/concepts/workloads/pods/podpreset/) so users don't have to modify their deployments.

[jpkrohling]

I sincerely do not know what would be the most common case. Based on my experience with OpenShift, I have guessed that multiple Jaeger installations would exist under the same Kubernetes cluster, but if one installation is the common scenario, I have no problems at all changing the template.

I have added this to my queue and will be working on it soon!

I think the default usecase will be single-tenant clusters.. the sidecar scenario is fine too but should probably be configured using pod presets (https://kubernetes.io/docs/concepts/workloads/pods/podpreset/) so users don't have to modify their deployments.

Thanks for the tip about pod presets. I wasn't aware of it! Is it new?

[jpkrohling]

@pieterlange : would you be willing to contribute with a PodPreset PR for our Kubernetes templates? I have some questions about it, which you might already know the answer, so, it's probably easier/faster for you to come up with a great solution.

For instance: I have read the doc about PodPreset and it looks like it decorates the target pod based on labels. This means that the deployment would have to be changed anyway, as there's no standard label that we could make use of. For instance, we could say that we'd add agents only to pods annotated with jaeger: true (or traced: true, or any other specific label for us). I'm not sure we can do a PodPreset without a restriction, but even if we can, I don't think it's a good idea.

For now, I'm just reverting then the change I made when I removed the DaemonSet from the template.

[pieterlange]

I'm not using jaeger yet, but when i do i'll make a pull request to cover multiple usecases. (Daemonset, pod preset, ??)

[jpkrohling]

I'm working on the DaemonSet right now, so, that part should be covered. I'd appreciate a review though, if you'd be up to :)

[jpkrohling]

@pieterlange, could you test/review this PR: #75 ?

[pieterlange]

Sure.

[RouR]

In case of DaemonSet - How to secure nodePort connections from outside of cluster?
I guess that sidecar is more secure

[jpkrohling]

@RouR that would be part of your deployment architecture and you'd probably have appropriate firewall rules in place.