Akka tracing instrumentation

[codefromthecrypt]

Historically, https://github.com/levkhomich/akka-tracing has been the answer for akka in zipkin. This project has recently gone quiet, which has impacted users looking for a solution. It would be great for this project to pick-up again, and likely that would imply multiple owners so that the same doesn't recur.

Meanwhile, there are some other works related to akka tracing on github:

• https://github.com/bizreach/play-zipkin-tracing - play zipkin tracing using brave 4.x
• https://github.com/lucidsoftware/akka-thread-context - thread context similar to brave's
• https://github.com/FindHotel/akka-stream-trace - akka stream tracing with htrace
• https://github.com/gallathanmdsol/tracing - akka experiment using brave 3.x
• https://github.com/lloydmeta/zipkin-futures - future tracing (play support) using brave 2.x

There have been zipkin users spending time with mixed success with a goal of tracing akka, which is the motivation of this issue. Notably, @metamorph @kriloleg @tsteff @capitanbatata @ShaneDelmore @andersenleo @lawrencefinn. Those using play often ended using @takezoe's https://github.com/bizreach/play-zipkin-tracing for a combination of reasons including fast turnarounds on support and questions, and being present on gitter.

There are a number of users successfully using akka-tracing, including @sandeep-paliwal in an open pull request for openwhisk. An ideal solution might be a sustainable setup for akka-tracing which provides for multiple people contributing to it. To be fair, @levkhomich has done a stellar job, starting this work and updating the project a few months ago, and responding to questions as recently as last month. Whichever direction taken, we owe a great deal of credit to Lev.

So, what do people think about this topic?

[ShaneDelmore]

@adriancole Kamon committers @dpsoft and @ivantopo have also done some work in this area and I believe are desiring to stay compatible.

[LarryFinn]

@adriancole i think it would be nice. i couldn't fully use the play-zipkin-tracing lib because of compatibility issues and im pretty sure it doesn't automatically pass the traceid between futures via the default dispatcher or default execution context. the akka-thread-context partially solves this but doesnt solve the problem of the default execution context. or maybe i did something wrong...

[codefromthecrypt]

Whatever fills the akka need, if it employs the CurrentTraceContext (directly or indirectly via Tracer.withSpanInScope), it could automagically integrate with SLF4J or Finagle's context synchronization libraries.

[andersenleo]

We built our Akka tracing to avoid any explicit, hard-coded, tracing in application code. Since we're in control over the entire stack our solution isn't easily portable to a generic "library". Note that it's currently based on the Brave3 API.

I'll try to describe it anyway - hoping that it can be partially reusable :-)

All our actors extends a common super-class. And we've got a central factory where Props for actors are created. This allows us to create the actor with a custom UnboundedMailboxWithTracing configuration.

The mailbox queue will intercept all messages enqueued and wrap the message in a TracedMessage that holds the current ServerSpan from the thread-local Brave3 context.

Our custom super-class will handle all incoming TracedMessages, unwrap them and set the ServerSpanThreadBinder#setCurrentSpan during processing of the wrapped message.

This works in most cases - but some of akkas internal actors uses mailboxes/message-queues that don't enqueue on the same thread as the sender - and in that case there is no tracing context available. This is no big deal for us since we're only interested in tracing application level actors.

Somewhere in the future we'll look into migrating to Brave4 - but we're not there yet.

[dpsoft]

Hi, we already went through the process of making our tracer a OpenTracing compliant one, but that is not quite the same as Brave ... if you have are using Kamon you will be able to capture the trace data and send it to Zipkin and benefit from all available instrumentation in Kamon + any Opentracting instrumentation out there, but we think Zipkin/Brave are another world. Anyway, If they are curious about the kamon-akkamodule that is based on OpenTracing API, can look here and here for kamon-play(WIP).

[codefromthecrypt]

Hi, we already went through the process of making our tracer a OpenTracing compliant one, but that is not quite the same as Brave ... if you have are using Kamon <https://github.com/kamon-io/Kamon/tree/kamon-1.0-develop> you will be able to capture the trace data and send it to Zipkin and benefit from all available instrumentation in Kamon + any Opentracting instrumentation out there, but we think Zipkin/Brave are another world. Anyway for the

Curious.. I don't see any opentracing types in kamon or kamon-akka. It seems maybe the tracer is inspired by OpenTracing, but not OpenTracing, right? This isn't terribly important here, but just wondering if I'm missing something or looking at the wrong repos.

If they are curious about the kamon-akkamodule that is based on OpenTracing API, can look here <https://github.com/kamon-io/kamon-akka/tree/kamon-1.0> ;)

This isn't opentracing, but it is helpful as your tracer api is very similar to Brave's. Thanks!

[dpsoft]

@adriancole plese take a look to kamon-core in the branch kamon-1.0-develop which is the one that implements OpenTracing, and yes is influenced by Brave ;)

[codefromthecrypt]

@adriancole <https://github.com/adriancole> plese take a look to kamon-core <https://github.com/kamon-io/Kamon/blob/kamon-1.0-develop/kamon-core/src/main/scala/kamon/trace/Tracer.scala#L31-L32> in the branch kamon-1.0-develop which is the one that implements OpenTracing.

thx @ivantopo pointed it out to me a little while ago, too

[codefromthecrypt]

long discussion on gitter and seems one supportable route would be to enhance the existing play library. Here's a tracking issue: openzipkin-contrib/play-zipkin-tracing#21

This wouldn't preclude any other options including the upcoming work in kamon (there is more than one answer sometimes: that's ok as long as at least one is maintained!)

[llinder]

This is something I've been interested in using for some time. I'm primarily interested in Akka HTTP but having comprehensive tracing support for Scala Future/ExecutionContext, Akka and Akka HTTP would be really nice. I honestly haven't looked at bizreach/play-zipkin-tracing, mostly because it sounds focused entirely on the Play framework but maybe that isn't the case?

Anyway, I'm going to try and must up some spare cycles to catch up on Akka instrumentation and hopefully contribute were there are gaps.

[codefromthecrypt]

I honestly haven't looked at bizreach/play-zipkin-tracing, mostly because it sounds focused entirely on the Play framework but maybe that isn't the case?

it definitely started as this. One thing to consider is that @takezoe is doing a great job like others in working together on problems. In other words, you might have a buddy in maintaining if collaborating there. If you see something that needs to change, I would dare to propose it! Once there's more community around this sort of instrumentation we can still take any next steps, and whatever you decide makes sense, let's make sure all are aware (to help you not end up on your own)

Anyway, I'm going to try and must up some spare cycles to catch up on Akka instrumentation and hopefully contribute were there are gaps.

Very excited and grateful

[asereda-gs]

Some people mentioned libraries which are using code instrumentation / byte-code modification to achieve tracing (and telemetry) of akka:

1. Kamon - Apache 2.0 license
2. Cinnamon - Commercial offering by Lightbend

The reason I prefer this technique is because it is more transparent and less intrusive than modifying all existing code (Actor / ActorRef / Message etc.). However, I'm not a big fan of java agents since they may conflict among each-over and introduce an operational burden (more complicated unit / integration tests, you have to initialize them before other classes etc.)

What people in this group think about it ?

I'm trying to develop a solution which doesn't impose code modification nor use of java agents (custom Mailboxes / ActorRefProviders etc.) will let you know if it succeeds

[LarryFinn]

i've been silent on this thread (sorry). i tend to find code instrumentation via java agent to be a bit worrisome in production. i've seen some tools actually cause the JVM to crash, which is pretty bad.

[asereda-gs]

After talking to @adriancole and @takezoe we'll try to explore some ideas in separate repository created under openzipkin-contrib. Contributors are welcome.

[codefromthecrypt]

ok I opened https://github.com/openzipkin-contrib/brave-akka, adding @asereda-gs and @takezoe as write with @llinder as admin (as he is very familiar with our release processes etc)

Keep us posted!

[jkubrynski]

Why is this issue closed as it's not fixed?

[codefromthecrypt]

This issue was moved to openzipkin-contrib/brave-akka#1

[codefromthecrypt]

@jkubrynski it was closed because it won't be fixed in this repo. @asereda-gs had some work in progress not sure what happened with it, but it or similar would end up in brave-akka