ClassNotFoundException when starting multi-project sbt Play 2.8 app

[gwak]

Hi,

We are building a multi-project sbt app with Play 2.8.8. I'm trying to follow your instructions for installing Kamon and report metrics and traces to APM with no luck.

// I've added the following dependencies:
object Metrics {
    val kamonBundle = "io.kamon" %% "kamon-bundle" % "2.2.3"
    val kamonApmReporter = "io.kamon" %% "kamon-apm-reporter" % "2.2.3"
  }
// And sbt Kanela plugin:
addSbtPlugin("io.kamon" % "sbt-kanela-runner-play-2.8" % "2.0.10")

// In build.sbt
lazy val `myapp` = (project in file("myapp"))
  .enablePlugins(play.sbt.PlayScala, JavaAgent)
  .settings(MyApp.settings: _*)

I first had issues with specifying the application configuration file in Run mode. After a lot of searching through the horrendous Gitter channel I finally found that I needed to use:

-Dconfig.file=my-app/conf/application-local.conf
# Instead of 
-Dconfig.resource=application-local.conf

But then it seems that the Kanela class loader doesn't find a custom Serializer class that's dynamically loaded from Akka. It's defined in the conf like that:

akka.actor {
    serializers {
      play-json = "myapp.PlayJsonSerializer"
    }

    serialization-identifiers {
      "myapp.PlayJsonSerializer" = 1000004
    }

    serialization-bindings {
      "myapp.JsonSerializable" = play-json
      "java.io.Serializable" = play-json
    }
}

But I get the error below:

[info] Running the application with the Kanela agent

 _  __                _        ______
| |/ /               | |       \ \ \ \
| ' / __ _ _ __   ___| | __ _   \ \ \ \
|  < / _` | '_ \ / _ \ |/ _` |   ) ) ) )
| . \ (_| | | | |  __/ | (_| |  / / / /
|_|\_\__,_|_| |_|\___|_|\__,_| /_/_/_/

==============================
Running with Kanela, the Kamon Instrumentation Agent :: (v1.0.8)

--- (Running the application, auto-reloading is enabled) ---

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by kamon.instrumentation.executor.ExecutorInstrumentation$ (file:/Users/cwanty/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/io/kamon/kamon-bundle_2.13/2.2.3/kamon-bundle_2.13-2.2.3.jar) to field java.util.concurrent.Executors$DelegatedExecutorService.e
WARNING: Please consider reporting this to the maintainers of kamon.instrumentation.executor.ExecutorInstrumentation$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[error] java.lang.ClassNotFoundException: myapp.PlayJsonSerializer
[error] 	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
[error] 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
[error] 	at kamon.instrumentation.sbt.KanelaOnSystemClassLoader.loadClass(KanelaOnSystemClassLoader.java:33)
[error] 	at kamon.instrumentation.sbt.play.SbtKanelaRunnerPlay$SbtKanelaClassLoader.loadClass(SbtKanelaRunnerPlay.scala:76)
[error] 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
[error] 	at java.base/java.lang.Class.forName0(Native Method)
[error] 	at java.base/java.lang.Class.forName(Class.java:398)
[error] 	at akka.actor.ReflectiveDynamicAccess.$anonfun$getClassFor$1(ReflectiveDynamicAccess.scala:29)
[error] 	at scala.util.Try$.apply(Try.scala:210)
[error] 	at akka.actor.ReflectiveDynamicAccess.getClassFor(ReflectiveDynamicAccess.scala:28)
[error] 	at akka.actor.ReflectiveDynamicAccess.createInstanceFor(ReflectiveDynamicAccess.scala:47)
[error] 	at akka.serialization.Serialization.serializerOf(Serialization.scala:390)
[error] 	at akka.serialization.Serialization.$anonfun$serializers$2(Serialization.scala:424)
[error] 	at scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
[error] 	at scala.collection.immutable.HashMapBuilder.addAll(HashMap.scala:2360)
[error] 	at scala.collection.immutable.HashMap$.from(HashMap.scala:2182)
[error] 	at scala.collection.immutable.HashMap$.from(HashMap.scala:2158)
[error] 	at scala.collection.MapOps$WithFilter.map(Map.scala:348)
[error] 	at akka.serialization.Serialization.<init>(Serialization.scala:424)
[error] 	at akka.serialization.SerializationExtension$.createExtension(SerializationExtension.scala:18)
[error] 	at akka.serialization.SerializationExtension$.createExtension(SerializationExtension.scala:14)
[error] 	at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:1158)
[error] 	at akka.actor.ActorSystemImpl.$anonfun$loadExtensions$1(ActorSystem.scala:1201)
[error] 	at scala.collection.immutable.Vector.foreach(Vector.scala:1856)
[error] 	at akka.actor.ActorSystemImpl.loadExtensions$1(ActorSystem.scala:1195)
[error] 	at akka.actor.ActorSystemImpl.loadExtensions(ActorSystem.scala:1214)
[error] 	at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:1035)
[error] 	at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:1022)
[error] 	at akka.actor.ActorSystemImpl._start(ActorSystem.scala:1022)
[error] 	at akka.actor.ActorSystemImpl.start(ActorSystem.scala:1045)
[error] 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:272)
[error] 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:316)
[error] 	at akka.actor.ActorSystem$.apply(ActorSystem.scala:290)
[error] 	at play.core.server.DevServerStart$.$anonfun$mainDev$1(DevServerStart.scala:291)
[error] 	at play.utils.Threads$.withContextClassLoader(Threads.scala:22)
[error] 	at play.core.server.DevServerStart$.mainDev(DevServerStart.scala:76)
[error] 	at play.core.server.DevServerStart$.mainDevHttpMode(DevServerStart.scala:50)
[error] 	at play.core.server.DevServerStart.mainDevHttpMode(DevServerStart.scala)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[error] 	at kamon.instrumentation.sbt.play.KanelaReloader$.startDevMode(KanelaReloader.scala:296)
[error] 	at play.sbt.run.KanelaPlayRun$.devModeServer$lzycompute$1(KanelaPlayRun.scala:63)
[error] 	at play.sbt.run.KanelaPlayRun$.devModeServer$1(KanelaPlayRun.scala:45)
[error] 	at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3(KanelaPlayRun.scala:70)
[error] 	at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$3$adapted(KanelaPlayRun.scala:31)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] stack trace is suppressed; run last Compile / run for the full output
[error] (Compile / run) java.lang.reflect.InvocationTargetException
[error] Total time: 7 s, completed Sep 15, 2021, 11:12:40 AM

An if I disable my custom Serializer, I also get an error:

Uncaught error from thread [play-dev-mode-akka.actor.default-dispatcher-5]: 'play.runsupport.Reloader$CompileResult play.sbt.run.PlayReload$.compile(scala.Function0, scala.Function0, scala.Function0)', shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play-dev-mode]
java.lang.NoSuchMethodError: 'play.runsupport.Reloader$CompileResult play.sbt.run.PlayReload$.compile(scala.Function0, scala.Function0, scala.Function0)'
	at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$4(KanelaPlayRun.scala:42)
	at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$1(KanelaReloader.scala:474)
	at kamon.instrumentation.sbt.play.KanelaReloader$$anon$1.run(KanelaReloader.scala:44)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at kamon.instrumentation.sbt.play.KanelaReloader$.kamon$instrumentation$sbt$play$KanelaReloader$$withReloaderContextClassLoader(KanelaReloader.scala:49)
	at kamon.instrumentation.sbt.play.KanelaReloader.reload(KanelaReloader.scala:474)
	at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:147)
	at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:302)
	at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:224)
	at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1418)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:775)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:790)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at kamon.instrumentation.executor.ExecutorInstrumentation$InstrumentedForkJoinPool$TimingRunnable.run(ExecutorInstrumentation.scala:678)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[error] a.a.ActorSystemImpl - Uncaught error from thread [play-dev-mode-akka.actor.default-dispatcher-5]: 'play.runsupport.Reloader$CompileResult play.sbt.run.PlayReload$.compile(scala.Function0, scala.Function0, scala.Function0)', shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play-dev-mode]
java.lang.NoSuchMethodError: 'play.runsupport.Reloader$CompileResult play.sbt.run.PlayReload$.compile(scala.Function0, scala.Function0, scala.Function0)'
	at play.sbt.run.KanelaPlayRun$.$anonfun$playRunTask$4(KanelaPlayRun.scala:42)
	at kamon.instrumentation.sbt.play.KanelaReloader.$anonfun$reload$1(KanelaReloader.scala:474)
	at kamon.instrumentation.sbt.play.KanelaReloader$$anon$1.run(KanelaReloader.scala:44)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at kamon.instrumentation.sbt.play.KanelaReloader$.kamon$instrumentation$sbt$play$KanelaReloader$$withReloaderContextClassLoader(KanelaReloader.scala:49)
	at kamon.instrumentation.sbt.play.KanelaReloader.reload(KanelaReloader.scala:474)
	at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:147)
	at play.core.server.AkkaHttpServer.handleRequest(AkkaHttpServer.scala:302)
	at play.core.server.AkkaHttpServer.$anonfun$createServerBinding$1(AkkaHttpServer.scala:224)
	at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1418)
[info] p.c.s.AkkaHttpServer - Stopping Akka HTTP server...
[info] p.c.s.AkkaHttpServer - Terminating server binding for /0:0:0:0:0:0:0:0:9000
[warn] a.a.CoordinatedShutdown - Coordinated shutdown phase [service-unbind] timed out after 5000 milliseconds
[info] p.c.s.AkkaHttpServer - Running provided shutdown stop hooks

This issue seems related to kamon-io/sbt-kanela-runner#15

If I rollback Play to 2.8.7 and remove my custom Akka serializer, it seems to be working fine, but I need that custom Serializer...

Is there something I can do ? If not, I think it should clearly be specified on your documentation website that you do not support dynamically loaded classes.

Thanks

[ivantopo]

Hey @gwak, I see three different issues here:

• The issue with the sbt-kanela-runner is due to incompatible versions. We released version 2.0.12 a few minutes ago and it works fine with Play 2.8.8. That should solve the first problem.
• The issue about Kamon not taking your configuration seems a bit weird. If Play takes it, Kamon should take it because we always reconfigure Kamon using Play's configuration. Do you maybe have a custom application loader in your app? You might need to add calls to reconfigure/loadModules/stop as shown here: https://kamon.io/docs/latest/guides/installation/play-framework/#custom-application-loaders
• The thing about the serializer, that might be a delegation issue in our custom classloaders for dev mode. Do you by any chance have a tiny reproducible we could use to troubleshoot this?