Expose JMX metrics in Kamon prometheus exporter

[sebarys]

Hello,

I would like to expose some JMX metrics by prometheus exporter (https://kamon.io/docs/latest/reporters/prometheus/).
Particularly I'm interested in Slick/Hikari metrics - they can be reported to JMX when configured in such way (https://scala-slick.org/doc/3.3.3/config.html#monitoring).

Is it possible? I would prefer to not add instrumentation agent if possible. Can I just expose those metrics in such way?

[ivantopo]

There is no automatic way to turn JMX attributes into metrics in Kamon 2.x. We used to have something like that in 1.x, but we never got to upgrade it. I'm personally not a fan of using JMX to expose metrics because the only things that sort of make sense to expose are monotonic counters and gauges, and there were not that many people using it so we will probably not upgrade ourselves (open to contributions, though).

That said, I wanted to see how hard would it be to take the values exposed by Slick's AsyncExecutorMXBean and turn them into Kamon gauges that get exposed to every reporter. Turns out it is not that complicated! It all boils down to these two functions:

  def registerBean(): Unit = {
    val slickBeans = new ObjectName("slick:type=AsyncExecutor,*")

    ManagementFactory
      .getPlatformMBeanServer
      .queryMBeans(slickBeans, null)
      .asScala
      .foreach(objectInstance => {
        val executorName = objectInstance.getObjectName.getKeyProperty("name")
        val proxy = JMX.newMBeanProxy(
          ManagementFactory.getPlatformMBeanServer,
          objectInstance.getObjectName,
          classOf[AsyncExecutorMXBean]
        )

        registerAsyncExecutorMetrics(proxy, executorName)
      })
  }

  def registerAsyncExecutorMetrics(asyncExecutorMXBean: AsyncExecutorMXBean, executorName: String): Unit = {
    Kamon.gauge("slick.executor.queue-size.limit")
      .withTag("name", executorName)
      .autoUpdate(g => g.update(asyncExecutorMXBean.getMaxQueueSize): Unit)

    Kamon.gauge("slick.executor.queue-size.current")
      .withTag("name", executorName)
      .autoUpdate(g => g.update(asyncExecutorMXBean.getQueueSize): Unit)

    Kamon.gauge("slick.executor.threads.limit")
      .withTag("name", executorName)
      .autoUpdate(g => g.update(asyncExecutorMXBean.getMaxThreads): Unit)

    Kamon.gauge("slick.executor.threads.active")
      .withTag("name", executorName)
      .autoUpdate(g => g.update(asyncExecutorMXBean.getActiveThreads): Unit)
  }

The first function finds all registered AsyncExecutorMXBean instances and creates a proxy to them, that's all JMX-specific code. The second one is the interesting one, it does three things:

• Create a metric for each exposed attribute
• Add the name tag to match the AsyncExecutor name
• Use Kamon's autoUpdate to automatically get the values from the MBean and update Kamon's metrics

I got it to run for a couple minutes and send the metrics to Kamon APM and got this data for example:

Just using kamon-core and the APM reporter. If you are doing Prometheus it should also work just fine. The entire example is on this gist.

[sebarys]

Thanks @ivantopo ! I wasn't aware about autoUpdate capability :)
below code for both asyncExecutor and hikari related beans if someone will be interested

    val slickBeans = new ObjectName("slick:type=AsyncExecutor,name=*")
    val hikariPoolConfigBeans = new ObjectName("com.zaxxer.hikari:type=PoolConfig (*)")
    val hikariPoolBeans = new ObjectName("com.zaxxer.hikari:type=Pool (*)")
   
   // asyncExecutor is defined above in @ivantopo answer

  // hikari config - only static values but maybe someone will need this
  ManagementFactory
      .getPlatformMBeanServer
      .queryMBeans(hikariPoolConfigBeans, null)
      .asScala
      .foreach(objectInstance => {
        val hikariConfigMXBeanProxy =
          JMX.newMBeanProxy(ManagementFactory.getPlatformMBeanServer, objectInstance.getObjectName, classOf[HikariConfigMXBean])
        Kamon
          .gauge("hikari.pool.config.idle-timeout")
          .withTag("pool-name", hikariConfigMXBeanProxy.getPoolName)
          .autoUpdate(g => g.update(hikariConfigMXBeanProxy.getIdleTimeout.toDouble): Unit)

        Kamon
          .gauge("hikari.pool.config.connection-timeout")
          .withTag("pool-name", hikariConfigMXBeanProxy.getPoolName)
          .autoUpdate(g => g.update(hikariConfigMXBeanProxy.getConnectionTimeout.toDouble): Unit)

        Kamon
          .gauge("hikari.pool.config.max-pool-size")
          .withTag("pool-name", hikariConfigMXBeanProxy.getPoolName)
          .autoUpdate(g => g.update(hikariConfigMXBeanProxy.getMaximumPoolSize.toDouble): Unit)
      })

   // hikari pool
    ManagementFactory.getPlatformMBeanServer
      .queryMBeans(hikariPoolBeans, null)
      .asScala
      .foreach(objectInstance => {
        val configType = objectInstance.getObjectName.getKeyProperty("type")
        val hikariPoolMXBeanProxy =
          JMX.newMBeanProxy(ManagementFactory.getPlatformMBeanServer, objectInstance.getObjectName, classOf[HikariPoolMXBean])

        Kamon
          .gauge("hikari.pool.idle-connections")
          .withTag("name", configType)
          .autoUpdate(g => g.update(hikariPoolMXBeanProxy.getIdleConnections.toDouble): Unit)

        Kamon
          .gauge("hikari.pool.total-connections")
          .withTag("name", configType)
          .autoUpdate(g => g.update(hikariPoolMXBeanProxy.getTotalConnections.toDouble): Unit)

        Kamon
          .gauge("hikari.pool.active-connections")
          .withTag("name", configType)
          .autoUpdate(g => g.update(hikariPoolMXBeanProxy.getActiveConnections.toDouble): Unit)

        Kamon
          .gauge("hikari.pool.threads-awaiting-connections")
          .withTag("name", configType)
          .autoUpdate(g => g.update(hikariPoolMXBeanProxy.getThreadsAwaitingConnection.toDouble): Unit)
      })

btw: AsyncExecutor bean is registered lazily so potentially you would need kind of await around

ManagementFactory
      .getPlatformMBeanServer
      .queryMBeans(slickBeans, null)
      .asScala
      .foreach ...
```  until desired bean is present before you consider registration finished with success.

[ivantopo]

Thanks for sharing!