Metrics with different sets of labels can't be registered

[seglo]

Hi @anbarasantr. I've run into a problem with the feature introduced in PR #61. It appears that the Java prometheus client library does not let you register metrics more than once with different sets of labels. I created this PR which recreates the issue, by modifying one of your tests to add a second set of custom labels for another cluster definition. It results in the following exception (see the inner exception).

Could not create Prometheus Endpoint
java.lang.Exception: Could not create Prometheus Endpoint
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.$anonfun$apply$2(PrometheusEndpointSink.scala:23)
	at scala.util.Failure.fold(Try.scala:240)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.apply(PrometheusEndpointSink.scala:23)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSinkTest.$anonfun$new$5(PrometheusEndpointSinkTest.scala:60)
	at org.scalatest.fixture.TestSuite$TestFunAndConfigMap.$anonfun$apply$1(TestSuite.scala:132)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.fixture.TestSuite$TestFunAndConfigMap.apply(TestSuite.scala:132)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSinkTest.withFixture(PrometheusEndpointSinkTest.scala:31)
	at org.scalatest.fixture.FreeSpecLike.invokeWithFixture$1(FreeSpecLike.scala:540)
	at org.scalatest.fixture.FreeSpecLike.$anonfun$runTest$1(FreeSpecLike.scala:551)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
	at org.scalatest.fixture.FreeSpecLike.runTest(FreeSpecLike.scala:551)
	at org.scalatest.fixture.FreeSpecLike.runTest$(FreeSpecLike.scala:532)
	at org.scalatest.fixture.FreeSpec.runTest(FreeSpec.scala:232)
	at org.scalatest.fixture.FreeSpecLike.$anonfun$runTests$1(FreeSpecLike.scala:594)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:373)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:410)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
	at org.scalatest.fixture.FreeSpecLike.runTests(FreeSpecLike.scala:594)
	at org.scalatest.fixture.FreeSpecLike.runTests$(FreeSpecLike.scala:593)
	at org.scalatest.fixture.FreeSpec.runTests(FreeSpec.scala:232)
	at org.scalatest.Suite.run(Suite.scala:1147)
	at org.scalatest.Suite.run$(Suite.scala:1129)
	at org.scalatest.fixture.FreeSpec.org$scalatest$fixture$FreeSpecLike$$super$run(FreeSpec.scala:232)
	at org.scalatest.fixture.FreeSpecLike.$anonfun$run$1(FreeSpecLike.scala:615)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
	at org.scalatest.fixture.FreeSpecLike.run(FreeSpecLike.scala:615)
	at org.scalatest.fixture.FreeSpecLike.run$(FreeSpecLike.scala:614)
	at org.scalatest.fixture.FreeSpec.run(FreeSpec.scala:232)
	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1346)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1340)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1340)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
	at org.scalatest.tools.Runner$.run(Runner.scala:850)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: kafka_partition_latest_offset
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
	at io.prometheus.client.SimpleCollector$Builder.register(SimpleCollector.java:246)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.$anonfun$metrics$4(PrometheusEndpointSink.scala:38)
	at scala.collection.immutable.List.map(List.scala:286)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.$anonfun$metrics$1(PrometheusEndpointSink.scala:33)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	at scala.collection.immutable.Map$Map2.foreach(Map.scala:159)
	at scala.collection.TraversableLike.map(TraversableLike.scala:237)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink.<init>(PrometheusEndpointSink.scala:31)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.$anonfun$apply$1(PrometheusEndpointSink.scala:22)
	at scala.util.Try$.apply(Try.scala:213)
	at com.lightbend.kafkalagexporter.PrometheusEndpointSink$.apply(PrometheusEndpointSink.scala:22)
	... 47 more

It seems that adding dynamic labels in this manner is an anti-pattern in prometheus that should be handled with relabeling rules in Prometheus server config instead. Whether or not we agree with this assessment I'll have to rollback this feature until another solution is found, and cut a new release. Based on the info in the following ticket, it's possible that you could create a custom metrics collector that works around the problem.

prometheus/client_java#121

[seglo]

Another idea is to let users configure arbitrary labels for all clusters and then configure custom label values for each cluster if they wish (otherwise they'll just be blank).

[anbarasantr]

@seglo Thanks for the explanation. It was an oversight from my part. I did not test with multiple cluster setup.
I still think this would be benefit the user and I also like the idea that you suggested. I will first try to see if there are any alternative approaches around.

[seglo]

Resolved with #82