fix duplicate counted metrics like op time for GpuCoalesceBatches

[binmahone]

We observed that op time metrics for GpuCoalesceBatches is often larger than expected.
After some digging I found that in some cases GpuMetrics will be duplicately counted. (a test case example is shown in the PR)

I tried to to fix the duplicate issue GpuCoalesceBatches, but I have no idea how many other duplicates are there for other operators. So I refined NvtxWithMetrics, MetricRange and GpuMetric#ns a little bit to avoid duplicate counting elapsed time.

closes #11063

[binmahone]

build

[revans2]

Would it be simpler to have each GpuMetric track itself? This feels like we are adding in a lot of overhead with the ThreadLocal values.

sealed abstract class GpuMetric extends Serializable {
  def value: Long
  def set(v: Long): Unit
  def +=(v: Long): Unit
  def add(v: Long): Unit

  private var isTimerActive = false

  final def tryActivateTimer(): Bool = {
    if (!isTimerActive) {
      isTimerActive = true
      true
    } else {
      // output a warning if this is not NoopMetric
      false
    }
  }

  final def deactivateTimer(duration: Long): Unit = {
    if (isActive) {
      isActive = false
      add(duration)
    }
  }

  final def ns[T](f: => T): T = {
     if (tryActivateTimer()) {
      val start = System.nanoTime()
      try {
        f
      } finally {
        deactivateTimer(System.nanoTime() - start)
      }
    }
  }
}

[revans2]

I should add that this is mostly about code complexity. I don't think the performance difference will be significant.

[binmahone]

Hi Bobby, refined the PR accordingly. Threadlocal removed

[binmahone]

build