[APM] Replace manual rate calculation with rate agg

[sorenlouv]

For calculating throughput we should use the rate agg.

There is currently a bug causing the rate agg to produce incorrect results for metric documents with a custom _doc_count.

We can work around this issue by using rate agg with "mode": "value_count", thus side-stepping doc count entirely. See examples below

Transaction-based

{
  "timeseries": {
    "date_histogram": {
      "field": "@timestamp",
      // ...
    },
    "aggs": {
        "throughput": {
          "rate": {
            "unit": "minute",
            "field": "transaction.duration.us",
            "mode": "value_count"
          }
        }
    }
  }
}

Metrics-based

{
  "timeseries": {
    "date_histogram": {
      "field": "@timestamp",
      // ...
    },
    "aggs": {
        "throughput": {
          "rate": {
            "unit": "minute",
            "field": "transaction.duration.histogram",
            "mode": "value_count"
          }
        }
    }
  }
}

Considerations

The could be the perf implications of using "mode": "value_count" over the custom doc count. If transaction.duration.histogram contains many buckets and we don’t use _doc_count it will be more expensive.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[axw]

The could be the perf implications of using "mode": "value_count" over the custom doc count. If transaction.duration.histogram contains many buckets and we don’t use _doc_count it will be more expensive.

It is substantially slower. I just ran an experiment, indexing 10000 transaction metric documents with 1000 buckets each. I then compared the aggregation using doc_count vs. value_count modes:

GET /metrics-apm.internal-*/_search?request_cache=false
{
  "size": 0,
  "aggs": {
    "timeseries": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "60s"
      },
      "aggs": {
        "throughput": {
          "rate": {
            "unit": "minute"
          }
        }
      }
    }
  }
}

GET /metrics-apm.internal-*/_search?request_cache=false
{
  "size": 0,
  "aggs": {
    "timeseries": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "60s"
      },
      "aggs": {
        "throughput": {
          "rate": {
            "unit": "minute",
            "mode": "value_count",
            "field": "transaction.duration.histogram"
          }
        }
      }
    }
  }
}

The first aggregation (doc_count mode) takes single digit milliseconds, while the second takes approximately 300-500ms. If elastic/elasticsearch#74359 were implemented I would expect the time to be in the same order.