Explore an autoscaling Elasticsearch output based on the APM Server autoscaling work

[cmacknz]

APM Server now has an autoscaling Elasticsearch output (specialized to the APM data model). We should explore porting their implementation into the shipper with the goal of eliminating the need to manually tune the number of workers and the output batch size.

The initial APM autoscaling implementation was done in elastic/apm-server#9393 and linked to the following tracking issues:

• https://github.com/elastic/apm-dev/issues/820
  • Autoscale number of modelindexers to increase throughput and ensure full resource usage apm-server#9181

When we begin this work we should sync with the APM Server team to come up to speed on the latest state of their autoscaling implementation.

[pierrehilbert]

@cmacknz & @leehinman: what would be the expected outcome of this one?

[cmacknz]

A proof of concept of an autoscaling Elasticsearch output based on the work the APM team has done.

The ideal outcome is a demo showing that more Elasticsearch output workers are created as the rate of shipper data ingestion increases, with the number of workers then decreasing as the data ingestion rate decreases.

[cmacknz]

The Elasticsearch output used by APM server has been moved into a separate package: https://github.com/elastic/go-docappender

We should explore if we can just reuse this directly.