Tail Based Sampling GA

[simitt]

This is a task list of necessary work for declaring tail based sampling GA. It is a follow up of #5484, where some preliminary testing was done to identify the current status.

Required Feature Work

• [tail sampling] Review behaviour when Elasticsearch license checks fail #5125
• [tail sampling] make sure apm-server has enough privileges for tail-based sampling #5490
• Document tail-based sampling #4586
• [ ] Introduce efficient codec for model events #4120 Moved out of the QA requirements
• Support TBS configuration options
  • Add TBS configuration options to apmpackage #6895
  • Add Tail Based Sampling configuration options to APM Integration kibana#121534
• Clarify sampling error logging #5101

Stretch

• model: create a specification for valid event docs #4410
• TBS support for APM Integration when using non-Elasticsearch output #7025

Identified Bugs

• Monitoring data is not indexed in ES  #6581
• Defining apm-server.sampling.tail.interval causes throughput to cease #6638
• ES search rate very high when TBS enabled #6639
• Multiple service policies result in unusual sample rates #6640
• APM Server hung in high CPU utilization #6642

Test Plan

• Verify that sampling logic is applied correctly (use at least one service with an actual apm agent)
• Verify the reservior sampling implementation (mix of slow and fast traces)
• Check that logs are sufficient and helpful (on error, when reaching max number of dynamic services etc. )
• Check that metrics are sufficient and helpful
• Test performance, on a larger scale with multiple indices and shards
• Follow documentation and verify that it is self-explanatory and sufficient (depends on Document tail-based sampling #4586 )
• Test behavior on Elastic Cloud

[simitt]

Everything required has been implemented in 8.1. Let's tick the testplan boxes when testing through the feature. Closing the issue, as the work has been done.