Multiple service policies result in unusual sample rates

[bryce-b]

APM Server version (apm-server version): 7.16.0-SNAPSHOT

Description of the problem including expected versus actual behavior:
In testing, I used multiple tail-based sampling policies designed to apply unique sample rates to specific services. I used a comparison between the request transaction metrics count vs the request transaction traces captured to determine the actual sample rate. useful dashboard for comparison is found here: https://github.com/bryce-b/apm-server/blob/tbs-metrics/cmd/TBS/tbs_charts_3.2.ndjson

Here is the policies used in testing:

apm-server:
 sampling:
   keep_unsampled: false
   tail:
     enabled: true
   policies: 
        - service:
          name: "opbeans-java"
          sample_rate: 0.5
        - service:
          name: "opbeans-python"
          sample_rate: 0.2
        - service:
          name: "opbeans-ruby"
          sample_rate: 0.8
        - service:
          name: "opbeans-go"
          sample_rate: 0.6
        - sample_rate: 0.1

In testing, each service had some drift from the expected sample rate, which isn't unexpected, but the sample rate for opbeans-ruby is drastically different from the expected sample rate.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. start cloud deployment
2. apply apm integration to deployment
3. start apm-integration-testing for traffic
4. apply apm-server config described above

Provide logs (if relevant):

[axw]

@bryce-b do you recall how you ran apm-integration-testing? Did you start all the opbeans services together?

If all opbeans services are started together, then the services will proxy some requests to one another to demonstrate distributed tracing. Thus, a request to opbeans-ruby may have started in another service -- hence the sampling rate for that other service would take effect.

Then, to add on top: I just ran apm-integration-testing with opbeans-go and opbeans-ruby, and none of the transactions coming through from Ruby are root transactions. Seems there's something wrong there, so I'd ignore the opbeans-ruby results.

[axw]

Opened elastic/apm-agent-ruby#1231 for the Ruby issue. Closing this, as TBS not working is a symptom.