ES search rate very high when TBS enabled

[bryce-b]

APM Server version (apm-server version): 7.16.0-SNAPSHOT

Description of the problem including expected versus actual behavior:
When Tail-based sampling is enabled, ES search rate doubles

In the above screencap you can see when TBS is enabled at 21:30

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. start cloud deployment
2. apply APM integrations to deployment
3. generate data w/ apm_integration_testing
4. apply configuration to APM Server:

apm-server:
  data_streams:
    enabled: true
  sampling:
    keep_unsampled: false
    tail:
      enabled: true
	 policies:
	        - sample_rate: 0.1

[axw]

With some additional debug logging in place, I can see that the TBS subscriber is repeatedly returning the same trace IDs. Still digging...

[axw]

OK, found the issue. The subscriber is not updating the minimum constraint on _seq_no when performing multiple searches, when there are more than 1000 docs, causing this loop to never terminate:

apm-server/x-pack/apm-server/sampling/pubsub/pubsub.go

Line 233 in 52e0763

for maxObservedSeqno < maxSeqno {