Refactor tail-sampling processor - Experiment and validate the scheduling mechanism

[jpkrohling]

During some performance tests, I noticed that some data appeared to be lingering around even when nothing was expected to still be in the queue. This issue is about creating enough load on the collector to reproduce this kind of issue, adding the necessary telemetry to either identify when this happens or to evidence that it doesn't.

[github-actions]

Pinging code owners for processor/tailsampling: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jamesmoessis]

@jpkrohling when you say there's lingering data do you have any more specifics? Is it just un-GCed memory or things that remain in the cache when they shouldn't? Or you don't know and we need to do more specific validation.

In the coming weeks I'll be testing this processor at quite a high load looking to optimise it, so I'll put any updates here if I see anything related.

[jpkrohling]

I'm not 100% sure: what I have seen in the past, and this might be even fixed already by now, is that traces would still be kept there in the internal map, likely due to concurrent updates issues.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.