[CapMan visibility] add throttling to ReferrerGuardRail policy

[xurui-c]

https://getsentry.atlassian.net/browse/SNS-2793

The ReferrerGuardRail policy prevents referrers from sending too many queries by rejecting them once a referrer reaches a rejection threshold. I add a throttle threshold so the offending referrer will get throttled first before getting rejected. ReferrerGuardRailPolicy will half the threads when the number of concurrent queries goes over 50% of the rejection threshold for that referrer (e.g. subscriptions_executor rejects at 200 concurrent queries, we throttle at 100 queries)

[volokluev]

good overall just one change

[volokluev]

How are you planning to roll this out?

[xurui-c]

How are you planning to roll this out?

In this PR, I set the throttle dividers to 1 so that the rejection threshold and the throttle threshold are the same, so effectively this PR will create the options on Snuba Admin without doing any throttling. Then, within S4S, I will adjust the throttle dividers. With this, I expect to see queries being throttled in DD under the metric concurrent_queries_throttled. I will change the throttle dividers to 0.5 afterwards.

[codecov]

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 1614 tests with 1 failed, 1611 passed and 2 skipped.

View the full list of failed tests

Test Description

Failure message

Testsuite: pytest Test name: tests.query.allocation_policies.test_per_referrer.TestPerReferrerPolicy::test_throttle Envs: - default

Traceback (most recent call last): File ".../query/allocation_policies/test_per_referrer.py", line 69, in test_throttle assert first_quota_allowance.max_threads == policy.max_threads AssertionError: assert 0 == 10 + where 0 = QuotaAllowance(can_run=True, max_threads=0, explanation={'reason': 'within limit', 'policy': 'referrer_guard_rail_policy', 'referrer': 'statistical_detectors', 'storage_key': 'StorageKey.GENERIC_METRICS_DISTRIBUTIONS'}).max_threads + and 10 = <snuba.query.allocation_policies.per_referrer.ReferrerGuardRailPolicy object at 0x7f20c1ca39d0>.max_threads

[onewland]

attached some suggestions for changes, but fundamentally the concept and approach seem sound