[CapMan visibility] add throttling to BytesScannedRejectingPolicy

[xurui-c]

https://getsentry.atlassian.net/browse/SNS-2794

The BytesScannedRejectingPolicy prevents a (project_id|organization_id, referrer) combination from sending too many queries by rejecting them once such combination reaches a rejection threshold. I add a throttle threshold so the offending combination will get throttled first before getting rejected. BytesScannedRejectingPolicy will half the threads when the number of bytes scanned goes over 50% of the rejection threshold for that combination.

Roll out plan:
In this PR, I set the throttle dividers to 1 so that the rejection threshold and the throttle threshold are the same, so effectively this PR will create the options on Snuba Admin without doing any throttling. Then, within S4S, I will adjust the throttle dividers. With this, I expect to see queries being throttled in DD under the metric concurrent_queries_throttled. I will change the throttle dividers to 0.5 afterwards.

[codecov]

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 2459 tests with 1 failed, 2454 passed and 4 skipped.

View the full list of failed tests

Test Description

Failure message

Testsuite: pytest Test name: tests.web.test_db_query::test_db_query_success Envs: - default

Traceback (most recent call last): File ".../tests/web/test_db_query.py", line 265, in test_db_query_success assert stats["quota_allowance"] == { AssertionError: assert {'BytesScannedRejectingPolicy': {'can_run': True,\n 'explanation': {'reason': 'within_limit but '\n 'throttled',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'BytesScannedWindowAllocationPolicy': {'can_run': True,\n 'explanation': {'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'ConcurrentRateLimitAllocationPolicy': {'can_run': True,\n 'explanation': {'overrides': {},\n 'reason': 'within '\n 'limit',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'CrossOrgQueryAllocationPolicy': {'can_run': True,\n 'explanation': {'reason': 'pass_through',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'ReferrerGuardRailPolicy': {'can_run': True,\n 'explanation': {'policy': 'referrer_guard_rail_policy',\n 'reason': 'within limit',\n 'referrer': 'something',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10}} == {'BytesScannedRejectingPolicy': {'can_run': True,\n 'explanation': {'reason': 'within_limit',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'BytesScannedWindowAllocationPolicy': {'can_run': True,\n 'explanation': {'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'ConcurrentRateLimitAllocationPolicy': {'can_run': True,\n 'explanation': {'overrides': {},\n 'reason': 'within '\n 'limit',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'CrossOrgQueryAllocationPolicy': {'can_run': True,\n 'explanation': {'reason': 'pass_through',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10},\n 'ReferrerGuardRailPolicy': {'can_run': True,\n 'explanation': {'policy': 'referrer_guard_rail_policy',\n 'reason': 'within limit',\n 'referrer': 'something',\n 'storage_key': 'StorageKey.ERRORS_RO'},\n 'max_threads': 10}} Common items: {'BytesScannedWindowAllocationPolicy': {'can_run': True, 'explanation': {'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'ConcurrentRateLimitAllocationPolicy': {'can_run': True, 'explanation': {'overrides': {}, 'reason': 'within ' 'limit', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'CrossOrgQueryAllocationPolicy': {'can_run': True, 'explanation': {'reason': 'pass_through', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'ReferrerGuardRailPolicy': {'can_run': True, 'explanation': {'policy': 'referrer_guard_rail_policy', 'reason': 'within limit', 'referrer': 'something', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}} Differing items: {'BytesScannedRejectingPolicy': {'can_run': True, 'explanation': {'reason': 'within_limit but throttled', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}} != {'BytesScannedRejectingPolicy': {'can_run': True, 'explanation': {'reason': 'within_limit', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}} Full diff: { 'BytesScannedRejectingPolicy': {'can_run': True, - 'explanation': {'reason': 'within_limit', & - + 'explanation': {'reason': 'within_limit but ' & +++++ + 'throttled', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'BytesScannedWindowAllocationPolicy': {'can_run': True, 'explanation': {'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'ConcurrentRateLimitAllocationPolicy': {'can_run': True, 'explanation': {'overrides': {}, 'reason': 'within ' 'limit', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'CrossOrgQueryAllocationPolicy': {'can_run': True, 'explanation': {'reason': 'pass_through', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, 'ReferrerGuardRailPolicy': {'can_run': True, 'explanation': {'policy': 'referrer_guard_rail_policy', 'reason': 'within limit', 'referrer': 'something', 'storage_key': 'StorageKey.ERRORS_RO'}, 'max_threads': 10}, }