Server side rejection of in-coming search requests based on resource consumption

[tushar-kharbanda72]

Is your feature request related to a problem? Please describe.

#1042 aims to build back-pressure support for Search requests. This feature will start rejecting incoming search requests when system resources allocated to search requests starts breaching the static/dynamic thresholds defined.

Describe the solution you'd like

Currently, Search rejections are solely based on the number of tasks in queue for Search ThreadPool. That doesn't provide fairness in rejections as multiple smaller queries can exhaust this limit but are not resource intensive and the node can take in much more requests and vice versa. Essentially count is not the reflection of actual work.

Hence based on metrics in point 1 above, we want to build a frame which can perform more informed rejections based on point in time resource utilisation. The new model will take the admission decision for search requests on the node. These admission decisions or rejection limits can be added at:

1. Shard level - With support for burst, if resources are available at the node level.
2. Node level - This will be the final guard rail (last line of defence) and model will not be admitting any new requests once this limit is breached.

The above will provide the required isolation of accounting and fairness in the rejections which is currently not there. This is still a reactive back-pressure mechanism as it only focusses on the current consumption and does not estimate the future work.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.