NonBlocking Sender Scheduling and Bulk Request Buffering

[AndreKurait]

Description

Updates the DocumentReindexer logic to use shared threads for subscribing on the bulk request mono (which can be done due to async non-blocking client) as well as updated the unit tests against this behavior.

Updates the DocumentReindexer buffering logic to be 50 bulk requests which can be a more consistent size based on workloads than on a document level.

• Category: Enhancement
• Why these changes are required? Improve throughput and reduce memory pressure
• What is the old behavior before changes and new behavior after changes? See Testing below

Issues Resolved

MIGRATIONS-1961

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Tested in AWS against main commit e8e2770 with OR1 and 4vcpu 8gb tasks with --max-connections 20 against 4 node OR1 16xl OS 2.13 cluster with 100 byte docs. Observing lower and more consistent CPU utilization and much lower memory utilization along with ~10% request throughput increase

Check List

• New functionality includes testing
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.51%. Comparing base (143ce1a) to head (1327378).
Report is 8 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #914      +/-   ##
============================================
- Coverage     74.55%   74.51%   -0.04%     
+ Complexity     1964     1961       -3     
============================================
  Files           343      343              
  Lines         13552    13555       +3     
  Branches        786      786              
============================================
- Hits          10104    10101       -3     
- Misses         2960     2964       +4     
- Partials        488      490       +2     

Flag	Coverage Δ
gradle-test	69.00% <100.00%> (-0.06%)	⬇️
python-test	88.58% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.