feat: Upload to AI Search in Batches

[cmw2]

Purpose

• Allows large PDFs to be ingested.
• Fixes Batch calls to AI Search to Support large PDFs #1267

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

• Follow steps document in Local Deployment

• Test the code

  • Ingest a large file. My tested one had ~850 pages and was 18.5 Megs in size.

What to Check

Verify that the following are valid

• All chunks should be in search index (can take a while)

Other Information

• Modified existing upload test to pass with 1 batch.
• Added another test to check for multiple batches.
• Also had to do some code cleanup per black and flake8, though they conflicted in some rules so added exceptions in flake config file.

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
code/backend/batch/utilities/helpers
env_helper.py	147	6	95%	256–258, 277–279
code/backend/batch/utilities/helpers/embedders
push_embedder.py	79	1	98%	93
TOTAL	2665	592	77%

Tests	Skipped	Failures	Errors	Time
300	0 💤	0 ❌	0 🔥	40.516s ⏱️

[adamdougal]

LGTM! Thanks for this!

[github-actions]

🎉 This PR is included in version 1.11.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀