feat: hard document length limit at max_chars_check

[ZanSara]

Related Issues

• fixes https://github.com/deepset-ai/haystack-private/issues/62

Proposed Changes:

• Split documents into shorted ones if they are longer than max_chars_check.
• The split is recursive, i.e. if the split document is still too long, it will be split again and again until all the chunks are below the threshold.

How did you test it?

• Updated test/nodes/test_preprocessor.py::test_preprocessor_very_long_document

Notes for the reviewer

n/a

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 5355137661

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 105 unchanged lines in 6 files lost coverage.
• Overall coverage increased (+0.1%) to 42.627%

---

Files with Coverage Reduction	New Missed Lines	%
nodes/prompt/shapers.py	2	91.3%
utils/openai_utils.py	5	90.22%
agents/base.py	6	95.81%
nodes/other/shaper.py	13	92.4%
nodes/prompt/prompt_template.py	21	89.55%
nodes/preprocessor/preprocessor.py	58	82.57%

Totals
Change from base Build 5335925213:	0.1%
Covered Lines:	9592
Relevant Lines:	22502

---

💛 - Coveralls

[anakin87]

Hey @ZanSara!

In this implementation, I can't understand if we generate a new document id for the tail_document(s) (it seems not).
WDYT?

[ZanSara]

Uh good catch! You're right, we're likely not. I'm fixing that 👍

[anakin87]

@ZanSara I am excited because this is the first time I am doing a review in a PR by you. 😄

LGTM!

Just a comment: I would change the docstring of _long_documents to reflect the changes...