fix bug: docker container process will crash when upload a big PDF(100M). #473
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
@sszgwdk is attempting to deploy a commit to the pingcap Team on Vercel. A member of the Team first needs to authorize it. |
| session.add(document) | ||
| session.commit() | ||
|
|
||
| build_index_for_document.delay(kb_id, document.id) | ||
| build_index_for_document.delay(kb_id, document.id, original_content) |
There was a problem hiding this comment.
It seems to increase Redis' memory overhead? Can we refer to Git Large File Storage (LFS) for storing references to large files in the database instead of the raw content?
There was a problem hiding this comment.
We can set a max-upload-size parameters to protect redis and suppose user(admin) have enough sense that he/her SHOULD not set it a very big value(somehow bigger than redis's ram size)
So currenttly I think it looks good to me.
| original_content = self.content | ||
| self.content = self.content[:max_length] + "..." | ||
| return original_content | ||
| return None |
There was a problem hiding this comment.
If we truncate a big file, it will fail when "retry" to index it(based on current workflow)
|
I have one question, does truncated content matter? |
For the backend, apart from the initial chunk splitting and 'retry' interface, there seems to be no use of the original content elsewhere. |
The bug originated from Tidb's limitation on the length of the TEXT field to 6MB. When inserting document data into the db, the SQL execution failed due to the content being too long, exceeding 6MB. For this situation, consider truncating the content to meet the length limit and retaining the original content for indexing construction, so as not to affect the execution of subsequent processes.