Replace async workaround within document loader #1774
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After talking with Timo, the document loader uses a seperate thread due to the fact that PyLD is synchronous code and calls into the document loader (which requires asynchronous code). When swapping out the cache with a Redis based cache, new exceptions rose up due to this implementation. No matter what I tried, it seemed impossible to keep the separate thread/asyncio event loop due to the error `got Future attached to a different loop`. Switching to `nest_asyncio` over using a separate thread/event loop resolves the issues that I was observing. Signed-off-by: Colton Wolkins (Indicio work address) <[email protected]>
TheTechmage
changed the title
Codecov Report
@@ Coverage Diff @@
## main #1774 +/- ##
==========================================
- Coverage 95.23% 95.23% -0.01%
==========================================
Files 525 525
Lines 33056 33059 +3
==========================================
+ Hits 31482 31484 +2
- Misses 1574 1575 +1 |
TimoGlastra
approved these changes
May 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After talking with Timo, the document loader uses a separate thread due to the fact that PyLD is synchronous code and calls
into the document loader (which requires asynchronous code). When swapping out the cache with (for example) a Redis based cache, the exception
got Future attached to a different loopmay arise due to the current separate thread/event loop based implementation.Switching to
nest_asyncioover using a separate thread/event loop resolves the issues that were observed.