community: Implement DirectoryLoader lazy_load function #19537
Merged
+161
−36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
size:L
eyurtsev
requested changes
Mar 26, 2024
| @@ -197,3 +198,55 @@ def load(self) -> List[Document]: | |||
| pbar.close() | |||
|
|
|||
| return docs | |||
|
|
|||
| def lazy_load(self) -> Iterator[Document]: | |||
There was a problem hiding this comment.
Great if we're upgrading to lazy_load() could you delete the load() implementation? (it will proxy to lazy_load automatically)
There was a problem hiding this comment.
Deleted the previous load_file function, and made load proxy to lazy_load
| for i in items: | ||
| yield from self.lazy_load_file(i, p) | ||
|
|
||
| def lazy_load_file(self, item: Path, path: Path) -> Iterator[Document]: |
There was a problem hiding this comment.
Could you make this private? This is a non standard part of the document loader API so generally we do not expect users to be relying on it.
Could you also update load_file to proxy to this method, so we can avoid code duplication?
There was a problem hiding this comment.
Renamed lazy_load_file to _lazy_load_file. Made load proxy to lazy_load. load_file wasn't used anymore, so I deleted it.
eyurtsev
approved these changes
Mar 27, 2024
dosubot
bot
added
the
lgtm
|
For the documentation, file directory loader is documented here:
3 options:
|
|
@DasDingoCodes i reverted doc changes for now so we can merge. The doc changes in the mdx file didn't look right, and we don't want the other notebook present given that there's already a place for this documentation. If you want you can follow up with a PR to replace the mdx file with your original notebook (or somehow combine the content together) |
gkorland
pushed a commit
to FalkorDB/langchain
that referenced
this pull request
Mar 30, 2024
…hain-ai#19537) Thank you for contributing to LangChain! - [x] **PR title**: "community: Implement DirectoryLoader lazy_load function" - [x] **Description**: The `lazy_load` function of the `DirectoryLoader` yields each document separately. If the given `loader_cls` of the `DirectoryLoader` also implemented `lazy_load`, it will be used to yield subdocuments of the file. - [x] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access: `libs/community/tests/unit_tests/document_loaders/test_directory_loader.py` 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory: `docs/docs/integrations/document_loaders/directory.ipynb` - [x] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: Eugene Yurtsev <[email protected]>
hinthornw
pushed a commit
that referenced
this pull request
Apr 26, 2024
Thank you for contributing to LangChain! - [x] **PR title**: "community: Implement DirectoryLoader lazy_load function" - [x] **Description**: The `lazy_load` function of the `DirectoryLoader` yields each document separately. If the given `loader_cls` of the `DirectoryLoader` also implemented `lazy_load`, it will be used to yield subdocuments of the file. - [x] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access: `libs/community/tests/unit_tests/document_loaders/test_directory_loader.py` 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory: `docs/docs/integrations/document_loaders/directory.ipynb` - [x] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: Eugene Yurtsev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to LangChain!
PR title: "community: Implement DirectoryLoader lazy_load function"
Description: The
lazy_loadfunction of theDirectoryLoaderyields each document separately. If the givenloader_clsof theDirectoryLoaderalso implementedlazy_load, it will be used to yield subdocuments of the file.Add tests and docs: If you're adding a new integration, please include
libs/community/tests/unit_tests/document_loaders/test_directory_loader.pydocs/docs/integrationsdirectory:docs/docs/integrations/document_loaders/directory.ipynbLint and test: Run
make format,make lintandmake testfrom the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/Additional guidelines:
If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17.