[community] [feature]: Implementation of excel parser and including it in o365 loader

[MacanPN]

Description

Currently there is no implementation of .xlsx parser and consequently FileSystemBlobLoader as well as all loaders derived from O365BaseLoader are limited to .doc, .docx, .pdf and .txt. This PR uses unstructured to implement ExcelParser parser very similar to MsWordParser (with just an extra bit of post processing).

Dependencies:

• htmltabletomd

Tests

I've implemented a unit test for the .xlsx parser however it depends on unstructured module and I have a problem getting any version of unstructured to pyproject.toml that would work and didn't break something somewhere. Please let me know if anyone is willing to help me with that.

Twitter handle: @martintriska1

Is anyone willing to review this please? Thanks! @baskaryan, @eyurtsev, @ccurme

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Oct 8, 2024 8:08am

[petergoldstein]

XLSX files are important for a variety of enterprise use cases. Having a parser will be very handy for some of our current challenges.

[avalon-swain]

This will be helpful

[ccurme]

Thanks for this. How do you feel about using langchain-unstructured here? Are there important differences in the resulting Document objects?

[MacanPN]

@ccurme Actually in the meantime, I've written up this wrapper that allows to turn a DocumentLoader into Parser. Excel parser is already implemented here so I'd abandon this PR in favor of merging the DocumentLoaderAsParser wrapper together with some more changes to UnstructuredExcelLoader (or possibly langchain-unstructured.document_loaders.UnstructuredLoader). What's the current thinking? Is the implementation in community being deprecated? (In community there is implementation with inheritance UnstructuredExcelLoader : UnstructuredFileLoader : UnstructuredBaseLoader : BaseLoader where as in langchain-unstructured there is independent implementation only inheriting directly from BaseLoader.

[MacanPN]

@ccurme I see that the DocumentLoaderAsParser was assigned status needs support. I would appreciate if you could take a look at that one in context of being substitute for this PR. Thanks!

[MacanPN]

@ccurme I'm thinking I might not have made clear enough about why having BlobParser rather than DocumentLoader is important. Please let me breifly explain:

• Document loaders function as <whatever> -> documents
• BlobParsers process blob -> documents.
• Many document loaders actually implement logic until they have a blob. Then the actual parsing is done by calling parser with similar name. As an example you can look at AmazonTextractPDFLoader and AmazonTextractPDFParser.
• Some document loaders may want to process numerous filetypes. An example is a SharePointLoader that is supposed to fetch all parsable files, parse them and produce documents. Until recently it processed only 3 file types doc, docx, pdf and ignored everything else. Recently I got this pr merged that enables a user to pass in dict mapping file types (or mime types) to parsers. Now one can easily use SharePointLoader with any files where suitable parser is available.
• Unfortunatelly many document loaders do not define BlobParser but rather handle everything directly inside the loader class. UnstructuredLoader is one of those.
• Originally I went ahead and implemented separate ExcelParser (this PR). However this duplicated some of the code. That could be mitigated by replacing the part of the code inside UnstructuredExcelLoader responsible for parsing the file with calling the Parser. This would basically bring it to agreement with how things are done for ex. in PDF Loaders.
• However I found out that I'd have to do this shuffling for a fairly long list of document loaders. Given how much time such effort would require, I opted to create DocumentLoaderAsParser wrapper. This wrapper can be used on any DocumentLoader that accepts file_path argument and turns it into BaseBlobParser.

So the idea is to be able to do:

xlsx_parser = DocumentLoaderAsParser(UnstructuredExcelLoader, mode="paged")
mp3_parser = DocumentLoaderAsParser(GoogleSpeechToTextLoader, project_id="...")

# create dictionary mapping file types to handlers (parsers)
handlers = {
    "xlsx": xlsx_parser,
    "mp3": mp3_parser
}
loader = SharePointLoader(document_library_id="...",
                            handlers=handlers # pass handlers to SharePointLoader
                            )
documents = loader.load()

Hope this makes sense.

[ccurme]

Closing in favor of #27749, let me know if I misunderstood the intent.