Support unstructured encoder #1672
Conversation
jieguangzhou
force-pushed
the
feat/pdf-encoder
branch
2 times, most recently
from
ed97db2 to
b7516d4
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1672 +/- ##
==========================================
- Coverage 80.33% 79.93% -0.41%
==========================================
Files 95 120 +25
Lines 6602 8452 +1850
==========================================
+ Hits 5304 6756 +1452
- Misses 1298 1696 +398 ☔ View full report in Codecov by Sentry. |
jieguangzhou
force-pushed
the
feat/pdf-encoder
branch
from
b7516d4 to
19ab738
Compare
| unstructure_kwargs: t.Dict[str, t.Any] = dc.field(default_factory=dict) | ||
|
|
||
| def __post_init__(self): | ||
| self.encoder = create_encoder(self.unstructure_kwargs) |
There was a problem hiding this comment.
What do these do? Convert a .pdf file to bytes and back?
There was a problem hiding this comment.
Here I want to keep all source data/element(page_num, text_type(title, text, table...)).
If we don’t want the bytes, we can convert them into a list of dict, also can directly merge them into a text.
Do you have any suggestion? @blythed @kartik4949
- Add a tag to choose:
element`json\text` - Add a merge func?
- only keep the source
element
There was a problem hiding this comment.
Yes keeping all of that information is good.
|
|
||
| def create_decoder(unstructure_kwargs): | ||
| def decoder(b: bytes): | ||
| if isinstance(b, str): |
There was a problem hiding this comment.
b: bytes?
how can it be str
There was a problem hiding this comment.
If we use encoder(uri=file_path), the uri(a str) will be passed in here
What do you think if I ban the use of uri=xxx format?
Because our downloader processes uri, it is currently not very compatible with unstructured
jieguangzhou
force-pushed
the
feat/pdf-encoder
branch
2 times, most recently
from
84f5a68 to
42cc873
Compare
jieguangzhou
force-pushed
the
feat/pdf-encoder
branch
from
8c6b293 to
3fe34a5
Compare
Description
Related Issues
Checklist
make unit-testingandmake integration-testingsuccessfully?Additional Notes or Comments