Advanced text extraction by asabramo #767
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
October 16, 2018 10:14
Add VSCode and GitEye ignores
Higher numbers represent deeper levels of debug. dbg(0, 'text') should always print it dbg(10, 'text') should print only if debugLevel is set to 10 or more
The optional lineCallback argument is called for each line in the extracted text. The callback receives a list of line elements, each containing text, x and y.
Use the BT and Td/TD tags to figure out the real position of a text element in a document. TD are relative, but BT resets the position. This corrects the line detection algorithm to work on documents like BezeqInt
This works for the BezeqInt document. Probably need to support other forms of CMAP. Definitely need to support other forms of encoding - e.g. "differences", as used in the Amex document. This code is based on: https://github.com/mstamy2/PyPDF2/pull/201/commits
This fixes the "Fenix" documents
Only supporting the case where Tm is used to do the same as TD This fixes the "Fenix"
Slightly generalize the text handling to support the TJ element as well. This should work when there are no specific glyph positioning. This solves the case of the Supergaz bill.
CMAP - add support for char based mapping TJ - fix translation This enables the Bezeq Phone bills
TODO - this may not support a combination of bfrange and bfchar in the same cmap... However, this does enable the Pelephone bill format :-)
The electricity bill has some garbage (HTML page) after the EOF marker... PyPDF allows up to 1K of such garbage so the file is rejected. Increased it to make it work. TODO: need a more robust solution, to accept files with more garbage. One way to do it is to cut that garbage before uploading.
The names of the glyphs seem to be "known" with their unicode mappings. This commit brings such name mappings from two different github projects to create a common list. Then the /Differences array is read and parsed into a CMAP compatible structure so that the translation works. This removes the hard coded encoding used for Amex and Mastercard Hebrew documents.
Added extractFonts and extractImages to the PDF reader class. This allows extraction of the fonts and images form the whole PDF in a single command, in a standard way.
Seems that the DIFFERENCES array handling was specific to the way it appeared in the Amex docs. Changed it to handle the Supergaz case properly too. Supergaz fonts are TrueType Also introduced mergine of cmaps from tounicode and differnces in order to make the unittest pass.
This way one can iterate over the response without checking for None. This makes the Hot Mobile bill work
Look for \nend at the end, which could actually be the endobject marker. This is because in the CellcomTV bill there's just no endstream marker for some of the streams (CMAPs)
Seems like Tm should be treated as an absolute location (actually offset from the last location, replacing the previous matrix) This fixes the CellcomTV bill to read the lines correctly
This fixes the PazGaz parsing.
Still, it seems that Pazgas don't provide
the right name for the newsheqel symbol, and it's identified as {
This is a hack for now, until the line size would be determined properly So for now, the caller needs to tell the pdf extractor what Y difference should be considered a line break. This fixes the BezeqPhone bill
This prevents crashing when parsing the IRS names file.
The T*, " and ' elements were not converted to using the new handleTextElement function. This came up when trying to open the IRS NAMES file.
Faking a newline operator when the text is over so that the new line handler is called also for the last line in the text.
Support ranges of characters in the form: <0000><0002>[<0032><0033><0034>] This appears in the GetTaxi receipt
This fixes the Pelephone bill parsing
TODO: how should we decide how many bytes represent each character in the string? Fixes GetTaxi after the change to support the Pelephone single byte...
Keep all the lines in a dictionary by Y coordinate and run the line callback only at the end (of each page) This fixes the Pelephone date signature
Text coordinates are provided in a transformed space, which is controlled by the 'cm' operator. In this commit, we read the cm matrix, take only the x and y scale operands and store them in a stack. When text operands arrive, we use the y value to translate the Y value. When q and Q operators we push and pop the graphics state respectively. For example, a pdf can push a new cm when displaying an image, and then go back to the previous cm values. These are all demonstrated in the supergas bill...
The Amisragas bill exposed issues with the cmap handling.
In the Amisragas bill, there's an empty XMP data (might be an error). This prevents the parser from crashing in this case, and just pretend there's no xmp data.
The Amisragas bill exposed some issues with the translation code, as well as the TJ operator handling. See comments in code. Also fixed the unit test and reference text filebecause we now extract the right unicode characters for the sample PDF.
The split('\n') doesn't work on such cases, so instead just parse all
the options with a regex in the beginning.
This is needed in the Mey Avivim bill
This fixes GetTaxi parsing. The target should be incremented only if the target is a single char. If it's a list, the next one on the list is popped. Add support for multiple sub-maps in one CMAP. E.g. bfrange + bfchar This fixes the HotTV bills!
The Kvish6 document uses the cm matrix with Y shift value to set the Y value on all the lines.
Use non greedy regex to correctly extract all the submaps. This fixed the "Fake Amex report", created by MS word. The document still shows bad latin chars instead of final Hebrew chars, but it seems the final Hebrew letters are just skipped in the ToUnicode encoding in the document, and all the viewers present them that way too...
Provide the left and right X values for each text element to alloow the caller to determine the columns in the text.
When a Tm operator appears, save that matrix in the text state This is needed, for example in the BezeqPhone bill, where the font size is always 1 and the text actual size is controlled by the X and Y multipliers in the text matrix. The width calculation is integrated into the translation operation because both these operations go over the glyph IDs so instead of calcualting the byte width and iterating again, just do it in the same loop.
using extractTextState allows the caller insight into the data collected about the page. For example, the lines, the char widths, etc. That's in addition to the text itself.
To use unichr on Python2
Wasn't tested on Python3 and therefore failed.
…o/PyPDF2 into asabramo-advanced_text_extraction
|
This PR is essentially just #464 with fixed merge conflicts |
|
Interesting, we are running in https://bugs.python.org/issue32582 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.