Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Whitespace issues in extract_text() #42

Closed
SharmileeS opened this issue Nov 21, 2013 · 6 comments
Closed

Whitespace issues in extract_text() #42

SharmileeS opened this issue Nov 21, 2013 · 6 comments
Labels
is-feature A feature request whitespace While doing extract_text, getting the right number of whitespaces (spaces and newlines) is hard. workflow-text-extraction From a users perspective, text extraction is the affected feature/workflow

Comments

@SharmileeS
Copy link

I am not able to read text which proper formatting and spaces are not handled during extraction:

PreemptiveInformationExtractionusingUnrestrictedRelationDiscoveryYusukeShinyamaSatoshiSekineNewYorkUniversity715,Broadway,7thFloorNewYork,NY,10003fyusuke,sekineg@cs.nyu.eduAbstractWearetryingtoextendtheboundaryofInformationExtraction(IE)systems.Ex-istingIEsystemsrequirealotoftimeandhumanefforttotuneforanewscenario.

Is it true that pypdf2 is not format aware as given here: http://victorwyee.com/python/convert-pdf-to-text-pypdf-pdfminer-first-impression/
@mstamy2
Copy link
Collaborator

mstamy2 commented Dec 9, 2013

It is true that pyPdf was created primarily for manipulating the document structure and not for text extraction. Extracting text could definitely be improved upon in several ways, and it's a feature that should be developed for PyPDF2.

@jgyllinsky
Copy link

This issue is still a problem in May 2017. Version was pypdf2-1.26.0 as installed via pip on Ubuntu 16.04 lts.

@attibalazs
Copy link

I am facing similar issues when extract text from a lot of research papers from arxiv.org

Thisnoteismotivatedbythecelebratedoneandtwomutualfundtheorems[3].
ThesebuildontheworkofMarkowitzin[2]andtellusthat,intheMarkowitz
marketmodelwithnorestrictionsonshortselling,aninvestorwhoisonly
interestedintheoptimalinvestmentproblemscansafelyignoreallbutatwo-
dimensionalsubspaceofthespaceofportfolios.Butwhatifone'sinterests
aremorebroadrangingthantheseoptimalinvestmentproblems?Arethere

@jgyllinsky
Copy link

If someone knows how to get spaces (' '), from PyPDF2 that would be very helpful. @attibalazs please let me know if you resolve this.

@attibalazs
Copy link

i switched to using pdfminer, i installed the version for python 3.6 pdfminer.six

polyglot-jones pushed a commit to polyglot-jones/PyPDF2 that referenced this issue Aug 11, 2020
@kurtmckee
Ensure that pypdfChr() always returns a str object
c4efab6 
Fixes py-pdf#42
@MartinThoma MartinThoma added the workflow-text-extraction From a users perspective, text extraction is the affected feature/workflow label Apr 16, 2022
@MartinThoma
Copy link
Member

#924 improved the situation a lot. As here is no PDF linked / uploaded, I assume the upcoming release (2.1.0) will fix the issue.

@MartinThoma MartinThoma changed the title PyPDF2 is not format aware Whitespace issues in extract_text() Jun 6, 2022
MartinThoma added a commit that referenced this issue Jun 6, 2022
@MartinThoma
REL: 2.1.0
4e44122 
The highlight of the 2.1.0 release is the most massive improvement to the
text extraction capabilities of PyPDF2 since 2016 🥳🎊 A very big thank you goes
to [pubpub-zz](https://github.com/pubpub-zz) who took a lot of time and
knowledge about the PDF format to finally get those improvements into PyPDF2.
Thank you 🤗💚

In case the new function causes any issues, you can use `_extract_text_old`
for the old functionality. Please also open a bug ticket in that case.

There were several people who have attempted to bring similar improvements to
PyPDF2. All of those were valuable. The main reason why they didn't get merged
is the big amount of open PRs / issues. pubpub-zz was the most comprehensive
PR which also incorporated the latest changes of PyPDF2 2.0.0.

Thank you to [VictorCarlquist](https://github.com/VictorCarlquist) for #858 and
[asabramo](https://github.com/asabramo) for #464 🤗

New Features (ENH):
-  Massive text extraction improvement (#924). Closed many open issues:
    - Exceptions / missing spaces in extract_text() method (#17) 🕺
      - Whitespace issues in extract_text() (#42) 💃
      - pypdf2 reads the hifenated words in a new line (#246)
    - PyPDF2 failing to read unicode character (#37)
      - Unable to read bullets (#230)
    - ExtractText yields nothing for apparently good PDF (#168) 🎉
    - Encoding issue in extract_text() (#235)
    - extractText() doesn't work on Chinese PDF (#252)
    - encoding error (#260)
    - Trouble with apostophes in names in text "O'Doul" (#384)
    - extract_text works for some PDF files, but not the others (#437)
    - Euro sign not being recognized by extractText (#443)
    - Failed extracting text from French texts (#524)
    - extract_text doesn't extract ligatures correctly (#598)
    - reading spanish text - mark convert issue (#635)
    - Read PDF changed from text to random symbols (#654)
    - .extractText() reads / as 1. (#789)
-  Update glyphlist (#947) - inspired by #464
-  Allow adding PageRange objects (#948)

Bug Fixes (BUG):
-  Delete .python-version file (#944)
-  Compare StreamObject.decoded_self with None (#931)

Robustness (ROB):
-  Fix some conversion errors on non conform PDF (#932)

Documentation (DOC):
-  Elaborate on PDF text extraction difficulties (#939)
-  Add logo (#942)
-  rotate vs Transformation().rotate (#937)
-  Example how to use PyPDF2 with AWS S3 (#938)
-  How to deprecate (#930)
-  Fix typos on robustness page (#935)
-  Remove scripts (pdfcat) from docs (#934)

Developer Experience (DEV):
-  Ignore .python-version file
-  Mark deprecated code with no-cover (#943)
-  Automatically create Github releases from tags (#870)

Testing (TST):
-  Text extraction for non-latin alphabets (#954)
-  Ignore PdfReadWarning in benchmark (#949)
-  writer.remove_text (#946)
-  Add test for Tree and _security (#945)

Code Style (STY):
-  black, isort, Flake8, splitting buildCharMap (#950)

Full Changelog: 2.0.0...2.1.0
@MartinThoma MartinThoma added is-feature A feature request and removed Feature labels Jun 22, 2022
@MartinThoma MartinThoma added the whitespace While doing extract_text, getting the right number of whitespaces (spaces and newlines) is hard. label Jan 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
is-feature A feature request whitespace While doing extract_text, getting the right number of whitespaces (spaces and newlines) is hard. workflow-text-extraction From a users perspective, text extraction is the affected feature/workflow
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@attibalazs @MartinThoma @mstamy2 @SharmileeS @jgyllinsky