Fix bib resource finding, subprocess call, and upcoming deprecation (#13

)

* add tests for commands inside pre- and postnotes

* fix #2

* fix doc links

* fix docs

* update some testenvs to python3.5

* Bump version: 1.0.4 → 1.0.5

* use 2.7 instead of 3.5 for general test envs

* remove coverage tests on travis to avoid fails

* Update changelog

* Fix #3

* Update CHANGELOG.rst

* Bump version: 1.0.5 ? 1.0.6

* Update README.rst

* Switched broken pypip.in badges to shields.io (#8)

* Fix crash when no authors present

Updated `make_author_year_tokens_from_bib()` so that bib items that don't have authors (e.g. books with editor lists) are still handled correctly. Also moved some re.compile() calls outside the loop for efficiency.

* print useful errors for missing author/year data in bib item

* scan for \addbibresource & allow mutliple files

* fix run_latexdiff command arguments and error capturing

* fix invalid escape sequence (deprecated in Python 3.12) and other minor style fixes

---------

Co-authored-by: Christer van der Meeren <[email protected]>
Co-authored-by: Christer van der Meeren <[email protected]>
Co-authored-by: Michael Overmeyer <[email protected]>
Co-authored-by: Tim Wilson <[email protected]>

.travis.yml

@@ -6,11 +6,8 @@ env:
     LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so
   matrix:
     - TOXENV=check
-    - TOXENV=2.7,coveralls,codecov
     - TOXENV=2.7-nocover
-    - TOXENV=3.3,coveralls,codecov
     - TOXENV=3.3-nocover
-    - TOXENV=3.4,coveralls,codecov
     - TOXENV=3.4-nocover
 before_install:
   - python --version

CHANGELOG.rst

@@ -2,6 +2,17 @@
 Changelog
 =========
 
+
+1.0.6 (2017-02-26)
+-----------------------------------------
+
+* Fix bug where temp files has not finished writing before calling latexdiff
+
+1.0.5 (2017-02-21)
+-----------------------------------------
+
+* Fix crash when LaTeX commands were used in pre-notes and post-notes
+
 1.0.4 (2015-06-08)
 -----------------------------------------
 

CONTRIBUTING.rst

@@ -7,13 +7,13 @@ Bug reports, feature suggestions and other contributions are greatly appreciated
 Short version
 =============
 
-* Submit bug reports and feature requests at `GitHub <https://github.com/cmeeren/latexdiffcite/issues>`_
+* Submit bug reports and feature requests at `GitHub <https://github.com/cmeeren/latexdiffcite/issues>`__
 * Make pull requests to the ``develop`` branch
 
 Bug reports
 ===========
 
-When `reporting a bug <https://github.com/cmeeren/latexdiffcite/issues>`_ please include:
+When `reporting a bug <https://github.com/cmeeren/latexdiffcite/issues>`__ please include:
 
 * your operating system name and version
 * any details about your local setup that might be helpful in troubleshooting
@@ -32,12 +32,12 @@ Feel free to add additional configuration examples. This should include, in the
 * an example ``.tex`` file
 * the output `latexdiffcite` would produce after replacing citation commands
 
-Please run your JSON through a `JSON validator and formatter <http://jsonlint.com>`_ before adding it to the docs.
+Please run your JSON through a `JSON validator and formatter <http://jsonlint.com>`__ before adding it to the docs.
 
 Feature requests and feedback
 =============================
 
-The best way to send feedback is to file an issue at `GitHub <https://github.com/cmeeren/latexdiffcite/issues>`_.
+The best way to send feedback is to file an issue at `GitHub <https://github.com/cmeeren/latexdiffcite/issues>`__.
 
 If you are proposing a feature:
 
@@ -50,7 +50,7 @@ Development
 
 To set up `latexdiffcite` for local development:
 
-1. Fork `latexdiffcite` on `GitHub <https://github.com/cmeeren/latexdiffcite/fork>`_.
+1. Fork `latexdiffcite` on `GitHub <https://github.com/cmeeren/latexdiffcite/fork>`__.
 2. Clone your fork locally::
 
     git clone [email protected]:your_name_here/latexdiffcite.git
@@ -61,7 +61,7 @@ To set up `latexdiffcite` for local development:
 
    Now you can make your changes locally. If you add functionality, also add a test in ``tests/test_latexdiffcite.py``. The tests are run with ``py.test`` and can be written as normal functions (starting with ``test_``) containing a standard ``assert`` statement for testing output.
 
-4. When you're done making changes, run all the checks, doc builder and spell checker with `tox <http://tox.readthedocs.org/en/latest/install.html>`_:[1]_ ::
+4. When you're done making changes, run all the checks, doc builder and spell checker with `tox <http://tox.readthedocs.io/en/latest/install.html>`__:[1]_ ::
 
     tox
 
@@ -85,7 +85,7 @@ For merging, you should:
 3. Add yourself to ``AUTHORS.rst``.
 
 .. [1] If you don't have all the necessary python versions available locally you can rely on Travis -- it will
-       `run the tests <https://travis-ci.org/cmeeren/latexdiffcite/pull_requests>`_ for each change you add in the pull request. It will be a bit slower than testing locally, though.
+       `run the tests <https://travis-ci.org/cmeeren/latexdiffcite/pull_requests>`__ for each change you add in the pull request. It will be a bit slower than testing locally, though.
 
 Tips
 ----

README.rst

@@ -1,3 +1,12 @@
+Help wanted!
+============
+
+I lack both the time and interest to maintain and develop latexdiffcite further. Give me a holler if you want to take over.
+
+
+
+
+
 | |docs| |version| |downloads| |supported-versions|
 | |travis| |appveyor| |codecov| |landscape| |scrutinizer|
 
@@ -13,7 +22,7 @@
     :alt: PyPI Package monthly downloads
     :target: https://pypi.python.org/pypi/latexdiffcite
 
-.. |supported-versions| image:: https://pypip.in/py_versions/latexdiffcite/badge.svg?style=flat
+.. |supported-versions| image:: https://img.shields.io/pypi/pyversions/latexdiffcite.svg?style=flat
     :alt: Supported versions
     :target: https://pypi.python.org/pypi/latexdiffcite
 

docs/conf.py

@@ -23,7 +23,7 @@
 year = u'2015'
 author = u'Christer van der Meeren'
 copyright = '{0}, {1}'.format(year, author)
-version = release = u'1.0.4'
+version = release = u'1.0.6'
 
 pygments_style = 'trac'
 templates_path = ['.']

docs/config_examples/author_year_AGU_bbl/doc.rst

@@ -4,7 +4,7 @@
 Author-year AGU (bbl)
 =====================
 
-This configuration parses ``.bbl`` files in the style created by the American Geophysical Union's `LaTeX Formatting Toolkit <http://publications.agu.org/author-resource-center/author-guide/text-requirements/latex-formatting-toolkit>`_ and renders written-out citations the same way BibTeX does using the AGU templates.
+This configuration parses ``.bbl`` files in the style created by the American Geophysical Union's `LaTeX Formatting Toolkit <https://www.latextemplates.com/template/american-geophysical-union>`__ and renders written-out citations the same way BibTeX does using the AGU templates.
 
 The output is exactly the same as :ref:`AGU_bib`.
 

docs/config_examples/author_year_AGU_bib/doc.rst

@@ -4,7 +4,7 @@
 Author-year AGU (bib)
 =====================
 
-This configuration parses ``.bib`` files and creates written-out citations in the same style BibTeX does using the American Geophysical Union's `LaTeX Formatting Toolkit <http://publications.agu.org/author-resource-center/author-guide/text-requirements/latex-formatting-toolkit>`_.
+This configuration parses ``.bib`` files and creates written-out citations in the same style BibTeX does using the American Geophysical Union's `LaTeX Formatting Toolkit <https://www.latextemplates.com/template/american-geophysical-union>`__.
 
 The output is exactly the same as :ref:`AGU_bbl`.
 

docs/usage.rst

@@ -18,7 +18,7 @@ To compare two files ``FILE_OLD`` and ``FILE_NEW`` on disk::
     latexdiffcite file FILE_OLD FILE_NEW
 
 Comparing revisions of a file in a git repository
-------------------------------------------------
+-------------------------------------------------
 
 To compare two revisions ``REV_OLD`` and ``REV_NEW`` of a file ``FILE`` in a git repository::
 
@@ -61,4 +61,4 @@ In ``bib`` mode (default), `latexdiffcite` will look for a ``\bibliography{}`` c
 
 In ``.bbl`` mode, `latexdiffcite` ignores ``\bibliography{}`` commands and instead reads the compiled ``.bbl`` files. You need to supply a configuration file with a `regular expression <http://www.regular-expressions.info>`_ (regex) used to parse the ``bbl`` file and return capture groups which you can use in the citation formatting. For more details, see :ref:`Configuration` and :ref:`Configuration_Examples`. Note that when you run `latexdiff` in ``git`` mode, the ``.bbl`` file naturally needs to be versioned (and exist in the repository for both revisions).
 
-For both of these modes, `latexdiffcite` will include a ``\nocite{ref1,ref2,...}`` just before ``\end{document}`` containing all the references, so that the bibliography will be rendered as usual.
+For both of these modes, `latexdiffcite` will include a ``\nocite{ref1,ref2,...}`` just before ``\end{document}`` containing all the references, so that the bibliography will be rendered as usual.

setup.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.0.4
+current_version = 1.0.6
 files = setup.py docs/conf.py src/latexdiffcite/__init__.py src/latexdiffcite/latexdiffcite.py
 commit = True
 tag = False

setup.py

@@ -20,7 +20,7 @@ def read(*names, **kwargs):
 
 setup(
     name='latexdiffcite',
-    version='1.0.4',
+    version='1.0.6',
     license='BSD',
     description='Wrapper around latexdiff to make citations diff properly',
     long_description='%s\n%s' % (read('README.rst'), re.sub(':[a-z]+:`~?(.*?)`', r'``\1``', read('CHANGELOG.rst'))),

src/latexdiffcite/__init__.py

@@ -1,4 +1,4 @@
 # -*- coding: utf-8 -*-
 
 
-__version__ = '1.0.4'
+__version__ = '1.0.6'

src/latexdiffcite/latexdiffcite.py

@@ -15,7 +15,7 @@
 import tempfile
 import subprocess
 
-__version__ = '1.0.4'
+__version__ = '1.0.6'
 
 log = logging.getLogger(__name__)
 
@@ -380,11 +380,11 @@ def get_all_ref_keys(oldnew):
 
     # find arguments of all LaTeX citation commands in document
     all_cite_commands = '|'.join(Config.cmd_format.keys())
-    args_all_commands = re.findall(r'\\(?:' + all_cite_commands + r')\s*\[?.*?\]?\s*\{(.*?)\}', remove_comments(s), flags=re.S)
+    args_all_commands = re.findall(r'\\(?:' + all_cite_commands + r')\s*(?:\[[^\]]*?\]\s*){0,2}\{(.*?)\}', remove_comments(s), flags=re.S)
 
     # for each citation command, save new references
     for args in args_all_commands:
-        ref_list = re.split('\s*,\s*', args)
+        ref_list = re.split(r'\s*,\s*', args)
         log.debug('references found: %s', ref_list)
         new_refs = [r for r in ref_list if r not in refkeys]
         log.debug('new references: %s', new_refs)
@@ -409,7 +409,7 @@ def get_capture_groups_from_bbl(oldnew):
     # create empty dict if bbl_contents is empty
     if not bbl_contents:
         setattr(References, 'capture_groups_' + oldnew,
-                dict(zip(getattr(References, 'refkeys_' + oldnew), [tuple()]*len(refkeys))))
+                dict(zip(getattr(References, 'refkeys_' + oldnew), [tuple()] * len(refkeys))))
         return
 
     capture_groups = {}
@@ -470,16 +470,17 @@ def replace_capture_groups(s, ref, oldnew):
     '''Replaces all capture group tokens in string'''
 
     for i, replacement in enumerate(getattr(References, 'capture_groups_' + oldnew)[ref]):
-        s = s.replace('%CG{}%'.format(i+1), replacement or '')
+        s = s.replace('%CG{}%'.format(i + 1), replacement or '')
     return s
 
 
 def read_bibfile(oldnew):
     '''Reads contents of bibtex files'''
 
     # get bibtex file
-    bibarg = find_bibliography_arg(getattr(FileContents, 'tex_' + oldnew))
-    find_bibfiles(bibarg, oldnew)
+    bibargs = find_bibliography_args(getattr(FileContents, 'tex_' + oldnew))
+    for bibarg in bibargs:
+        find_bibfiles(bibarg, oldnew)
     bibfiles = getattr(Files, 'bib_' + oldnew + '_path')
 
     # read bibtex files
@@ -489,20 +490,23 @@ def read_bibfile(oldnew):
             getattr(FileContents, 'bib_' + oldnew).append(f.read())
 
 
-def find_bibliography_arg(s):
+def find_bibliography_args(s):
     '''Looks through string for \bibliography{} command and retrieves the argument'''
 
+    bibfiles = []
     log.debug('searching for \\bibliography{} entry in tex file')
-    bibfile = re.search(r'$[^%]*\\bibliography\s*{(.*?)}', s, flags=re.M).group(1)
-    log.debug('bibliography argument found: %s', bibfile)
-    return bibfile
+    bibfiles.extend(re.findall(r'^\s*\\bibliography\s*{(.*?)}', s, flags=re.M))
+    log.debug('searching for \\addbibresource{} entries in tex file')
+    bibfiles.extend(re.findall(r'^\s*\\addbibresource\s*{(.*?)}', s, flags=re.M))
+    log.debug('bibliography arguments found: %s', str(bibfiles))
+    return bibfiles
 
 
 def find_bibfiles(arg, oldnew):
     '''Searches for the bibfiles on the system'''
 
     sourcepath = os.path.dirname(getattr(Files, 'tex_' + oldnew + '_path'))
-    fnames = re.split('\s*,\s*', arg)
+    fnames = re.split(r'\s*,\s*', arg)
     for i, fname in enumerate(fnames):
         fnames[i] = fname if fname.endswith('.bib') else fname + '.bib'
 
@@ -526,13 +530,22 @@ def make_author_year_tokens_from_bib(oldnew):
     # if numeric mode (%AUTHOR% and %YEAR% not used in any fields), return empty strings
     if all('%AUTHOR%' not in fmt['author'] and '%YEAR%' not in fmt['year'] for fmt in Config.cmd_format.values()):
         log.debug('no %AUTHOR% or %YEAR% tokens detected in any fields, skipping formatting of author/year')
-        authyear = dict(zip(refkeys, [('', '')]*len(refkeys)))
+        authyear = dict(zip(refkeys, [('', '')] * len(refkeys)))
         setattr(References, 'authyear_' + oldnew, authyear)
         return
 
     # keys = reference key, values = tuple of (%AUTHOR%, %YEAR%)
     authyear = {}
 
+    # find author list in entry and create author string
+    author_re = [re.compile(r'author\s*=\s*[{"]((?:[^{}]+?|{[^}]+?})+?)[}"]', re.I | re.M | re.S),
+                re.compile(r'editor\s*=\s*[{"]((?:[^{}]+?|{[^}]+?})+?)[}"]', re.I | re.M | re.S),
+                re.compile(r'howpublished\s*=\s*[{"]((?:[^{}]+?|{[^}]+?})+?)[}"]', re.I | re.M | re.S),
+            ]
+
+    # find year in entry and create year string
+    year_re = re.compile(r'\s*year\s*=\s*["{]?\s*(\d+)\s*["}]?', flags=re.IGNORECASE)
+
     # process each reference individually
     for ref in refkeys:
 
@@ -552,11 +565,17 @@ def make_author_year_tokens_from_bib(oldnew):
 
             # AUTHOR
 
-            # find author list in entry and create author string
-            author_re = re.compile(r'author\s*=\s*[{"]((?:[^{}]+?|{[^}]+?})+?)[}"]', re.I | re.M | re.S)
-
             # split into a list of all authors
-            authors = re.split('\s+and\s+', author_re.search(entry).group(1))
+            # print("entry: ",entry)
+            # print("re search: ",author_re.search(entry))
+            authors = ""
+            for author_type in author_re:
+                author_search = author_type.search(entry)
+                if author_search is not None:
+                    authors = re.split(r'\s+and\s+', author_search.group(1))
+                    break
+            if authors == "":
+                raise NameError(f"Failed to find author/editor/etc information for {entry}")
 
             # get a list of only the surnames
             if any(',' in a for a in authors):
@@ -578,8 +597,10 @@ def make_author_year_tokens_from_bib(oldnew):
             # YEAR
 
             # find year in entry and create year string
-            year_re = re.compile(r'\s*year\s*=\s*["{]?\s*(\d+)\s*["}]?', flags=re.IGNORECASE)
-            year = year_re.search(entry).group(1)
+            try:
+                year = year_re.search(entry).group(1)
+            except Exception:
+                raise NameError(f"Failed to find year info for {entry}")
 
             # append the name and the year to the list
             authyear[ref] = (name, year)
@@ -605,9 +626,9 @@ def format_authorlist(surnames):
     '''Given a list of surnames, formats a string of all surnames correctly'''
 
     n = len(surnames)
-    serialcomma = ','*(n > 2 and Config.bib['author_serialcomma'])  # serial comma if at least 3 names
-    return (('{}' + Config.bib['sep_authors_first'])*(n-2) +  # name + first-kind separator if names > 2
-            ('{}' + serialcomma + Config.bib['sep_authors_last'])*(n > 1) +   # penultimate name and last-kind separator
+    serialcomma = ',' * (n > 2 and Config.bib['author_serialcomma'])  # serial comma if at least 3 names
+    return (('{}' + Config.bib['sep_authors_first']) * (n - 2) +  # name + first-kind separator if names > 2
+            ('{}' + serialcomma + Config.bib['sep_authors_last']) * (n > 1) +   # penultimate name and last-kind separator
             '{}').format(*surnames)  # final (or only) author name
 
 
@@ -656,18 +677,18 @@ def replace_refs_in_tex(oldnew):
     s = getattr(FileContents, 'tex_' + oldnew)
 
     # find all LaTeX citation commands in the string (exclude commented-out commands)
-    matches = re.findall(r'(\\(cite[tp]?)\s*(\[?.*?\]?)\s*\{(.*?)\})', remove_comments(s), flags=re.S)
+    matches = re.findall(r'(\\(cite[tp]?)\s*((?:\[[^\]]*?\]\s*){0,2})\{(.*?)\})', remove_comments(s), flags=re.S)
 
     # process the references for each citation command
     for full_cmd, cite_cmd, opt_args, cite_args in matches:
 
         log.debug('replacing %s', full_cmd)
 
         # split args to get a list of reference keys
-        refs_this = re.split('\s*,\s*', cite_args)
+        refs_this = re.split(r'\s*,\s*', cite_args)
 
         # find prenote/postnote if present
-        arg1, arg2 = re.search('(?:\[(.*?)\])?\s*(?:\[(.*?)\])?', opt_args).groups()
+        arg1, arg2 = re.search(r'(?:\[(.*?)\])?\s*(?:\[(.*?)\])?', opt_args).groups()
         if arg2 is None:
             prenote = None
             postnote = arg1
@@ -740,7 +761,7 @@ def format_refs(oldnew, replace_refs, cite_cmd, prenote, postnote):
         author = fmt['author']
         author = replace_capture_groups(author, ref, oldnew)
         author = author.replace('%AUTHOR%', authyear[ref][0])
-        author = author.replace('%NUMERIC%', str(getattr(References, 'refkeys_' + oldnew).index(ref)+1))
+        author = author.replace('%NUMERIC%', str(getattr(References, 'refkeys_' + oldnew).index(ref) + 1))
         out += author
 
         # author-year separator
@@ -799,23 +820,25 @@ def write_tex_to_temp(oldnew):
     '''Writes processed file contents to temp files'''
 
     log.debug('writing to file %s', getattr(Files, 'tex_' + oldnew + '_tmp_path'))
-    getattr(Files, 'tex_' + oldnew + '_tmp_hndl').write(getattr(FileContents, 'tex_' + oldnew).encode('utf-8'))
+    fh = getattr(Files, 'tex_' + oldnew + '_tmp_hndl')
+    fh.write(getattr(FileContents, 'tex_' + oldnew).encode('utf-8'))
+    fh.flush()
 
 
 def run_latexdiff(file1, file2):
     '''Runs latexdiff on file1 and file2 and writes output to outfile'''
 
     args = ['latexdiff', file1, file2]
     if Config.latexdiff_args:
-        args.append(Config.latexdiff_args)
+        args.extend(Config.latexdiff_args)
     log.info('running %s', ' '.join(args))
     log.debug('sending result to %s', Files.out_path)
     with io.open(Files.out_path, 'w', encoding='utf-8') as f:
         process = subprocess.Popen(args, stdout=f, stderr=subprocess.PIPE)
         _, stderr = process.communicate()
         ret_code = process.wait()
         if ret_code:
-            raise ValueError('latexdiff returned with code {}. Error from latexdiff:\n\n'.format(ret_code) + stderr)
+            raise ValueError('latexdiff returned with code {}. Error from latexdiff:\n\n'.format(ret_code) + str(stderr))
 
 
 if __name__ == '__main__':

tests/ascii_CRLF/test.tex

@@ -8,6 +8,7 @@
 Postnote only \citep[and references therein]{foo2010}.
 Testing spaces \citep [pre] [post] {foo2011lorem, bar2013}.
 Testing commented-out stuff % \cite{notused}
+Testing command inside args \citep[\odot$\dot{T}$][\odot$\dot{T}$]{foo2010}.
 Testing multi-line stuff \citep{foo2010, foo2011dolor,
 bar2013}