Support multiple bibliographies per topic (etc.)

[wlupton]

Note: I'm creating this PR for discussion. I fully expect that further changes might be necessary before merging it.

The full list of changes is:

• Support multiple bibliographies per topic, i.e., in the 'bibliography' metadata, allow topic entries to be list-valued
• Support the same topic structure for the 'references' metadata item, so can override references per topic
• Ignore duplicate references, i.e. if the same reference exists in multiple topics (the first one encountered is used, but this is not satisfactory because the topic processing order is indeterminate). Note that something had to be done here, because these duplicates would have the same ids and therefore would be ambiguous
• Take some changes from the diverged multiple-bibliographies repo, notably don't use utils.citeproc() because it doesn't have a 'quiet' option (which is needed for the second pass)
• Fix an undefined 'orig_bib' bug (that probably caused no problems)
• Add filters to renumber citations in the case where they're referenced by citation number. These filters won't do anything in other cases, but nevertheless probably shouldn't be here

Also:

• The existence of the slightly different (and older?) pandoc-lua-filters multiple-bibliographies filter is rather confusing. I don't think there's anything else there that needs to be carried over (it was only really the citeproc 'quiet' support)
• I think it would be great if utils.citeproc() supported a 'quiet' flag, and plan to create a pandoc issue for this
• There's a comment in the code about not copying all the metadata before running citeproc, but I think this is probably a mistake because it means that any new relevant metadata items will have to be explicitly added, e.g., link-bibliography was added in pandoc 2.14.2 but hadn't been added to the filter
• As noted in the commit comment, the 'renumber' filters should probably be moved to another file

[wlupton]

@tarleb (and others), any thoughts on this PR? Thanks.

[jerome-white]

What I like about this PR is that reference numbers do not "reset" between bibliographies (see #6). This PR generally works, with a couple caveats:

1. When sorting the bibliography by "citation-number": the reference numbers in the document do not always align to the correct numbers in the bibliography. The citation may list "4-7", for example, when in fact it should be "31-34". This is also an issue I notice with the implementation on the master branch, so it is not something this PR introduces.
2. When calling Pandoc, I specify the option --metadata link-citations:true. While grouped citations (for example [@a; @b; @c]) are properly linked to references, single citations ([@a]) are not. I do not see this issue with the implementation on master.

[wlupton]

Sorry for the delay in replying. I suspect that both of your caveats can be addressed. I'll take a look.

Re your caveat 1 please note that there's already an attempt to address this problem, but it doesn't support ranges.

[tarleb]

Very cool! Could you split this up into separate PRs? Things that I'm definitely in favor of:

• General cleanup
• Allow lists of bib files for each topic
• Allow to renumber references (but maybe that should be configurable?)

I'm a bit skeptical towards others:

• We may want to wait for Lua: add a pandoc.log module. [API change] jgm/pandoc#9668, as that will allow to run citeproc quietly.
• Copying all metadata might be the right thing to do, but I like to be explicit about these things, so I'm not sure.

[wlupton]

@tarleb I'll take a look.

And @jerome-white apologies for not doing what I said I'd do (perhaps on this upcoming iteration...).

[wlupton]

I've rebased against the latest pandoc-ext/multibib in preparation for making the requested changes. I know this has been hanging around for a long time, but I really will try to take a look next week (the thing is... I don't actually use this filter, but it would be nice to get this PR cleaned up and merged).

[wlupton]

@tarleb with which pandoc versions should we try to make this filter work? I installed 2.11 (which is specifically called out as the earliest possible version), ran with it and all the other m.n versions that I happen to have installed, and see the following (it fails for all 2.x, works with warnings for 3.0 and 3.1, and works silently since 3.2):

% for v in 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 3.0 3.1 3.2 3.3 3.4 3.5 3.6; \
    do echo "#### pandoc-$v ####"; make PANDOC=pandoc-$v PANDOC_VERSION=$v >/dev/null; done
#### pandoc-2.11 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	[C]: in function 'pandoc.utils.run_json_filter'
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-2.12 ####
/bin/sh: pandoc-2.12: command not found
make: *** [Makefile:16: test] Error 1
#### pandoc-2.13 ####
/bin/sh: pandoc-2.13: command not found
make: *** [Makefile:16: test] Error 1
#### pandoc-2.14 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	[C]: in function 'pandoc.utils.run_json_filter'
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-2.15 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22,1] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	[C]: in function 'pandoc.utils.run_json_filter'
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-2.16 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22,1] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	[C]: in function 'pandoc.utils.run_json_filter'
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-2.17 ####
/bin/sh: pandoc-2.17: command not found
make: *** [Makefile:16: test] Error 1
#### pandoc-2.18 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22,2] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-2.19 ####
JSON parse error: Error in $: Incompatible API versions: encoded with [1,22,2] but attempted to decode with [1,23,1].
Error running filter multibib.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	multibib.lua:72: in upvalue 'citeproc'
	multibib.lua:106: in function <multibib.lua:78>
make: *** [Makefile:16: test] Error 1
#### pandoc-3.0 ####
[WARNING] Citeproc: citation Bae not found
[WARNING] Citeproc: citation Knu86 not found
[WARNING] Citeproc: citation Bel not found
[WARNING] Citeproc: citation Nie72 not found
#### pandoc-3.1 ####
[WARNING] Citeproc: citation Bae not found
[WARNING] Citeproc: citation Knu86 not found
[WARNING] Citeproc: citation Bel not found
[WARNING] Citeproc: citation Nie72 not found
#### pandoc-3.2 ####
#### pandoc-3.3 ####
/bin/sh: pandoc-3.3: command not found
make: *** [Makefile:16: test] Error 1
#### pandoc-3.4 ####
/bin/sh: pandoc-3.4: command not found
make: *** [Makefile:16: test] Error 1
#### pandoc-3.5 ####
#### pandoc-3.6 ####

[wlupton]

@tarleb, I've removed everything relating to renumbering and hope that this PR is now acceptable, and can be merged.

I've also created PR #18 with a separate multibib-renumber filter but this is just the code that I removed from this PR, and will need further work. I'll take a look at @jerome-white's issues:

• The first issue should definitely be looked at on PR Add 'multibib-renumber.lua' filter #18 because it relates to numbered references
• The second issue might be related to either of the two PRs ... not sure yet