Integrate with jgm/pandoc

[Witiko]

Currently, a user of the Markdown package is restricted in their choice of syntax extensions to the ones provided by the Lua parser implemented in markdown.lua. To provide experimental ground for implementing new syntax extensions, support for the internal abstract syntax tree (AST) format of the jgm/pandoc converter will be added.

Currently, jgm/pandoc can be used to provide conversion from various input formats to Markdown:

pandoc -f docx -t markdown input.docx -o input.md

The Markdown package can then be used to convert the Markdown document to the TeX abstract syntax tree format (TeX AST) produced by the Markdown package:

texlua /path/to/markdown-cli.lua input.md input.tex

This representation can then be typeset. This is useful, but limited to the Markdown syntax extensions supported by our Lua parser.

---

The plan is to provide a jgm/pandoc Lua writer (see jgm/pandoc issues 4341 and 1541 for futher information) that will directly convert the jgm/pandoc AST to the TeX AST, circumventing the Lua parser altogether:

pandoc -f docx -t /path/to/markdown-pandoc_writer.lua input.docx -o input.tex

Adding initial support for a new syntax extension already supported by jgm/pandoc will then be as easy as adding a new procedure to the writer and defining the corresponding \markdownRenderer… macros. Full support can be added later by extending our Lua parser.

---

TODOs:

• Finish lunamark implementation, open pull request.
• Create a proof of concept in drehak/pandoc-to-markdown.
• Write a TUGboat article introducing the proof of concept.

[Witiko]

Adding a jgm/pandoc reader for reconstructing a jgm/pandoc AST from a TeX AST would be also benefitial. This reader would be more of a plumbing tool for restoring a document from the intermediary TeX AST files for cases where the original sources are unavailable. However, there does not seem to be any Lua API for readers, so we either need to abuse the AST and create a Lua filter, or we would need to create a Haskell reader and contribute it to jgm/pandoc, as discussed in jgm/pandoc#1541 (comment). Similarly to a Haskell reader, we could also contribute a Haskell writer that would replace the Lua writer in the long run and would be maintained as a part of jgm/pandoc.

[Witiko]

An exhaustive specification of the elements of Pandoc's AST format is available on Hackage.
The full list of Lua functions reserved for Lua writers is available in jgm/pandoc's src/Text/Pandoc/Writers/Custom.hs.

[Witiko]

[...] create a Haskell reader and contribute it to jgm/pandoc, as discussed in jgm/pandoc#1541 (comment). Similarly to a Haskell reader, we could also contribute a Haskell writer that would replace the Lua writer in the long run and would be maintained as a part of jgm/pandoc.

@drehak I have created a development environment for Pandoc using Docker at witiko/pandoc-devenv. We can use it to develop Haskell readers and writers for Pandoc without littering our base OS with Haskell. Those who want to litter their base OS can take inspiration in our Dockerfile.

[Witiko]

A preliminary analysis by for the implementation has been authored by @drehak and published in the CSTUG Bulletin 2021/1-4 (landing page, PDF).

[Witiko]

@drehak We should aim to close this issue before milestone 2.15.0 (due on March 31), since the defense of your student project will likely take place before then and also because we'd like to publicize your proof of concept in a journal article for TUGboat 43:1 (also due on March 31, see #120).

[Witiko]

Tentative roadmap for @xrehak's bachelor's thesis

• In summer 2022:
  • Close Modernize LaTeX code using xparse and expl3 #96.
  • Help prepare materials for presentation at TUG 2022.
  • Close Add missing functions to pandoc-to-markdown.lua drehak/pandoc-to-markdown#8
  • Place CSTUG and TUGboat articles to Publications in IS MU.
• In fall 2022:
  • Discuss future directions with @jgm and pandoc-discuss when TUGboat 43:1 with our article has come out.
  • Write the bachelor's thesis.
    • Copy the template
    • Write the outline (Introduction, State of the Art, ..., Conclusion)
    • Migrate the CSTUG article to the State of the Art chapter

Future work

• Implement feedback of @jgm and pandoc-discuss to the Lua writer proof of concept:
  • Replace \pandocSpace with space character drehak/pandoc-to-markdown#18
  • Replace \pandocSoftBreak with newline character drehak/pandoc-to-markdown#19
• Rewrite the Lua writer to Haskell using pandoc-devenv and offer it to @jgm in Plain TeX writer jgm/pandoc#1541.
• Make the \pandoc* commands less dependent on the internal format of the Markdown package and more faithful to the structure of the Pandoc AST. A good guiding principle is that we should be able to write a reader that allows a round trip from AST to plain TeX and back. Practically speaking, this means that we will be moving a lot of the logic out of the Lua reader and into the plain TeX code that rewrites the \pandoc* commands into the \markdown* commands.
• Implement support for syntax extensions:
  • Support strike-through drehak/pandoc-to-markdown#9
  • Support superscripts and subscripts drehak/pandoc-to-markdown#10
  • Support fancy lists drehak/pandoc-to-markdown#11
• Finish Lunamark proof of concept and open a pull request to jgm/lunamark.
  • Outline issues with the current proof of concept.
  • Close the issues.
  • Open pull request to jgm/lunamark.
• Update our custom writer for Pandoc 3.0.