Improve ConTeXt interface

[Witiko]

The ConTeXt interface would benefit from a proper user interface such as the one LaTeX has. Being an academician first and a TeX enthusiast second, I have only had cursory exposure to ConTeXt. Therefore, I don't feel I understand the ConTeXt philosophy and syntax conventions enough to design a user interface for it. This leaves the ConTeXt users of the Markdown package with the plain TeX interface, which is not ugly per se, but which does feel out of place in a ConTeXt document.

See also the accurate description of the status quo by on the ConTeXt mailing list by @mojca:

Hi,

There's a really nice module for processing markdown:
https://github.com/Witiko/markdown
that has been presented during the TUG meeting and is included in TeX Live.

It theoretically works for ConTeXt (the output is perfectly usable),
but it needs a cleanup of the user interface to turn the options on
and off.

For example, it would be nice to use something like

\setupmarkdown
    [fencedCode=true]

rather than

\def\markdownOptionFencedCode{true}

Is anyone willing to look into the code and help the author fix the interface?

(I can add it to our distribution if needed.)

Mojca

I will appreciate any suggestions on how the user interface for ConTeXt should be redesigned to better fit in with the rest of ConTeXt and which ConTeXt mechanisms should be used to implement the suggested redesign.

[sami10007]

Hi,

Do you have any drafts on this? Any tests? Any timeline? Any first goals?

Do you they mean by the following the same as \setupmarkdown[fencedCode=true]?

 \setupmarkdown
        [fencedCode=true]

Best regards,

Sami

[Witiko]

Hi,

Do you have any drafts on this? Any tests? Any timeline? Any first goals?

There are no tests and no timeline.

There is currently minimal syntactic sugar for ConTeXt on top of the plain TeX interface (\startmarkdown and \stopmarkdown instead of \markdownBegin and \markdownEnd). I would like to extend this syntactic sugar to option setup (such as \setupmarkdown[fencedCode=true] instead of \def\markdownOptionFencedCode{true}).

As I stated in the original post, I rarely use ConTeXt, so I don't really have a feel for the ConTeXt philosophy and syntax conventions. Therefore, I am looking for a draft of the syntax and how it would be best implemented using the existing ConTeXt mechanisms.

Do you they mean by the following the same as \setupmarkdown[fencedCode=true]?

 \setupmarkdown
        [fencedCode=true]

Sure, these are equivalent from the viewpoint of ConTeXt.

[Witiko]

@mojca, I recall that you proposed a more complete ConTeXt syntax extension for the Markdown package after the TUG@BachoTeX 2017 conference. Have you noticed any desire to integrate the package more tightly with ConTeXt or other recent relevant feedback from the ConTeXt users?

[adityam]

@Witiko I have never looked at the package in detail, but I can help with the ConTeXt interface. Can you give more details on what you are looking at.

From the discussion in this thread, I understand that the current interface is driven using macros. Changing that to keyword interface should be straight forward. Is there a document where all the interface options are described?

In general, if you have any questions on anything related to ConTeXt, feel free to ping me.

[Witiko]

@Witiko I have never looked at the package in detail, but I can help with the ConTeXt interface. Can you give more details on what you are looking at.
From the discussion in this thread, I understand that the current interface is driven using macros. Changing that to keyword interface should be straight forward. Is there a document where all the interface options are described?

@adityam Thank you for your response. In LaTeX, users can configure the package when loading the package:

\documentclass{article}
\usepackage[hybrid, pipeTables]{markdown}
...
\begin{document}
...

They can also configure the package using the \markdownSetup command:

...
\markdownSetup{
  hybrid,
  pipeTables,
  renderers = {
    table = { ... },
  },
}
...

They can also configure the package locally for every piece of markdown code:

...
\begin{markdown}
$\sqrt{-1}$ *equals* $i$
\end{markdown}

\begin{markdown*}{hybrid}
$\sqrt{-1}$ *equals* $i$
\end{markdown*}
...

For ConTeXt, there is no such support and the users have to rely on redefining macros, which seems out-of-place in a ConTeXt document:

\usemodule[t][markdown]
\starttext

\startmarkdown
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown

\def\markdownOptionHybrid{true}
\startmarkdown
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown

\stoptext

I am looking for a suggestion as to how things should look, so that the interface is more friendly to the ConTeXt users.

In general, if you have any questions on anything related to ConTeXt, feel free to ping me.

Thanks again, much appreciated.

[mojca]

For ConTeXt one would probably want to use something like this

\usemodule
    [markdown]
\setupmarkdown
    [hybrid=off,
     tables=on]

\starttext

\startmarkdown
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown

\startmarkdown[hybrid] % or hybrid=on
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown

\stoptext

But there are some other options that could use other tricks. I've seen the blankBeforeHeading or shiftHeadings option, for example, which could be handled different, by merely setting up the headings in the desired way in ConTeXt. I wouldn't want markdown module to be too smart for me. Just convert the markdown heading into the suitable tex heading, and I can set up how the headings themselves look like in my own way.

@adityam: There is a long document with documentation describing all options in detail.

It's unlikely that I would manage to sit down and spend any reasonal amount of time for this myself, in particular given that I'm unlikely to ever use MarkDown myself (I hardly do any typesetting at all these days to be honest). I could review / provide feedback. But @adityam and some others like Wolfgang Schuster are big experts and can help with some excellent examples of good coding practice for ConTeXt.

[Witiko]

For ConTeXt one would probably want to use something like this

[...]

It's unlikely that I would manage to sit down and spend any reasonal amount of time for this myself, in particular given that I'm unlikely to ever use MarkDown myself (I hardly do any typesetting at all these days to be honest). I could review / provide feedback. But @adityam and some others like Wolfgang Schuster are big experts and can help with some excellent examples of good coding practice for ConTeXt.

@mojca Thank you, this is much appreciated.

But there are some other options that could use other tricks. I've seen the blankBeforeHeading or shiftHeadings option, for example, which could be handled different, by merely setting up the headings in the desired way in ConTeXt. I wouldn't want markdown module to be too smart for me. Just convert the markdown heading into the suitable tex heading, and I can set up how the headings themselves look like in my own way.

Not being too smart is the main philosophy behind the Markdown package (as opposed to Pandoc and other tools). The blankBeforeHeading option is a parser option, which decides whether

text
# heading
other text

will be parsed as follows:

text # heading other text

or as follows:

text

heading

other text

The Markdown spec is ambiguous in this regard, so we provide an option to choose the parser flavor. In general, most options control the parser, not the way Markdown elements are rendered. For this, there are renderers (redefinable TeX commands to which the Markdown elements expand to) and, indeed, the standard formatting setup of the ConTeXt format.

[adityam]

@adityam: There is a long document with [documentation](http://mirrors.ctan.org/macros/generic/markdown/markdown.html) describing all options in detail.

Thanks. I will go over this and try to understand all the options and then come back with suggestions of an interface that will fit with ConTeXt style.

[mojca]

The blankBeforeHeading option is a parser option,
/.../
The Markdown spec is ambiguous in this regard

Oh, I'm sorry. Yes, then this option makes perfect sense. I thought it was about how output was supposed to look like.

[adityam]

I am still trying to understand the overall structure and hope you don't mind me asking clarifying questions here. I am assuming that in ConTeXt, anyone running the module will be using MkIV (luatex engine).

1. By default, this module creates too many files.Even a simple hello world file creates jobname.markdown.in and cacheDir/md5sum.md.tex. Why? Why not simply store them in memory and write these file to disk if needed. That is the route taken my m-markdown. As a ConTeXt user I have gotten user to no temp file except a .log file and a tuc file. Any other temp file should be written only if absolutely necessary.

2. Is there a Lua function, which will take a string and an options table and return a parsed string? That will be the simplest route to go for ConTeXt instead of reading and writing to temp files.

3. I am by no means a Lua expert, but looking at the coding style used in ConTeXt I have always assumed that calls of the form namespace.function is more expensive than local function = namespace.function. markdown.lua uses explicit namespaces for the the parser code parser.something... Out of curiosity, is there any speed penalty due to using this style?

4. I am thinking that a good way to use the functionality will be write a context interface t-markdown.mkiv from scratch that does not call the plain tex interface at all. To streamline the transition, I will start writing a t-new-markdown.mkiv, which can then perhaps be merged if everything goes well. Is this okay?

5. Do you have a test suite with various examples? That will make it easier for me rather than creating examples "by hand".

[Witiko]

1. By default, this module creates too many files.

The main markdown document could be stored in-memory when LuaTeX is used. However, the code blocks markdown renderer expects a filename with the code block as an argument; these auxiliary files would be difficult to get rid of without breaking the established interface.

Note that auxiliary files can be useful. As an example, the Czech typography forbids a line ending with a non-syllabic preposition. To address this issue, the vlna program can be used to add a non-breaking space after all non-syllabic prepositions in a text file. With auxiliary files, you can easily use the vlna cacheDir/*.md.tex command to fix all Markdown texts in your document without touching the source.

2. Is there a Lua function, which will take a string and an options table and return a parsed string?

Yes, there is the Lua interface.

3. [...] Out of curiosity, is there any speed penalty due to using this style?

There is a minor speed penalty, since a hash table is used instead of local variables. This change was necessary for us, because there is an upper limit on the number of local variables in Lua, which we hit.

4. [...] Is this okay?

You will likely want to reuse some parts of t-markdown.tex, such as the ConTeXt renderers. Otherwise, yes, that would be ideal, thank you.

5. Do you have a test suite with various examples?

We have a rich set of unit tests.

[adityam]

1. By default, this module creates too many files.

The main markdown document could be stored in-memory when LuaTeX is used. However, the code blocks markdown renderer expects a filename with the code block as an argument; these auxiliary files would be difficult to get rid of without breaking the established interface.

OK. I'll stick with this for now.

1. Is there a Lua function, which will take a string and an options table and return a parsed string?

Yes, there is the Lua interface.

I meant something that does not create temp files. But I'll not go down that path for now.

1. [...] Is this okay?

You will likely want to reuse some parts of t-markdown.tex, such as the ConTeXt renderers. Otherwise, yes, that would be ideal, thank you.

Perhap. For most of the features, I think that it is better to follow the ConTeXt style for defining (and therefore changing) rendering options. For example, let's consider a simple ConTeXt file:

\usemodule[new-markdown]

\starttext
\startmarkdown
  Test of *basic* `markdown` **features**
\stopmarkdown
\stoptext

Here is a basic implmentation of the t-new-markdown.mkiv module which is able to parse and typeset the above example:

\unprotect

\definehighlight
  [markdownRendererEmphasis]
  [style=\em]

\definehighlight
  [markdownRendererStrongEmphasis]
  [style=bold]

\definehighlight
  [markdownRendererCodeSpan]
  [style=mono]

\startluacode
  thirddata = thirddata or {}
  thirddata.markdown = require("markdown")

  local markdown = thirddata.markdown

  function markdown.typesetbuffer(options, name)
      local convert = markdown.new(options)
      context(convert(buffers.getcontent(name)))
  end
\stopluacode

% Basic interface, without any option parsing
\definebuffer[markdown]

\unexpanded\def\stopmarkdown
      {\ctxlua{thirddata.markdown.typesetbuffer(nil, "\thedefinedbuffer{markdown}")}}

\protect

Using high-level ConTeXt macros (\definehighlight etc) instead of low level TeX commands (\def...) etc has the advantage that (i) you get a simple key-level driven interface to change features, (e.g., to get red colored emphasis, simply add \setuphighlight[markdownRendererEmphasis][color=red]. (ii) Generating structured outputs such as tagged-pdf and XHTML or EPUB export work out of the box.

[Witiko]

I meant something that does not create temp files

Not really. The auxiliary files are an early design decision, which affects the whole code base.

[...] For most of the features, I think that it is better to follow the ConTeXt style for defining (and therefore changing) rendering options.

That is fine, but the content of the renderer definitions can be reused.

[Witiko]

@adityam Any progress with the interface? In your #17 (comment), there is a redefinition of \startmarkdown and \stopmarkdown, which the package already defines, and a more idiomatic example code for redefining Markdown renderers, which is appreciated and will make its way to user examples for Markdown 2.10.0. However, the biggest desiderata are the Markdown options. Here is the current ConTeXt interface for options:

\usemodule[t][markdown]
\def\markdownOptionPipeTables{true}
\def\markdownOptionTableCaptions{true}
\starttext

\startmarkdown
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown

\begingroup
\def\markdownOptionHybrid{true}
\startmarkdown
$\sqrt{-1}$ *equals* $i$.
\stopmarkdown
\endgroup

\stoptext

Compare with the LaTeX interface for options, which is much more compact and idiomatic:

\documentclass{article}
\usepackage[pipeTables,tableCaptions]{markdown}
\begin{document}

\begin{markdown}
$\sqrt{-1}$ *equals* $i$
\end{markdown}

\begin{markdown*}{hybrid}
$\sqrt{-1}$ *equals* $i$
\end{markdown*}

\end{document}

Any change towards a more compact and idiomatic ConTeXt code would be greatly appreciated!

[Witiko]

Using high-level ConTeXt macros (\definehighlight etc) instead of low level TeX commands (\def...) etc has the advantage that (i) you get a simple key-level driven interface to change features, (e.g., to get red colored emphasis, simply add \setuphighlight[markdownRendererEmphasis][color=red]. (ii) Generating structured outputs such as tagged-pdf and XHTML or EPUB export work out of the box.

@adityam I would add these to the example documents, but we support ConTeXt MkII and the \definehighlight command doesn't seem to be available in MkII.

[adityam]

On Sun, 8 Aug 2021, Vít Novotný wrote: > Using high-level ConTeXt macros (\definehighlight etc) instead of low level TeX commands (\def...) etc has the advantage that (i) you get a simple key-level driven interface to change features, (e.g., to get red colored emphasis, simply add `\setuphighlight[markdownRendererEmphasis][color=red]`. (ii) Generating structured outputs such as tagged-pdf and XHTML or EPUB export work out of the box. @adityam I would add these to the example documents, but we support ConTeXt MkII and the `\definehighlight` command doesn't seem to be available in MkII.

Would it be possible to provide separate examples for MkII and MkIV. MkII has been in maintenance-only mode for almost a decade now. So, any ConTeXt user still using MkII will be an old user who is using MkII on a long-term project. Almost all ConTeXt documentation assumes that a new user is starting with either MkIV or even LMTX.

[Witiko]

@adityam We can definitely fork examples/context.tex into two separate files: context-mkii.tex for MkII and context-mkiv.tex for MkIV. Then, we can simplify examples/Makefile as follows:

diff --git a/examples/Makefile b/examples/Makefile
index e084a9a..548e4e2 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -14,13 +14,11 @@ all: $(OUTPUT)
 	$(MAKE) clean
 
 # This target typesets the ConTeXt example using the Mark II format.
-context-mkii.pdf: context.tex example.tex
-	cp context.tex context-mkii.tex
+context-mkii.pdf: context-mkii.tex example.tex
 	texexec --passon="--shell-escape --8bit" context-mkii.tex
 
 # This target typesets the ConTeXt example using the Mark IV format.
-context-mkiv.pdf: context.tex example.tex
-	cp context.tex context-mkiv.tex
+context-mkiv.pdf: context-mkiv.tex example.tex
 	context context-mkiv.tex
 
 # This target typesets the LaTeX example using the pdfTeX engine.

In order to include the two files in the distribution archive, the following line also needs to be updated in Makefile:

markdown/Makefile

Line 12 in 4b89f84

EXAMPLES_SOURCES=examples/context.tex examples/latex.tex

For LMTX, I am waiting for it to be added to TeX Live, since typesetting example documents is part of our continuous integration.

[Witiko]

I added an example document for ConTeXt MkIV.
Many thanks for your help, @adityam!

[Witiko]

It would be neat if we could have a \markdownSetup command that would replace this:

% Set options of the Markdown module.
\def\markdownOptionHashEnumerators{true}
\def\markdownOptionDefinitionLists{true}
\def\markdownOptionSmartEllipses{true}
\def\markdownOptionFootnotes{true}
\def\markdownOptionInlineFootnotes{true}
\def\markdownOptionFencedCode{true}
\def\markdownOptionContentBlocks{true}
\def\markdownOptionPipeTables{true}
\def\markdownOptionTableCaptions{true}
\def\markdownOptionTaskLists{true}

with this:

% Set options of the Markdown module.
\markdownSetup[hashEnumerators,
               definitionLists,
               smartEllipses,
               footnotes,
               inlineFootnotes,
               fencedCode,
               contentBlocks,
               pipeTables,
               tableCaptions,
               taskLists]

I wonder how we could do that.

[adityam]

I would recommend a slightly more context-ish syntax: \setupmarkdown[...] instead of \markdownSetup and using key=value rather than option (with only option, you need to figure out how to disable things).

ConTeXt doesn't use CamelCase a lot ... nevertheless, if you can consider HashEnumerators=yes (note capital H), then the interface is rather straight forward:

\unprotect

\definenamespace
  [markdown]
  [   \c!type=module,
      \c!name=markdown,
        setup=\v!yes,
    \s!parent=markdown,
  ]

\define[1]\markdown_parse_option
    {\doifelse{\markdownparameter{#1}}\v!yes
        {\setvalue{markdown#1}{true}}
        {\setvalue{markdown#1}{false}}}

\define\initialize_markdown
  {\processcommalist
        [
          HashEnumerators,
          DefinitionLists,
          SmartEllipses,
          Footnotes,
          InlineFootnotes,
          FencedCode,
          ContentBlocks,
          PipeTables,
          TableCaptions,
          TaskLists,
        ]\markdown_parse_option}

\appendtoks \initialize_markdown \to \everysetupmarkdown
        
\protect

You'd need to initialize the values somewhere:

\setupmarkdown
  [
    HashEnumerators=yes,
    DefinitionLists=yes,
    SmartEllipses=yes,
    Footnotes=yes,
    InlineFootnotes=yes,
    FencedCode=yes,
    ContentBlocks=yes,
    PipeTables=yes,
    TableCaptions=yes,
    TaskLists=yes,
  ]

Example usage:

\show\markdownHashEnumerators

\setupmarkdown[HashEnumerators=no]

\show\markdownHashEnumerators

If you'd prefer hashEnumerators=yes (with lowercase h), it may still be possible with some \uppercase magic, but my TeX-fu is weak when it comes to expansion and stuff...

[adityam]

... and of course, lowercase hashEumerator=yes is easy if we just type out everything:

\define\initialize_markdown
    {%
       \doifelse{\markdownparameter{hashEnumerators}}\v!yes
          {\def\markdownHashEnumerators{true}}
          {\def\markdownHashEnumerators{false}}
     % Same for other options
    }

[Witiko]

@adityam Thank you, that is helpful.

ConTeXt doesn't use CamelCase a lot ...

Does it use snake_case then? We could still make that work automatically.

... nevertheless, if you can consider HashEnumerators=yes (note capital H), then the interface is rather straight forward

Not all options are boolean. For example, the slice option is a string and the frozenCacheCounter option is a number. The Lua command-line interface sniffs the type from the default value of the options, so that it knows that slice=true should parse as a string rather than a boolean. The TeX interface of the package does not have access to this information and therefore ...

... and of course, lowercase hashEumerator=yes is easy if we just type out everything:

... that's what the LaTeX interface currently does. However, it's a pain to maintain and even moreso if we typed out everything for ConTeXt as well. Perhaps we should add runtime information about types, so that this can be automated.

[adityam]

ConTeXt doesn't use snake_case either. Almost all options are single word.

Another option is not to do any parsing at the TeX end. Simply pass everything as input by the user to Lua: something like ...

\directlua{ markdownSetup( { hashEnumerator = "\markdownparameter{HashEnumerator}", .... } )

This is laborious, but has to be done once in each format. Then, at the lua end, you can read the value of options.hashEnumerator and based on the value yes or no set the appropriate lua variable to be true or false. Since the parsing code is in Lua, we are not restricted by the limitations of TeX macros, and can easily write code which automates the processing.

[Witiko]

ConTeXt doesn't use snake_case either. Almost all options are single word.

Well, we can still run with that, it should be just a matter of replacing \markdownparameter{#1} with \markdownparameter{\lowercase{#1}}. We will need reflection, though, so that we don't need to maintain the comma list of all options in the ConTeXt module. I drafted an overly ambitious plan of how the reflection might look in #119.

Another option is not to do any parsing at the TeX end. Simply pass everything as input by the user to Lua

That would be technically possible, but I see two possible concerns:

1. Perhaps we don't want yes to be permissible in LaTeX. Generally, it seems to me that format-related conventions should be tied to the formats, not the generic Lua code.
2. We define the \markdownIfOption TeX command, which requires that we decide the truthiness of options in TeX, not Lua.

[Witiko]

This issue is blocked by #124.