Merge branch 'feat/yaml-programmatic-text' into feat/first-class-yaml

CHANGES.md

@@ -2,11 +2,21 @@
 
 ## 3.7.0
 
+Development:
+
+- Support programmatic text in YAML metadata values.
+  (istqborg/istqb_product_base#46, #440, #451, sponsored by @istqborg)
+
 Fixes:
 
 - Produce more meaningful error message when KPathSea fails to
   locate a file. (#458, #472, 33e6eef2, reported by @Yggdrasil128)
 
+Deprecation:
+
+- Deprecate `jekyllDataString` renderer and renderer prototype.
+  (istqborg/istqb_product_base#46, #440, #451, sponsored by @istqborg)
+
 ## 3.6.2 (2024-07-14)
 
 Fixes:

README.md

@@ -282,7 +282,7 @@ Acknowledgements
 | [<img width="150" src="https://www.fi.muni.cz/images/fi-logo.png">][fimu] | I gratefully acknowledge the funding from the [Faculty of Informatics][fimu] at the [Masaryk University][mu] in Brno, Czech Republic, for the development of the Markdown package in projects [MUNI/33/12/2015][], [MUNI/33/1784/2020][], [MUNI/33/0776/2021][], [MUNI/33/1654/2022][], and [MUNI/33/1658/2022][]. |
 | [<img width="150" src="https://cdn.overleaf.com/img/ol-brand/overleaf_og_logo.png">][overleaf] | Extensive user documentation for the Markdown package was kindly written by [Lian Tze Lim][liantze] and published by [Overleaf][]. |
 | [<img width="150" src="https://pbs.twimg.com/profile_images/1004769879319334912/6Bh1UthD.jpg">][omedym] | Support for content slicing (Lua options [`shiftHeadings`][option-shift-headings] and [`slice`][option-slice]) and pipe tables (Lua options [`pipeTables`][option-pipe-tables] and [`tableCaptions`][option-table-captions]) was graciously sponsored by [David Vins][dvins] and [Omedym][]. |
-| [<img width="150" src="https://www.istqb.org/static/istqb-logo-1b043e800a580724ad223567f9ea57c0.png">][istqb] | Fixes for issues [#359][issue-359], [#368][issue-368], [#401][issue-401], and [#424][issue-424] were graciously sponsored by the [International Software Testing Qualifications Board (ISTQB)][istqb]. |
+| [<img width="150" src="https://www.istqb.org/static/istqb-logo-1b043e800a580724ad223567f9ea57c0.png">][istqb] | Fixes for issues [#359][issue-359], [#368][issue-368], [#401][issue-401], [#424][issue-424], and [#440][issue-440] were graciously sponsored by the [International Software Testing Qualifications Board (ISTQB)][istqb]. |
 
  [dvins]:  https://github.com/dvins             "David Vins"
  [fimu]:   https://www.fi.muni.cz/index.html.en "Faculty of Informatics, Masaryk University"
@@ -294,6 +294,7 @@ Acknowledgements
  [issue-368]: https://github.com/witiko/markdown/issues/368 "Tables nested in list items have empty lines"
  [issue-401]: https://github.com/witiko/markdown/issues/401 "Create an unnumbered section"
  [issue-424]: https://github.com/witiko/markdown/issues/424 "E-mail addresses are incorrectly interpreted as bracketed citations"
+ [issue-440]: https://github.com/witiko/markdown/issues/440 "Support programmatic text in YAML metadata values"
 
  [option-pipe-tables]:    https://mirrors.ctan.org/macros/generic/markdown/markdown.html#pipe-tables          "Markdown Package User Manual"
  [option-shift-headings]: https://mirrors.ctan.org/macros/generic/markdown/markdown.html#option-shiftheadings "Markdown Package User Manual"

examples/optex.tex

@@ -226,7 +226,8 @@
 \_let  \markdownRendererJekyllDataSequenceEnd \_relax
 \_def  \markdownRendererJekyllDataBoolean #1#2{}
 \_def  \markdownRendererJekyllDataNumber #1#2{}
-\_def  \markdownRendererJekyllDataString #1#2{}
+\_def  \markdownRendererJekyllDataProgrammaticString #1#2{}
+\_def  \markdownRendererJekyllDataTypographicString #1#2{}
 \_def  \markdownRendererJekyllDataEmpty #1{}
 
 % Load the Markdown module and set TeX macros for the Markdown module

markdown.dtx

@@ -20320,12 +20320,24 @@ following \acro{yaml} serialization rules.
 %
 % \begin{markdown}
 
-The \mdef{markdownRendererJekyllDataString} macro represents a string scalar
-value in a \acro{yaml} document. This macro will only be produced when the
-\Opt{jekyllData} option is enabled. The macro receives two arguments: the
+The \mdef{markdownRendererJekyllDataTypographicString} and
+\mdef{markdownRendererJekyllDataProgrammaticString} macros represent string
+scalar values in a \acro{yaml} document. This macro will only be produced when
+the \Opt{jekyllData} option is enabled. The macro receives two arguments: the
 scalar key in the parent structure, cast to a string following \acro{yaml}
 serialization rules, and the scalar value.
 
+For each string scalar value, both macros are produced. Whereas
+\mref{markdownRendererJekyllDataTypographicString} receives the scalar value
+after all markdown markup and special \TeX{} characters in the string have been
+replaced by \TeX{} macros, \mref{markdownRendererJekyllDataProgrammaticString}
+receives the raw scalar value. Therefore, whereas the
+\mref{markdownRendererJekyllDataTypographicString} macro is more appropriate
+for texts that are supposed to be typeset with \TeX{}, such as document titles,
+author names, or exam questions, the
+\mref{markdownRendererJekyllDataProgrammaticString} macro is more appropriate
+for identifiers and other programmatic text that won't be typeset by \TeX{}.
+
 % \end{markdown}
 %
 % \iffalse
@@ -20334,9 +20346,80 @@ serialization rules, and the scalar value.
 % \fi
 %
 %  \begin{macrocode}
-\def\markdownRendererJekyllDataString{%
-  \markdownRendererJekyllDataStringPrototype}%
+\def\markdownRendererJekyllDataTypographicString{%
+  \markdownRendererJekyllDataTypographicStringPrototype}%
+\def\markdownRendererJekyllDataProgrammaticString{%
+  \markdownRendererJekyllDataProgrammaticStringPrototype}%
 \ExplSyntaxOn
+\seq_gput_right:Nn
+  \g_@@_renderers_seq
+  { jekyllDataTypographicString }
+\prop_gput:Nnn
+  \g_@@_renderer_arities_prop
+  { jekyllDataTypographicString }
+  { 2 }
+\seq_gput_right:Nn
+  \g_@@_renderers_seq
+  { jekyllDataProgrammaticString }
+\prop_gput:Nnn
+  \g_@@_renderer_arities_prop
+  { jekyllDataProgrammaticString }
+  { 2 }
+\ExplSyntaxOff
+%    \end{macrocode}
+% \par
+%
+% \iffalse
+%</tex>
+%<*manual-tokens>
+% \fi
+%
+% \begin{markdown}
+
+Before Markdown 3.7.0, the \mref{markdownRendererJekyllDataTypographicString}
+macro was named \mref{markdownRendererJekyllDataString} and the
+\mref{markdownRendererJekyllDataProgrammaticString} macro was not produced.
+The \mref{markdownRendererJekyllDataString} has been deprecated and will be
+removed in Markdown 4.0.0.
+
+% \end{markdown}
+%
+% \iffalse
+%</manual-tokens>
+%<*tex>
+% \fi
+%
+%  \begin{macrocode}
+\ExplSyntaxOn
+\cs_gset:Npn
+  \markdownRendererJekyllDataTypographicString
+  {
+    \cs_if_exist:NTF
+      \markdownRendererJekyllDataString
+      {
+        \markdownWarning
+          {
+            The~jekyllDataString~renderer~has~been~deprecated,~
+            to~be~removed~in~Markdown~4.0.0
+          }
+        \markdownRendererJekyllDataString
+      }
+      {
+        \cs_if_exist:NTF
+          \markdownRendererJekyllDataStringPrototype
+          {
+            \markdownWarning
+              {
+                The~jekyllDataString~renderer~prototype~has~been~deprecated,~
+                to~be~removed~in~Markdown~4.0.0
+              }
+            \markdownRendererJekyllDataStringPrototype
+          }
+          {
+            \markdownRendererJekyllDataTypographicStringPrototype
+          }
+      }
+  }
 \seq_gput_right:Nn
   \g_@@_renderers_seq
   { jekyllDataString }
@@ -20398,7 +20481,7 @@ following content:
 ``` tex
 \input markdown
 \def\markdownOptionJekyllData{true}
-\def\markdownRendererJekyllDataString#1#2{\gdef\name{#2}}
+\def\markdownRendererJekyllDataTypographicString#1#2{\gdef\name{#2}}
 \def\markdownRendererJekyllDataNumber#1#2{\gdef\age{#2}}
 \def\markdownRendererJekyllDataEnd{%
   \name{} is \age{} years old.}
@@ -20428,7 +20511,7 @@ following content:
 \usepackage[jekyllData]{markdown}
 \markdownSetup{
   renderers = {
-    jekyllDataString = {\gdef\name{#2}},
+    jekyllDataTypographicString = {\gdef\name{#2}},
     jekyllDataNumber = {\gdef\age{#2}},
     jekyllDataEnd = {\name{} is \age{} years old.},
   }
@@ -20460,7 +20543,7 @@ following content:
 ``` tex
 \usemodule[t][markdown]
 \setupmarkdown[jekyllData = yes]
-\def\markdownRendererJekyllDataString#1#2{\gdef\name{#2}}
+\def\markdownRendererJekyllDataTypographicString#1#2{\gdef\name{#2}}
 \def\markdownRendererJekyllDataNumber#1#2{\gdef\age{#2}}
 \def\markdownRendererJekyllDataEnd{%
   \name{} is \age{} years old.}
@@ -20502,9 +20585,26 @@ following text:
 \ExplSyntaxOn
 \cs_new:Nn \@@_define_renderers:
   {
-    \seq_map_function:NN
+    \seq_map_inline:Nn
       \g_@@_renderers_seq
-      \@@_define_renderer:n
+      {
+%    \end{macrocode}
+% \par
+% \begin{markdown}
+%
+% The \mref{markdownRendererJekyllDataString} has been deprecated and will be
+% removed in Markdown 4.0.0.
+%
+% \end{markdown}
+%  \begin{macrocode}
+        \str_if_eq:nnF
+          { ##1 }
+          { jekyllDataString }
+          {
+            \@@_define_renderer:n
+              { ##1 }
+          }
+      }
   }
 \cs_new:Nn \@@_define_renderer:n
   {
@@ -21220,9 +21320,26 @@ following text:
 \ExplSyntaxOn
 \cs_new:Nn \@@_define_renderer_prototypes:
   {
-    \seq_map_function:NN
+    \seq_map_inline:Nn
       \g_@@_renderers_seq
-      \@@_define_renderer_prototype:n
+      {
+%    \end{macrocode}
+% \par
+% \begin{markdown}
+%
+% The \mref{markdownRendererJekyllDataString} has been deprecated and will be
+% removed in Markdown 4.0.0.
+%
+% \end{markdown}
+%  \begin{macrocode}
+        \str_if_eq:nnF
+          { ##1 }
+          { jekyllDataString }
+          {
+            \@@_define_renderer_prototype:n
+              { ##1 }
+          }
+      }
   }
 \cs_new:Nn \@@_define_renderer_prototype:n
   {
@@ -25359,6 +25476,8 @@ function M.writer.new(options)
     ["{"] = "\\markdownRendererLeftBrace{}",
     ["}"] = "\\markdownRendererRightBrace{}",
     ["\\"] = "\\markdownRendererBackslash{}",
+    ["\r"] = " ",
+    ["\n"] = " ",
   }
   self.escaped_minimal_strings = {
     ["^^"] = "\\markdownRendererCircumflex\\markdownRendererCircumflex ",
@@ -25408,9 +25527,9 @@ function M.writer.new(options)
 %
 % Use the \luamref{writer->escaped_chars}, \luamref{writer->escaped_uri_chars},
 % and \luamref{writer->escaped_minimal_strings} tables to create the
-% \luamdef{writer->escape_typographic_text},
-% \luamdef{writer->escape_programmatic_text}, and
-% \luamdef{writer->escape_minimal} escaper functions.
+% \luamdef{escape_typographic_text},
+% \luamdef{escape_programmatic_text}, and
+% \luamdef{escape_minimal} local escaper functions.
 %
 % \end{markdown}
 %  \begin{macrocode}
@@ -32835,7 +32954,8 @@ M.extensions.jekyll_data = function(expect_jekyll_data, ensure_jekyll_data)
 % the key `p` in the parent table; if `p` is nil, then the table has no parent.
 % All scalar keys and values encountered in the table will be cast to a string
 % following \acro{yaml} serialization rules. String values will also be
-% transformed using the function `t`.
+% transformed using the function `t` for the typographic output format used by
+% the \mref{markdownRendererJekyllDataTypographicString} macro.
 %
 % \end{markdown}
 %  \begin{macrocode}
@@ -32918,7 +33038,12 @@ M.extensions.jekyll_data = function(expect_jekyll_data, ensure_jekyll_data)
               table.insert(buf, v)
               table.insert(buf, "}")
             elseif typ == "string" then
-              table.insert(buf, "\\markdownRendererJekyllDataString{")
+              table.insert(buf, "\\markdownRendererJekyllDataProgrammaticString{")
+              table.insert(buf, k)
+              table.insert(buf, "}{")
+              table.insert(buf, self.identifier(v))
+              table.insert(buf, "}")
+              table.insert(buf, "\\markdownRendererJekyllDataTypographicString{")
               table.insert(buf, k)
               table.insert(buf, "}{")
               table.insert(buf, t(v))
@@ -34048,7 +34173,17 @@ end
     { #1 }
     { #2 }
 }
-\def\markdownRendererJekyllDataStringPrototype#1#2{
+%    \end{macrocode}
+% \par
+% \begin{markdown}
+%
+% We will process all string scalar values assuming that they may contain
+% markdown markup and are intended for typesetting.
+%
+% \end{markdown}
+%  \begin{macrocode}
+\def\markdownRendererJekyllDataProgrammaticStringPrototype#1#2{}
+\def\markdownRendererJekyllDataTypographicStringPrototype#1#2{
   \markdown_jekyll_data_set_keyvals:nn
     { #1 }
     { #2 }

tests/support/keyval-setup.tex

@@ -147,7 +147,7 @@
     \TYPE{#0: \NORMALIZEPATH{#1}}},
   (ticked|halfTicked|unticked)Box = {%
     \TYPE{#0}},
-  jekyllData(Boolean|Number|String) = {%
+  jekyllData(Boolean|Number|(Programmatic|Typographic)String) = {%
     \TYPE{BEGIN #0}%
     \TYPE{- key:   #1}%
     \TYPE{- value: #2}%

tests/testfiles/unit/lunamark-markdown/jekyll-data-mixed-keys.test → tests/testfiles/regression/github/issue-433.test

@@ -19,14 +19,22 @@ BEGIN jekyllDataMappingBegin
 - key: null
 - length: 2
 END jekyllDataMappingBegin
-BEGIN jekyllDataString
+BEGIN jekyllDataProgrammaticString
 - key: 1
 - value: foo
-END jekyllDataString
-BEGIN jekyllDataString
+END jekyllDataProgrammaticString
+BEGIN jekyllDataTypographicString
+- key: 1
+- value: foo
+END jekyllDataTypographicString
+BEGIN jekyllDataProgrammaticString
+- key: a
+- value: bar
+END jekyllDataProgrammaticString
+BEGIN jekyllDataTypographicString
 - key: a
 - value: bar
-END jekyllDataString
+END jekyllDataTypographicString
 jekyllDataMappingEnd
 jekyllDataEnd
 END document