Trailing spaces or tabs are included in the code block’s content

[lostenderman]

See https://spec.commonmark.org/0.30/#example-118

[Witiko]

The corresponding unit test is testfiles/CommonMark_0.30/indented_code_blocks/012.test:

%   ---RESULT--- "example": 118,
%   
%   <pre><code>foo  
%   </code></pre>
%   
%   ---\RESULT---

<<<
    foo  
>>>
documentBegin
inputVerbatim: ./_markdown_test/e8d2133b16be538460e59776bb34676a.verbatim
documentEnd

Here is the result of running git checkout commonmark; cd tests; ./test.sh "testfiles/CommonMark_0.30/indented_code_blocks/012.test":

Testfile testfiles/CommonMark_0.30/indented_code_blocks/012.test
  Format templates/plain/
    Template templates/plain/input.tex.m4
      Command pdftex   --shell-escape                  --interaction=nonstopmode  test.tex
*** test-expected.log	2022-12-22 11:54:36.326074746 +0100
--- test-actual.log	2022-12-22 11:54:42.836019153 +0100
***************
*** 1,3 ****
  documentBegin
! inputVerbatim: ./_markdown_test/e8d2133b16be538460e59776bb34676a.verbatim
  documentEnd
--- 1,3 ----
  documentBegin
! inputVerbatim: ./_markdown_test/acbd18db4cc2f85cedef654fccc4a4d8.verbatim
  documentEnd

This issue seems related to the util.cache_verbatim() method, which "strips whitespaces from the end of [a string]". As far as I recall, this was supposed to ensure platform-independence, where as file such as the following would have a trailing \n on UNIX and no trailing \n on Windows:

Here is a fenced code without the closing fence:

```
foo

In theory, this should no longer be necessary, because we normalize the input before conversion (1, 2). In practice, we are just adding \n at the end of the input in the Lua CLI and plain TeX layes rather than only adding \n if it is missing at the end of the input. In the Lua layer used e.g. in the upcoming OpTeX layer (see Witiko#215 (comment)), no normalization takes place.

@lostenderman, can you please assign this issue to me? I will fix the normalization and move it directly to the convert() function in the Lua layer.

[lostenderman]

Closing with pull #121.