Verbatim line breaking issue

[hlandau]

\begin[class=book]{document}
\use[module=packages.autodoc]
\use[module=packages.verbatim]
\font[size=11pt,family=TeX Gyre Heros]
\chapter{Test}

Sollicitudin. Mauris aliquam mauris ut tortor. Pellentesque tincidunt mattis
nibh. In id lectus eu magna vulputate ultrices. Aliquam interdum varius enim.
Maecenas at mauris. Sed sed nibh. Nam non turpis. Maecenas fermentum nibh in
est. Pellentesque habitant morbi tristique senectus et netus et malesuada fames
ac turpis egestas.

Duis sagittis fermentum nunc. Nullam elementum erat. Quisque dapibus, augue nec
dapibus bibendum, velit enim scelerisque sem, accumsan suscipit lectus odio ac
justo. Fusce in felis a enim rhoncus placerat. Cras nec eros et mi egestas
facilisis. In hendrerit tincidunt neque. Maecenas tellus. Fusce sollicitudin
molestie dui. Sed magna orci, accumsan nec, viverra non, pharetra id, dui.

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nullam placerat mi
vitae felis. In porta, quam sit amet sodales elementum, elit dolor aliquam
elit, a commodo nisi felis nec nibh. Nulla facilisi. Etiam at tortor. Vivamus
quis sapien nec magna scelerisque lobortis.

Curabitur tincidunt viverra justo. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Sed eros ante, mattis ullamcorper,
posuere quis, tempor vel, metus. Maecenas cursus cursus lacus. Sed risus magna,
aliquam sed, suscipit sit amet, porttitor quis, odio. Suspendisse cursus justo
nec urna. Suspendisse potenti. In hac habitasse platea dictumst. Cras quis
lacus. Vestibulum rhoncus congue lacus. Vivamus euismod, felis quis commodo
viverra, dolor elit dictum ante, et mollis eros augue at est. Class aptent
taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos.
Nulla lectus sem, tristique sed, semper in, hendrerit non, sem. Vivamus
dignissim massa in ipsum. Morbi fringilla ullamcorper ligula. Nunc turpis.
Mauris vitae sapien. Nunc luctus bibendum velit.

Morbi faucibus volutpat sapien. Nam ac mauris at justo adipiscing facilisis.
Nunc et velit. Donec auctor, nulla id laoreet volutpat, pede erat feugiat ante,
auctor facilisis dui augue non turpis. Suspendisse mattis metus et justo.
Aliquam erat volutpat. Suspendisse potenti. Nam hendrerit lorem commodo metus
laoreet ullamcorper. Proin vel nunc a felis sollicitudin pretium. Maecenas in
metus at mi mollis posuere. Quisque ac quam sed massa adipiscing rutrum.
Vestibulum ipsum. Phasellus porta sapien. Maecenas venenatis tellus vel tellus.

Aliquam aliquam dolor at justo. Cum sociis natoque penatibus et magnis dis
parturient montes, nascetur ridiculus mus. Morbi pretium purus a magna. Nullam
dui tellus, blandit eu, facilisis non, pharetra consectetuer, leo. Maecenas sit
amet ante sagittis magna imperdiet pulvinar. Vestibulum a lacus at sapien
suscipit tempus. Proin pulvinar velit sed nulla. Curabitur aliquet leo ac
massa. Praesent posuere lectus vitae odio. Donec imperdiet urna vel ante. In
semper accumsan diam. Vestibulum porta justo. Suspendisse egestas commodo eros.

\begin{verbatim}
Linebreak test for verbatim block

sacjljdlsjadjsaodjsaodjsaldjsalkdjsalkdjksalkdjaslkdjkasldjsaldkjasdlajsldasjlkdasjladsadlasjdalkncmxlskmxclkaxmslakdjsalkdjaslkdjalkdjaskldjadlkslkajs
mu
\end{verbatim}

\begin[type=autodoc:codeblock]{raw}
Linebreak test for verbatim block

sacjljdlsjadjsaodjsaodjsaldjsalkdjsalkdjksalkdjaslkdjkasldjsaldkjasdlajsldasjlkdasjladsadlasjdalkncmxlskmxclkaxmslakdjsalkdjaslkdjalkdjaskldjadlkslkajs
mu
\end{raw}

\end{document}

Long lones of text (e.g. SHA512 hashes) in verbatim blocks run beyond the text area (and off the end of the page), unlike for text which contains spaces.

$ sile --version
SILE v0.14.9 (Lua 5.3)

[Omikhleia]

Long lones of text (e.g. SHA512 hashes) in verbatim blocks run beyond the text area (and off the end of the page), unlike for text which contains spaces.

If the question is not just theoretical (a SHA512 in print, really?), what is the expectation on where to break such text?
An input filter could easily add breakpoints between each character (or every 2 characters, so as to respect hexadecimal?), but this seems somewhat heavy handed ^^

[hlandau]

I'm exploring the viability of sile and other solutions for unattended typesetting... Long lines of text are certainly possible in transcripts of terminal output and it can't be guaranteed there's any spaces in that text. This pipeline of mine would handle arbitrary documents in an automated fashion, so these cases would need to be handled sensibly. Suboptimal linebreaking is OK, so long as it doesn't run off the text area. Would it be possible?

[Omikhleia]

Suboptimal linebreaking is OK, so long as it doesn't run off the text area. Would it be possible?

Yes, that would be doable, and I can even think of several ways to achieve it :)

• Since SILE's current "verbatim" environment is not really verbatim (it's somewhat ill-named, but so says the documentation, so it's not hidden) one can invoke commands there, and a command with an inputfilter could add breakpoints at "relevant" points. (Hence my question on the initial use case, because I would personally prefer not breaking hexadecimal numbers at random points, so I would split them 2 by 2 there... Still taking the risk of overlong lines, but what is the best solution for a given problem?).
• It would also be possible in a more general way, for instance with a dedicated typesetter with a different line-breaking strategy (i.e. not pattern/language-based). That would even be easier than the previous point.

But I'll be real honest, I am not sure how it really relates to "the viability" of a solution (would it be SILE or another). As I use to say to friends (whatever tool they are using), the ultimate aim is before all about getting typography done right in print-like quality, but please "Be Good Typographers"... Breaking lines at random points cannot be good typography, however one looks at it, it can only be a last-chance way of handling bad things with a weird fallback ultimately, IMHO.

Also, as of "viability" of a typesetting software is concerned, is "verbatim" the right thing to pick and compare? (This is a serious question.)

Nevertheless, I can however imagine and code such a thing -- One good point about SILE is that nearly everything is replaceable, so everything doable in Lua can make its way into it even in the most obscure place of it (i.e. it's not limited to a few hooks, literally everything is replaceable)... But as for the standard distribution, this shouldn't be the role of the "default" environments such as verbatim (or even autodoc:codeblock, just mentioning it because it's in your test script, though it's kind of a different beast), I think... A 3rd party package can provide a "terminal-transcript" environment, and I bet it would not be too hard to do.¹

There are no dumb questions, and yours is certainly not². The question however still boils down to what would be the best typographic approach. Do you really think "random breaks" (likely unnoticed) are more satisfying than an overfull line (that one can notice and add a manual \break where seems best depending on real context?) I wouldn't be that sure it's a "viable" answer... It really depends on context. For instance, SILE does try to identify more breakpoints on long URLs formatting (via an inputfilter adding appropriate penalties at places where it seems breaking a URL is indeed acceptable).

Footnotes

1. I'd bet on less than 10 lines of code. Let's play, who's taking the bet, and what are the stakes? EDIT: I mean the command logic here. Obviously a package has some lines of "boilerplate" code, let's not consider these. ↩

2. I would however question the need for keeping of 6 paragraphs of gibberrish "lorem ispsum" that do not relate to the point being raised, but heh :) 🤣 ↩

[hlandau]

There are no dumb questions, and yours is certainly not2. The question however still boils down to what would be the best typographic approach. Do you really think "random breaks" (likely unnoticed) are more satisfying than an overfull line (that one can notice and add a manual \break where seems best depending on real context?) I wouldn't be that sure it's a "viable" answer... It really depends on context. For instance, SILE does try to identify more breakpoints on long URLs formatting (via an inputfilter adding appropriate penalties at places where it seems breaking a URL is indeed acceptable).

It's certainly true that an unattended typesetting flow isn't going to get perfect 'hand-tuned' output for everything one runs through it. But for my use case — which involves generating XHTML, PDF, etc. from a single source XML file — it's essential that if things aren't optimal, at least some basic parameters are met. So with the above example for example, not only do things overrun the text area, they actually run off the page and characters are lost. I will say I find that a very strange default, both in TeX and SILE. I can't really imagine the circumstances in which anyone might want it to handle things like this... ¹

My basic ambition here is to get something that

a) will perform adequately for essentially all input
b) which I can hand-tune later to get something more optimal from a typesetting perspective

tldr; I'm actually quite passionate about optimal typesetting... but when producing documentation one doesn't always have time to get there initially. Especially when one considers the alternatives are toolchains like FOP, which frankly produces far inferior output, one can't help but feel like this approach of TeX/SILE is making the perfect the enemy of the good... ²

Admittedly the TeX (and descendant) typesetting world seems so set on the idea of painstaking, hand-tuned typesetting, maybe I'm just going against the grain trying to do this. Yet there's a surprising dearth of other open source PDF tooling that really generates good output. Though it seems like plenty of people have used TeX in automated ways... ³

Thanks for the thoughtful response & for making SILE. ⁴

Footnotes

1. If there were a 'suboptimal emergency line break' function for text like this, it occurs to me it could still emit a warning like those for overfull hboxes so an attending typographer can be alerted to the issue. ↩

2. A bit of background, if you find it interesting: I actually have an XML toolchain for producing various output formats and my vague plan is to have a "fork" system which allows manual input of different strings for different output types. So if my standard conversion scripts, or the schema, can't reasonably accommodate optimal output for both XHTML and TeX/SILE in some highly bespoke typesetting situation, you could do something like

   <fork>
     <choice type="tex">code to typeset something in the best way in TeX/SILE/etc.</choice>
     <choice type="xhtml">code to typeset something in the best way in XHTML</choice>
   </fork>
   

   The idea is to have decent output all the time and let document authors add and maintain optimisations for specific output formats in the circumstances needed. ↩

3. Currently I'm sort of a refugee from ConTeXt, which has a nice way to accept XML input directly... theoretically this is ideal. What is less ideal is that many areas of it are wholly undocumented, and I have never seen a codebase more incomprehensible to me than ConTeXt's, meaning when it goes wrong I honestly find it beyond me to fix. If SILE's source itself is simply more comprehensible, that in itself would probably be a massive pro to me. Quite interesting how it's basically all Lua. ↩

4. TIL GitHub markdown supports footnotes. Even though I'm such a ridiculous user of them on my own site, I had no idea... 🎉 ↩

[Omikhleia]

@hlandau I moved the topic to Q&A discussions, I don't think it is a bug issue.

This said, here is a "hack", among all the possibilities.
It's clearly ugly to tweak the letter space logic this way, introducing (breakable) glues between nearly every character.
Anyway...

\begin[papersize=a6]{document}
\use[module=packages.verbatim]
\lua{
local class = SILE.documentState.documentClass
class:registerCommand("my-ugly-verbatim", function (_, content)
  local savedLetterSpace = SILE.nodeMakers.unicode.letterspace
  SILE.nodeMakers.unicode.letterspace = function (self)
    if self.token then self:makeToken() end
    if self.lastnode and self.lastnode ~= "glue" then
      coroutine.yield(SILE.nodefactory.glue())
      self.lastnode = "glue"
    end
  end
  SILE.call("verbatim", {}, function ()
    SILE.process(content)
  end)
  SILE.nodeMakers.unicode.letterspace = savedLetterSpace
end)
}

\begin{my-ugly-verbatim}
sacjljdlsjadjsaodjsaodjsaldjsalkdjsalkdjksalkdjaslkdjkasldjsaldkjasdlajsldasjlkdasjladsadlasjdalkncmxlskmxclkaxmslakdjsalkdjaslkdjalkdjaskldjadlkslkajs

sacjljdlsjadjsaodjsaodjsaldjs
alkdjsalkdjksalkdjaslkdjkasldjsaldkjasdlajsldasjlkdasjladsadlasjdalkncmxlskmxclkaxmslakdjsalkdjaslkdjalkdjaskldjadlkslkajs
\end{my-ugly-verbatim}

\end{document}

[hlandau]

This is perfect. Thanks for your time — I appreciate it. 👍