Keep unknown commands

[stefan-kolb]

Hi @tomtung
You recently introduced the feature that unknown commands are preserved.
However, parameters in braces are removed from the brace environment or empty braces are stripped.
We discussed this at the JabRef dev call and came to the conclusion that it would be beneficial if the commands are just kept as they are if they are unknown.
I added a few tests that show the intended behavior.
Could you help us in bringing this into Scala code?
Or in general, do you agree with our thoughts?

Best regards,

Stefan and the JabRef team

  test("Unknown commands") {
    LaTeX2Unicode.convert("\\this \\is \\alpha test") shouldBe "\\this \\is α test"
    LaTeX2Unicode.convert("\\unknown command") shouldBe "\\unknown command"
    LaTeX2Unicode.convert("\\unknown{} empty params") shouldBe "\\unknown{} empty params"
    LaTeX2Unicode.convert("\\unknown{cmd}") shouldBe "\\unknown{cmd}"
  }

[tomtung]

Parsing-wise this is a little tricky, because generally braces are not kept in the output unless they are escaped (the same goes to consecutive spaces). Creating a special case here can be quite messy.

Problem is, even if we can add special cases to preserve braces, since the arity of an unknown command is, by definition, unknown, we still don't really know how many braces that follows should be kept. E.g. the output for \this{is}{a}{test} could be {\this}isatest or \this{is}atest or \this{is}{a}test or \this{is}{a}{test}, and there's no good way to know which is "correct".

From my understanding, you have problem with the output when an unknown commend is not followed by spaces. E.g. \test{abc} becomes \testabc, as if \testabc is a single command. What about simply adding braces around the unknown command itself? In that case, the behavior would be

  test("Unknown commands") {
    LaTeX2Unicode.convert("\\this \\is \\alpha test") shouldBe "{\\this} {\\is} α test"
    LaTeX2Unicode.convert("\\unknown command") shouldBe "{\\unknown} command"
    LaTeX2Unicode.convert("\\unknown{} empty params") shouldBe "{\\unknown} empty params"
    LaTeX2Unicode.convert("\\unknown{cmd}") shouldBe "{\\unknown}cmd"
  }

[koppor]

I think, we can go with the assumption \this{is}{a}{test} denotes a command with three parameters. Would that be possible? - All other interpretations feel strange and are IMHO not common in LaTeX usage.

[tomtung]

Not sure if I can agree.. For example, if \${\bf 1MM} should translate to $𝟏𝐌𝐌 (note the unicode boldface), why should \unknown_cmd{\bf 1MM} be translated to \unknown_cmd{𝟏𝐌𝐌} instead of {\unknown_cmd}𝟏𝐌𝐌?

Generally, I think it's really tricky to come up with well-defined rules that can guess the arity of an unknown command while sensibly covering all these edge cases. Even with such rules, context-dependent arity-guessing would add a lot of parsing complexity, likely with performance penalty, too.

[koppor]

I agree that it is very tricky. In our use case, we just need symbol replacement. Thus, all tex-commands not producing symbols directly can be left untranslated. This especially includes superscript, italics and bold face. Superscript is causing huge issues when converting it (see JabRef/jabref#3644 and JabRef/jabref#2596).

Is it possible to introduce a flag (second conversion method?) to replace symbols only? That would help us very much and leave the other features intact without any issues with unknown commands.

[tomtung]

I think that's a separate issue. From my understanding, the issue here is that LaTeX generally doesn't keep unescaped brackets {}, so it's very tricky to decide when to keep them for unkown commands without knowing the arity of the command.

Alternatively, maybe we can disregard the standard LaTeX behavior and have an option of always keeping the brackets in outputs?

[koppor]

+1 for the option to always keep the brackets in outputs!