libxml 2.10+ compatibility for <dot.> and <foo:colon> angle-bracket syntax

[voxik]

It seems that since libxml 2.10.3 has landed in Fedora, we observe the following test error ronn-ng test suite:

+ ruby -Ilib:test -e 'Dir.glob "./test/test_*.rb", &method(:require)'
Loaded suite -e
Started
F
===============================================================================
Failure: test_angle_bracket_syntax_HTML(RonnTest):
  --- /builddir/build/BUILD/ronn-ng-0.9.1/usr/share/gems/gems/ronn-ng-0.9.1/test/angle_bracket_syntax.html	2023-07-27 04:43:35.769599012 +0000
  +++ /builddir/build/BUILD/ronn-ng-0.9.1/usr/share/gems/gems/ronn-ng-0.9.1/test/angle_bracket_syntax.html.wrong	2023-07-27 04:43:38.400825815 +0000
  @@ -13,5 +13,5 @@
   
   <p>or when <code>&lt;WORD&gt;</code> is enclosed in backticks.</p>
   
  -<p>or when <var>WORD</var> has a <dot.> or <colon>.</colon></dot.></p>
  +<p>or when <var>WORD</var> has a <dot.> or <foo:colon>.</foo:colon></dot.></p>
   </div>
  .
/builddir/build/BUILD/ronn-ng-0.9.1/usr/share/gems/gems/ronn-ng-0.9.1/test/test_ronn.rb:43:in `flunk_with_diff'
/builddir/build/BUILD/ronn-ng-0.9.1/usr/share/gems/gems/ronn-ng-0.9.1/test/test_ronn.rb:96:in `block (2 levels) in <class:RonnTest>'
===============================================================================
Finished in 7.329725535 seconds.
-------------------------------------------------------------------------------
64 tests, 113 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
98.4375% passed
-------------------------------------------------------------------------------
8.73 tests/s, 15.42 assertions/s

Please note that this is not issue with pre-built nokogiri installed from rubygems.org, because that likely bundles older libxml2

[apjanke]

Ah, namespaces, great.

Wouldn't be surprised if this is just the new libxml version being more correct about namespace handling; some of our test code throws around "foo"s.

I'll have a look at it.

[yselkowitz]

Any progress on this?

[yselkowitz]

Ping? The Fedora 40 mass rebuild is expected in January, and without a fix that will be the second consecutive mass rebuild with an unsuccessful build for rubygem-ronn-ng, which puts its continued presence in Fedora at risk, as well as the various packages which depend on it to build.

[apjanke]

Oof, sorry. It has been a real crunch at work the last few months.

Thanks for the heads-up here. I've blocked out some time on my calendar to get to it this weekend. The holiday should give me some breathing room to make progress here.

[apjanke]

Okay, made some progress. I can now reliably reproduce this error under both Ruby 2.7.4 and 3.1.4, the versions in the latest Debian, and got my dev env working regularly. The trick was giving up on the system ruby and using rbenv and user-installed Rubies. And it's also reproducing on a Mac with a Homebrew-installed libxml2.12.

Think I'll be able to get this sorted out soon now.

[apjanke]

Okay, I think I've got a partial diagnosis here.

I'm not sure this is actually due to libxml 2.10 exclusively, or per se. I think it may be a larger issue, also involving the switch from rdiscount to Kramdown as the Markdown processing library, or changes between versions in Kramdown, or the use of GitHub Flavored Markdown.

I think:

The newly-failing test due to that output change where the "foo:" disappears is not the only problem. The current expected output in that test, where it expects has a <dot.> or <colon>.</colon></dot.>, was wrong in the first place. It should have matching open and close tags for dot. and colon; they shouldn't be HTML elements at all. This is for the special Ronn syntax extension for angle-bracket variable names, like <blah> with no matching </blah> turning in to an underlined "blah" in the output. In the ronn-format.7 man page, it's described like this:

  * `<anglequotes>` (non-compatible markdown extension):
    User-specified arguments, variables, or user input. Typically displayed with
    <u>underline</u> in roff output. HTML output: `<var>`.

That means that <blah> shouldn't turn in to a pair of <blah>...</blah> tags at all; it should turn in to <var>blah</var>.

That ronn-format.7 description doesn't say anything about special behavior for . and : characters; that's described in the angle_bracket_syntax.ronn content text.

A <WORD> in angle brackets is converted to <var>WORD</var>,

    except when <WORD> is
    part of a preformatted
    code block,

or when `<WORD>` is enclosed in backticks.

or when <WORD> has a <dot.> or <foo:colon>.

It doesn't say why dots and colons are an exception here. But I'll bet that's for compatibility with Markdown's special <url> syntax, where something in angle brackets that looks like a URL is treated as a URL and turned in to a link, instead of being interpreted as inline HTML (an HTML tag). A <word> with a colon or dot in it looks like a URL, so it's left as is to be handled as such. But that's a simple and over-broad test, so I'm guessing things like <dot.> and <foo:colon> are treated as URL-like exceptions by Ronn but aren't recognized as URLs by markdown/Kramdown, so they're left as HTML tags and processed as such, and then later the foo: is being interpreted as a namespace and for some reason stripped, and I'm guessing the corresponding closing tags are added due to HTML's loose closing tag rules.

I suspect that the "namespace" stripping happens either during the call to Nokogiri::HTML.parse(wrapped_html), or Nokogiri::XML::Node.to_s() inside Ronn::Document.to_html_fragment(), because (I think) this is using the "HTML serialization" of the content and the HTML serialization does not support explicit namespace qualification of tags and : or . are not valid for use in HTML tag names.

This stuff is done by the markdown_filter_angle_quotes and html_filter_angle_quotes methods in Ronn::Document.

---

Next steps: Figuring out exactly where the namespace stripping is happening, and try to revert it.

I'm thinking maybe the foo: namespace qualification could be restored by having the Nokogiri node do XHTML/XML serialization instead of HTML serialization, maybe just by calling its to_xhtml() or to_xml() method instead of to_s().

If I can't get that to work, I'm inclined to just mark the new output as the expected correct output, since a <word> that has a : or . but is not actually a URL seems like an unusual case that won't happen often in legit/common ronn input?

For the longer term, should probably define more thoroughly and formally what the behavior is for these :/. but non-URL <word> tags, so we know what the "correct" behavior we're trying to maintain here is.

[apjanke]

Oh. My. Gosh. I think I've been reading the test error message backwards this whole time.

  -<p>or when <var>WORD</var> has a <dot.> or <colon>.</colon></dot.></p>
  +<p>or when <var>WORD</var> has a <dot.> or <foo:colon>.</foo:colon></dot.></p>

That means the test's expected output definition currently has <colon> without the foo: prefix, and the code is now producing <foo:colon> instead.

I think the new output that includes the foo: is actually more correct, with respect to HTML/XML processing and the description of the special Ronn angle bracket syntax in ronn-format.7 and test/angle_bracket_syntax.html. Which means the fix here is to just mark the new ouptut as correct in the test fixture.

[apjanke]

Yeah, I think the right fix is to assume we'll be running under the newer libxml 2.10+ and recent kramdown/Nokogiri, mark the current output as correct in the test, and ship it. And then look deeper in to what the ". or : in tag contents" behavior should be for a future release.

I've checked in the fix on main. Tests are passing now for me, including on Fedora 39. Also tested on macOS 14 and Debian 11 bullseye and 12 bookworm. All under Ruby 2.7.

janke@fed39-test-as:~/repos/ronn-ng$ date
Wed Jan  3 08:36:07 PM EST 2024
janke@fed39-test-as:~/repos/ronn-ng$ cat /etc/fedora-release 
Fedora release 39 (Thirty Nine)
janke@fed39-test-as:~/repos/ronn-ng$ git log --oneline | head -3
989452e doc: changelog: format issues as links instead of URLs
c4459d4 tests: fix angle-bracket syntax test: expect namespace-like "foo:" prefixes to be preserved (#102)
afaf6b6 doc: fix dead links in README and use https
janke@fed39-test-as:~/repos/ronn-ng$ ruby --version
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [aarch64-linux]
janke@fed39-test-as:~/repos/ronn-ng$ bundle exec rake test
Loaded suite /home/janke/.rbenv/versions/2.7.6/bin/rake
Started
/warn: unrecognized inline tag: ["dot."]
/warn: unrecognized inline tag: ["ws-name"]
warn: unrecognized inline tag: ["ws-name"]
Finished in 5.662143699 seconds.
------------------------------------------------------------------------------------------------------------------------------------------
70 tests, 112 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
------------------------------------------------------------------------------------------------------------------------------------------
12.36 tests/s, 19.78 assertions/s
janke@fed39-test-as:~/repos/ronn-ng$ 

Dunno if I should bump to using Ruby 3.x yet; I'm not sure exactly how ruby dependencies in Homebrew and Fedora work. I'll have a look at their package definitions.

If you want to test this now from the repo, it's on main. I think I'll have an 0.10.1 prerelease gem published within a couple days.

[apjanke]

Might have found a more-precise cause: yeah, it's nokogiri and libxml2, and maybe I'm seeing somewhat different behavior in my testing than you're seeing in the built Fedora one because I'm using the pre-built nokogiri and its vendored libxml2 library even when I'm testing ronn-ng on Fedora.

Nokogiri ships ("vendors") its own libxml2 like you say. In my testing, this behavior change happened with nokogiri 1.14.3: earlier versions <= 1.14.2 have the old behavior that strips the "foo:" prefix, and 1.14.3+ preserve the "foo:" prefix. In my testing, this doesn't seme to depend what system libxml2 is installed on my box or brew environment; simply changing the nokogiri version in the ronn-ng.gemspec would change the behavior.

The nokogiri 1.14.3 release notes say that version 1.14.3 upgraded its vendored libxml2 from 2.10.3 to 2.10.4. (For security reasons, it says.) And hey, look: the libxml 2.10.4 release notes talk about a behavior change related to namespaces in HTML elements, which looks like what we saw here.

Though if it was 2.10.3 hitting Fedora that caused the change there, why is the nokogiri 1.14.2 gem that uses 2.10.3 showing the old behavior in my test environment?

Regressions

• SAX2: Ignore namespaces in HTML documents

And that seems consistent with what you say here:

Please note that this is not issue with pre-built nokogiri installed from rubygems.org, because that likely bundles older libxml2

That nokogiri 1.14.3 transition is maybe when the RubyGems version of nokogiri picked up the libxml2 transition that caused the Fedora build problem, and that's when it started showing up in my own testing, because I just test using gems and rbenv and don't go through the whole RPM package build process and take a dependency on the nokogiri RPM.

I wonder if there's an easy way for me to set up the Ronn-NG repo so I could run it from the source but using the nokogiri and other dependencies from the system-installed RPM packages instead of gems I pick up in my dev Ruby environments using bunder and gem?

---

Anyway, after this further reading and experimentation, I'm pretty confident that the new behavior of retaining the "foo:" is correct-er (as far as Ronn-NG is concerned), and updating the test fixture and gemspec dependency versions is the right fix. I merged the previous test fix to main, and altered the gemspec dep for nokogiri in b4deb42 (#108).

I think we're about ready to cut an 0.10.1 release that will build the Fedora RPM successfully.

[apjanke]

I've published a Ronn-NG 0.10.1.pre4 prerelease to GitHub and RubyGems that I think will fix this and get the Fedora RPM build working again (but am not sure, bc I haven't tested RPM building itself).

• https://github.com/apjanke/ronn-ng/releases/tag/v0.10.1.pre4
• https://rubygems.org/gems/ronn-ng/versions/0.10.1.pre4

[voxik]

I think the new output that includes the foo: is actually more correct, with respect to HTML/XML processing and the description of the special Ronn angle bracket syntax in ronn-format.7 and test/angle_bracket_syntax.html. Which means the fix here is to just mark the new ouptut as correct in the test fixture.

This sounds plausible to me. Please note that on Fedora, we run CI called Koschei, which can catch issues like this caused by library updates, so I am pretty sure this started to happen with update libxml2.

I'm using the pre-built nokogiri and its vendored libxml2 library even when I'm testing ronn-ng on Fedora.

On Fedora, you can either use the system version of Nokogiri:

$ sudo dnf install rubygem-nokogiri

With Bundler, you would also need to use something like bundle install --local, otherwise Bundler could have tried to download the dependencies instead of using the already installed.

Or you should be able to use Nokogiri installed from RubyGems like this:

$ sudo dnf install libxml2-devel libxslt-devel ruby-devel

$ NOKOGIRI_USE_SYSTEM_LIBRARIES=yes gem install nokogiri --platorm=ruby

Basically it is needed to install all the build dependencies and later 1) tell nokogiri configure script to use them 2) use the --platform=ruby to actually install the gem which does not ship the prebuilt binary extension.

[voxik]

Here is the PR fixing the Fedora RPM. It links to Fedora CI - scratch build, where you could eventually get to the build.log. There can be seen that 1) the patch was applied 2) the test is passing.

Please note there is applied custom patch instead yours, but we will use either the official patch or wait for official stable release. We don't do pre-releases often, because sometimes happens that there is some pre-release of new major version, which never materializes into official version. This likely won't be the case, but the patch is easier anyway :)

Also, thx a lot for looking into this!

BTW the bad news is that there is now also different fork of Ronn, which was recently started to be used by Bundler :(

rubygems/rubygems#7227

[voxik]

BTW the bad news is that there is now also different fork of Ronn, which was recently started to be used by Bundler :(

rubygems/rubygems#7227

The good news is that they are likely to suffer this issue, so ronn-ng is ahead with the fix ;)

[voxik]

Now I see the #87, so you are aware, sorry 🙈

[apjanke]

Please note that on Fedora, we run CI called Koschei, which can catch issues like this caused by library updates, so I am pretty sure this started to happen with update libxml2.

Sounds likely to me. I barely know what I'm doing here with the basic Ruby and Gems stuff, and have basically no experience with RPM packaging or how Fedora ships nokogiri.

BTW, thank you for a good bug report here. It was specific, readable, and had references, and I think you got the cause right in the first post. It just took me a while to get up to speed on this, because I didn't really know enough about RPMs, Fedora releases, and the nokogiri/libxml2 interaction to really understand what you were talking about until I spent some time looking around the code.

something like bundle install --local [...] or NOKOGIRI_USE_SYSTEM_LIBRARIES=yes gem install nokogiri --platorm=ruby [...]

Ah, that might be what I need here. I'll look in to it.

Please note there is applied custom patch instead yours, but we will use either the official patch or wait for official stable release.

Glad to see we agree on the approach here. (I think.) I think I'll have the time over the next couple days to pull this in to the official upstream release, and we can save the work/divergence of a patch at the RPM level.

Also, thx a lot for looking into this!

You're welcome! Sorry it's taken so long. I'm working at a startup for my day job and it got really busy this year, and I caught Covid a few months ago and it really sucked and gobbled up my free time.

BTW the bad news is that there is now also different fork of Ronn [...] [(nronn)]

A new challenger has appeared! :)

That fork only has a few changes in it, and I've already pulled them in to Ronn-NG here. (#87) Once this 0.10.1 release goes out, I'll be at feature parity and maybe there'll be no need for another fork.

[apjanke]

The fix here is merged to main, and I've tested it as well as I can at the moment. Setting this issue to Closed. Will ping you when it rolls out in a release.

[apjanke]

Aha! Think I figured out how to test properly on Fedora in its RPM-supplied environment: skip the rbenv and bundler install stuff, install the RPMs for the gem dependencies like nokogiri but not the RPM for ronn-ng itself, and test the repo code there with plain rake test instead of bundle exec rake test.

Tested that on Fedora 39 just now, and it passes:

janke@fed39-test-as:~/repos/ronn-ng$ date
Sat Jan  6 06:40:54 PM EST 2024
janke@fed39-test-as:~/repos/ronn-ng$ cat /etc/fedora-release 
Fedora release 39 (Thirty Nine)
janke@fed39-test-as:~/repos/ronn-ng$ git log --oneline | head -2
80a3d67 [rel] v0.10.1.pre6 (prerelease)
dd2ceeb [tests] Testing on Fedora: add notes and dep-install script
janke@fed39-test-as:~/repos/ronn-ng$ ./bin/ronn --version
ronn v0.10.1 (Ronn-NG)
janke@fed39-test-as:~/repos/ronn-ng$ rake test
Loaded suite /usr/bin/rake
Started
/warn: unrecognized inline tag: ["dot."]
/warn: unrecognized inline tag: ["ws-name"]
warn: unrecognized inline tag: ["ws-name"]
Finished in 3.42062891 seconds.
-------------------------------------------------------------------------------------------
70 tests, 112 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
-------------------------------------------------------------------------------------------
20.46 tests/s, 32.74 assertions/s
janke@fed39-test-as:~/repos/ronn-ng$ 

I've published this as 0.10.1.pre6 to both GitHub and RubyGems. Think this is the release candidate.

https://github.com/apjanke/ronn-ng/releases/tag/v0.10.1.pre6

Gonna sleep on this overnight and if no other problems occur to me, publish it as the real 0.10.1 release. (Don't think there's much to be gained by waiting further, since I don't have any users doing testing from the repo main or the prereleases.)

[voxik]

Aha! Think I figured out how to test properly on Fedora in its RPM-supplied environment: skip the rbenv and bundler install stuff, install the RPMs for the gem dependencies like nokogiri but not the RPM for ronn-ng itself, and test the repo code there with plain rake test

Yes, that is how we make the package. You could also use Bundler, but it is necessary to have the dependencies installed and run bundle install --local (at least that is the case for clean environment). But that does not provide any benefit, if there are already the only required dependencies.

There is also possibility to use https://packit.dev/, but I have not real experience how to setup it and I don't think that upstreams need to bother with PRMs.

[voxik]

There is also possibility to use https://packit.dev/, but I have not real experience how to setup it and I don't think that upstreams need to bother with PRMs.

And this is not RPM issue. We are just sometimes ahead of the game and Nokogiri will eventually move to more recent libxml

[apjanke]

and run bundle install --local [...]

Could that get Nokogiri built with its non-default options which have it use the system-supplied libxml2 instead of its vendored libxml2? I couldn't see a way to do that, since bundle uses the Gemfile/gemspec, and those declare their dependencies in terms of just gem name and version; I didn't see way to specify build options for the dependency gems. But again, I'm no Ruby expert.

That packit thing looks interesting, but I'm going to pass on it, because I'm not confident that I'd be able to get it to correctly reproduce the Fedora-supplied RPMs for ronn's dependencies, and the only thing I care about here is making sure ronn-ng works against the official RPM-supplied r"ubygem-*" RPMs. I can just build VMs for each of the Fedora releases to get those for use as test beds.

I added some notes about testing Ronn-NG on Fedora here in dd2ceeb, and will expand on that as I get better with it.

I'm feeling good about this Ronn-NG 0.10.1 build, and will go ahead and publish the real release for it today.

[voxik]

and run bundle install --local [...]

Could that get Nokogiri built with its non-default options which have it use the system-supplied libxml2 instead of its vendored libxml2? I couldn't see a way to do that, since bundle uses the Gemfile/gemspec, and those declare their dependencies in terms of just gem name and version; I didn't see way to specify build options for the dependency gems. But again, I'm no Ruby expert.

Actually, it might depend on a presence of Gemfile.lock. I suspect there might be stored architecture, which might make the difference. If you drop it, then it gets what it has available.

That packit thing looks interesting, but I'm going to pass on it

Yeah, right. No worries

I'm feeling good about this Ronn-NG 0.10.1 build, and will go ahead and publish the real release for it today.

Nice 🥳

[apjanke]

Ronn-NG 0.10.1 is published!

• GitHub: https://github.com/apjanke/ronn-ng/releases/tag/v0.10.1
• RubyGems: https://rubygems.org/gems/ronn-ng/versions/0.10.1

I think this will work for y'all with current Fedora releases. Please let me know if there's any problems!

[apjanke]

Hi, @voxik, just circling back on this to see if it worked out. Is ronn-ng v0.10.1 working out okay in the new Fedora builds?

On the Fedora rubygem-ronn-ng packaging page (https://packages.fedoraproject.org/pkgs/rubygem-ronn-ng/rubygem-ronn-ng/), I see that Rawhide now has an "0.10.1-2.fc40" package built under "Stable", which looks like a good sign?

[voxik]

Hi, @voxik, just circling back on this to see if it worked out. Is ronn-ng v0.10.1 working out okay in the new Fedora builds?

Thank you for asking. Yeah, the 0.10.1 have been just in Rawhide (the development version of Fedora) so fart, therefore it have not reached masses yet. But as far as I can say, the package itself builds fine and I am not aware it would cause troubles elsewhere in packages using Ronn-NG to generate their manpages. I am quite confident there won't be any further issues. Appreciate your support 👍

[apjanke]

Great! Swing on back if anything does crop up.

[boutil]

Hi!
I am one of the maintainers for the ronn-ng package in Debian. The tests were passing when I uploaded version 0.10.1, but Debian reverted in September the version of libxml2 to 2.9.x, because of ABI breakage. Therefore, this angle_bracket_syntax test is not passsing anymore on Debian.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1084367

I am therefore reverting the change d3b89fb
until Debian upgrades again to a newer version. I was wondering it is was possible to make the tests accomodate for the two possible outputs.

[apjanke]

Uh oh!

Yes, I think I can do something about this. Probably best to add something to ronn-ng for compatibility with both versions of libxml2, either switch to some code form that works in both versions, or do a run time test for which libxml2 version we're running against and handle it gracefully.

wondering it is was possible to make the tests accomodate for the two possible outputs

Yes, that sounds like the right thing to do here.

I'll make this a priority if I find some free time this weekend.

Here's a separate ticket for this specific problem: #123