Nested HTML inside block_html is escaped when escape=False, parse_block_html=True

[tdivis]

Normally with escape=False, nested HTML block is corectly not escaped:

>>> print markdown('<div id="special-part"><div class="subsection">text</div></div>', escape=False)
<div id="special-part"><div class="subsection">text</div></div>

But when I add parse_block_html=True, only out-most element is not escaped and the rest is escaped:

>>> print markdown('<div id="special-part"><div class="subsection">text</div></div>', escape=False, parse_block_html=True)
<div id="special-part">&lt;div class="subsection"&gt;text&lt;/div&gt;</div>

[lepture]

Actually, parse_html_block=True and escape=False is a wrong combination.

[tdivis]

Could you explain more about how is that a wrong combination?

I need exactly that, use markdown in HTML block, possibly nested, but it only works for not nested HTML blocks.

Another example (markdown inside table cell is interpreted correctly (<strong>), but nested tags are incorrectly escaped):

print markdown('<table class="special"><tr><td>**text**</td></tr></table>', escape=False, parse_block_html=True)
<table class="special">&lt;tr&gt;&lt;td&gt;<strong>text</strong>&lt;/td&gt;&lt;/tr&gt;</table>

[lepture]

@tdivis try the latest code.

[tdivis]

My examples work fine now, thanks :-). But I'm afraid, that this change broke some escaping elsewhere as these tests will fail:

ERROR: test_cases.test_normal('/home/glin/bin/mistune/tests/fixtures/normal', 'markdown_documentation_syntax')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/glin/bin/mistune/tests/test_cases.py", line 30, in render
    raise ValueError(msg)
ValueError:

ters.Ifyouwanttowriteabout'AT&T',youneedtowrite'<code>AT&amp
------Not Equal(4958)------
ters.Ifyouwanttowriteabout'AT&amp;T',youneedtowrite'<code>AT

======================================================================
ERROR: test_cases.test_normal('/home/glin/bin/mistune/tests/fixtures/normal', 'amps_and_angles_encoding')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/glin/bin/mistune/tests/test_cases.py", line 30, in render
    raise ValueError(msg)
ValueError:

<p>AT&Thasanampersandintheirname.</p
------Not Equal(6)------
<p>AT&amp;Thasanampersandintheirname

[teoguso]

@lepture, could you elaborate on why that combination is wrong? I believe this issue is related to a problem in Jupyter/nbconvert, where html tags inside markdown cells are not properly parsed.

Here's the issue: jupyter/nbconvert#328

[lepture]

@teoguso my mistake. It should be fixed on the master branch.

[teoguso]

@lepture thanks for the quick reply. I tested the latest master and the nbconvert issue is still there. My temporary solution is to downgrade to version 0.7.2. I'm sorry I can't be more helpful.

[lepture]

@teoguso could you add a minimized test case for your issue?

[teoguso]

@lepture Let me know if this helps: https://github.com/teoguso/nbconvert-mistune-test

[deroulers]

@teoguso I think your issue is due to commit 2a33458, which changed the regexp of mistune to parse HTML attributes. Namely, the new regexp in mistune 0.7.3 does not match anymore quote-less HTML attributes, whereas the old regexp did. Therefore, any HTML code with a quote-less (single or double quote) gets recognized as text instead of inline_html and gets (unappropriately) escaped.

For instance: <a href=foo> is treated by mistune 0.7.3 as text and < is escaped as <. But <a href="foo"> is correctly handled. In your minimized test case, the workaround is to put quotes around 700.

Workaround: systematically put quotes in HTML attributes, even if it is not mandatory according to the W3C standards.

@lepture You might want to change line 38 of mistune.py for, e.g.:
_valid_attr = r'''\s*[a-zA-Z\-](?:\=(?:"[^"]*"|'[^']*'|[^\s'">]+))*'''
This will still parse a subset of valid HTML, though.

[lepture]

@deroulers Thanks. Here is the fix: f7b5239

[teoguso]

@lepture @deroulers thanks! I confirm that the latest master now works without having to use quotes. Do we have to wait for 0.7.4 to get it on pip/conda by default? Cheers!

[danzimmerman]

I am having an ongoing issue, similar to @teoguso, with tags in jupyter/nbconvert even with mistune 0.74. If there are spaces around the equals sign for the HTML attribute, the tag is not parsed.

I assume this is related to the quotation mark issue.

[teoguso]

@danzimmerman I think standard HTML does not support spaces around the equal sign. This is not a bug I believe.

[mpacer]

@teoguso @lepture It actually does, even if it's a bad practice. So, right now it is not following the HTML5 spec:

The attribute name, followed by zero or more space characters, followed by a single U+003D EQUALS SIGN character, followed by zero or more space characters, followed by the attribute value, which, in addition to the requirements given above for attribute values, must not contain any literal space characters, any U+0022 QUOTATION MARK characters ("), U+0027 APOSTROPHE characters ('), U+003D EQUALS SIGN characters (=), U+003C LESS-THAN SIGN characters (<), U+003E GREATER-THAN SIGN characters (>), or U+0060 GRAVE ACCENT characters (`), and must not be the empty string.

source(emphasis mine)

The same restrictions & allowances (more or less) also apply to single quoted and double quoted values.

NB: There is this SO post on why it might be bad practice.

[teoguso]

@mpacer I see. Thanks, that's good to know! I guess the HTML landscape can be a bit messy...

[lepture]

Fixed in 0.8.4