Allow Markdoc to render HTML tags

[alex-sherwin]

Body

Summary

By default Markdoc does not render HTML in markup, instead, it's treated as raw strings that end up being escaped and rendered.

It would be very useful to have an opt-in ability to enable HTML in markup.

After all, Markdoc purports to be a superset of Markdown, which does support HTML in markup.

So converting a large corpus of existing Markdown which may contain HTML markup, and/or converting a corpus of markup from another CMS-like system which is HTML based into Markdoc is problematic if one cannot render that HTML markup.

Background & Motivation

By default, Markdoc tokenizes markup using markdown-it with HTML processing disabled.

The Markdoc GitHub repo had a previous discussion in a long-running issue and PR about baking HTML support directly into the markdoc library, however, the end result of this was the development of a solution that provides the desired result (full HTML markup integration) that did not require any changes to the markdoc library (see the issue and PR solution example) and is viewed as the ideal way to deal with this problem.

The solution is to apply a pure functional transform on either the tokens or AST (token time is easier) which parses HTML in the Markdoc markup and replaces the raw HTML string tokens with Markdoc Tag tokens that describe the HTML element that was parsed, while interleaving the HTML tree with any other Markdoc tags/nodes that are interleaved in the markup.

Goals

• Enable HTML inside Markdoc markup
• Make this behavior opt-in with a simple allowHTML option on the @astrojs/markdoc integration config
• Expose the pure token transform function in the dist so that for advanced use cases (outside of content collections) that the token transform function can be utilized directly.
• Expose the Markdoc html tag in the dist so that for advanced use cases (outside of content collections) that the html tag can be utilized directly.

Example

The pure functional token transform process is basically:

• enabling HTML parsing in markdown-it Tokenizer that Markdoc uses (a boolean switch), causing the produced Tokens to know when a string of content is HTML
• use a library like htmlparser2 (which was found to be an excellent fit since it's a lazy incremental parser; it is event based and can be fed chunks of HTML ad-hoc as they are encountered) and incrementally process HTML tokens as they are encountered. Each HTML open/close token encountered will generate a new "html" Markdoc Tag token which contains all the necessary metadata to render the parsed HTML element.
• preserve any other Tokens (thus interleaving other Markdoc markup such as nodes and tags in the proper tree/graph alongside the parsed & generated HTML Tags)

A complete working example can be found in this repo: https://github.com/alex-sherwin/astro-markdoc-html/tree/main complete with unit test coverage

Overall, this solution requires:

• A pure token transform function (bulk of complexity lies here)
• A "html" Markdoc Tag which renders the newly generated HTML Tags as native HTML elements (the output of the previous transform function)
• Some ancillary functions to aide with a few edge cases, such as
  • Inline style attributes must be parsed into react-like CSS Objects
  • A few special-case HTML attributes need to ensure they are camelCased for react-like compliance (like colSpan/rowSpan)
  • Since we need to tokenize markup found inside HTML elements, there are some edge cases that produce unwanted extra paragraph tokens around content, so these need to be un-wound

Concerns

There are potential security concerns with enabling this, since you're taking HTML markup and rendering it.

However, in theory, this shouldn't be materially different then rendering the HTML markup in Markdown/MDX.

I think the test cases should ensure that a few things are not possible to render via this feature, such as <script> tags or inline JavaScript in HTML attribute event handlers.

[bholmesdev]

Thanks for this Alex! This feels like a welcome addition to make Markdown -> Markdoc migration smoother. Bikeshed: I'm curious if allowHtml makes more sense as a flag name over enableHtml. Not a strong preference though!

[alex-sherwin]

Regarding the concerns around JavaScript/XSS,

Maybe I'm just over-thinking it, because Astro currently doesn't do anything to prevent this in the existing Markdown/MDX rendering (you can happily throw in a <script> tag in a Markdown document and it'll get rendered as a real <script> tag and execute)

So if XSS needed to be solved, I would think that should probably be a separate roadmap discussion which would span Markdown, Markdoc and MDX equally.

A proper XSS solution, if ever implemented, is not trivial (see OWASP recommendations) and given how Astro's rendering pipeline works in conjunction with Markdoc, there's not a clear way to deal with this using the OWASP recommended pattern (which is to apply a sanitization function to a completely rendered chunk of HTML via the DOMPurify library)

[bholmesdev]

This is a good callout. Not as much of a problem for content collections since they can't be fetched from an external source. Though if we open the Renderer component to render external Markdoc, it may matter more. I believe we could recommend Markdoc's html render with set:html here if we offer a more robust solution? Agreed this is OOS.

[phun-ky]

Tbh it's up to the user to sanitize. However, is this near implementation or do we have a workaround to allow inline html?

[bholmesdev]

@phun-ky This feature should be merged in the latest Markdoc release! Check the allowHTML config option.

[phun-ky]

Yeah, yesterday before you commented, I found this solution in the discussions/issues somewhere, and I am using @markdoc/markdoc, not @astrojs/markdoc:

function processTokens(tokens) {
  const output: any[] = [];
  const parser = new Parser({
    onopentag(name, attrs) {
      output.push({
        type: 'tag_open',
        nesting: 1,
        meta: {
          tag: 'html-tag',
          attributes: [
            { type: 'attribute', name: 'name', value: name },
            { type: 'attribute', name: 'attrs', value: attrs }
          ]
        }
      });
    },

    ontext(content) {
      if (typeof content === 'string' && content.trim().length > 0)
        output.push({ type: 'text', content });
    },

    onclosetag(name) {
      output.push({
        type: 'tag_close',
        nesting: -1,
        meta: { tag: 'html-tag' }
      });
    }
  });

  for (const token of tokens) {
    if (token.type.startsWith('html')) {
      parser.write(token.content);
      continue;
    }

    if (token.type === 'inline') token.children = processTokens(token.children);

    output.push(token);
  }

  return output;
}
// This creates a mapping between route and parsed Markdoc content.
export const createContentManifest = (ROOT_DIR) => {
  const files = glob.sync(`${ROOT_DIR}/**/*.md`);
  const manifest = {};

  files.forEach((file) => {
    const rawText = fs.readFileSync(file, 'utf-8');
    const tokenizer = new Markdoc.Tokenizer({ html: true });
    const tokens = tokenizer.tokenize(rawText);
    const processed = processTokens(tokens);
    const ast = Markdoc.parse(processed);
    const frontmatter = parseMarkdocFrontmatter(ast);

    manifest[frontmatter.route] = {
      ast,
      frontmatter
    };
  });

  return manifest;
};
const contentManifest = createContentManifest(CONTENT_DIR);
const path = req.path.replace('/docs/', '/');
const document = contentManifest[path];
const { ast } = document;
const config= {
  tags: {
    message,
    'html-tag': {
      attributes: {
        name: { type: String, required: true },
        attrs: { type: Object }
      },
      transform(node, config) {
        const { name, attrs } = node.attributes;
        const children = node.transformChildren(config);

        return new Markdoc.Tag(name, attrs, children);
      }
    }
  },
  
};
const content = Markdoc.transform(ast, config);
const rendered = Markdoc.renderers.html(content) || '';

[bholmesdev]

Closing since this shipped in the latest release of Markdoc! See the allowHTML docs 🥳