MDX compiledContent() support

[bholmesdev]

We've received multiple user requests for an MDX -> HTML string output (discord discussion, discord discussion, RSS PR thread). This is primarily for RSS feed support, where you often need the fully rendered content of a post to embed in your XML. MDX's use of a <Content /> component makes this a no-go.

This has led to some creative solutions in the community. Scott Willsey's is especially notable, as it uses Astro's slots.render as a hack to access the compiled HTML. This also proves it is possible to stringify MDX documents with some workarounds.

Background

MDX is built on Astro's own JSX runtime. This means we rely on our own transformers to process MDX into <JSXExpressions />. This unlocks Astro-like features in MDX, including the client: directive, the set:html directive, and support for any framework (React, Vue, Svelte, etc).

This also makes Astro's MDX integration incompatible with other transformers, including React's renderToString helper to stringify a JSX tree. In order to support this, Astro needs to build its own renderToString transform that can convert MDX output to an HTML string. The MDX compiler also offers an HTML string output, though it requires the unsafe use of eval, making it a no-go for Astro internally.

Potential solutions

Update A renderToString() conversation is now being tracked at #462! 🥳

1. Introduce a stringify utility to Astro's JSX runtime. There is ongoing work to expose an Astro-to-HTML utility that may be used for unit testing Astro components. Once this is complete, we can test the performance implications to see if a compiledContent() utility can be used.

2. Allow .xml output from .astro files. Scott Willsey's approach demonstrated that .astro files can be used for templating XML, as long as the user uses set:html to parse XML syntax as-needed. If Astro allows users to output .xml, we bypass the need for an RSS endpoint entirely by using the existing <Content /> component to generate XML. This solution comes with some major drawbacks:

• The Astro language tools are not designed for this use case.
• Templating via set:html may be intimidating to new users.
• This only solves the compiledContent() issue for RSS users. If others see a need for compiledContent() outside of RSS (ex. to output MDX from a JSON API), the issue still remains.

[jasikpark]

another impl of RSS is https://www.jsonfeed.org/, which would be nice to support as well

[scottaw66]

Thanks for carrying this conversation forward and for mentioning my method. I do want to make it super clear it was Chris Adiante who came up with the method and it wouldn’t have occurred to me it was possible that way. It only works for static sites, but it’s his creative and clever thinking that made it happen. Just wanted to make sure he gets due credit. 😊

[bholmesdev]

Related discussion: #409

[coding-in-public]

I would love to see support for rendered HTML content from MDX files for full content in RSS! I'm loving the experimental collections API and thought I might be able to render out the content with the render() method available on the returned collection data, but found this RFC after hacking around for a while without success. I wish I had some experience here to help contribute, but for now, just wanted to +1 the idea. Thanks, Ben!

[joaomelo]

This disccussion is super positive. I've also seen methods to generate the rss feed using the content experimental feature, but for me is important that astro enables a simple approach that not require collections just to generate the feed.

It would be great if we could use MDX files in the same way MD files are already supported in the rss example provided on the official guide (https://docs.astro.build/en/guides/rss/).

[nzakas]

I was also using compiledContent() to generate my RSS and JSON feeds in Astro v1.x. Now I'm unable to upgrade to Astro v2.x because there is no longer an equivalent of compiledContent().

What needs to happen to move this forward? Happy to jump in with some guidance.

[bholmesdev]

Join the discussion at #462 ! This is the missing piece to rendering MDX as HTML.

[nzakas]

Thanks, I'll check it out!

[mattly]

for those who don't want to join Discord just to get the info on Scott Willsey's workaround with slots.render, it is here: https://scottwillsey.com/rss-pt2/

[scottaw66]

Thanks for the link... I wonder though if this now works in MDX without my workaround required if you use content collections instead of glob imports? If so, I really should update that post encouraging people to stay away from that hacky workaround. 😄

Add an RSS feed - Including full post content

[mattly]

I wasn't able to figure out another way – content collections returns a CollectionEntry<collection>, and it has the unprocessed body, and an async render() method that produces { Content } which is meant to be rendered in an astro template, same as a page.

I hadn't read through and realized the extent of hackiness in your method before posting the link, but Discord is not suited towards being a source of truth either through searchability or inclusiveness (they capriciously permaban people who mod their webpage client) and wanted to provide that information here

[nemanjam]

Is there a solution for this?

[bholmesdev]

Yes, the container API is finally our official solution to this problem! We're working to make use cases like RSS more streamlined, but the primitives are there to hack out MDX rendering

[brian-montgomery]

Using the example code in Including full post content, you would get something similar to this:

import rss from '@astrojs/rss';

import { getCollection } from 'astro:content';
import sanitizeHtml from 'sanitize-html';

// Container API
import { experimental_AstroContainer as AstroContainer } from "astro/container";
import { loadRenderers } from "astro/container";
import { getContainerRenderer as mdxRenderer } from "@astrojs/mdx";

const renderers = await loadRenderers([mdxRenderer()]);
const container = await AstroContainer.create({renderers});

export async function GET(context) {
  const blog = await getCollection('blog');
  return rss({
    title: 'Buzz’s Blog',
    description: 'A humble Astronaut’s guide to the stars',
    site: context.site,
    items: await Promise.all(blog.map(async (post) => {
      const { Content } = await post.render();
      const postHtml = await container.renderToString(Content);
      // If using custom components, do this instead...
      // const postHtml = await container.renderToString(Content, { props: { components: { h1: Heading }}});
      
      return {
        link: `/blog/${post.slug}/`,
        content: sanitizeHtml(postHtml, {
          allowedTags: sanitizeHtml.defaults.allowedTags.concat(['img'])
        }),
        ...post.data,
      };
    })),
  });
}

However, import { loadRenderers } from "astro/container"; doesn't work unless vitest is loaded. Granted, it's not a complicated function, so it's not hard to copy/paste into a separate file and import it that way.

[ematipico]

We added new APIs for your use case:

import reactRenderer from "@astrojs/react/server.js";
import vueRenderer from "@astrojs/vue/server.js";

const container = await AstroContainer.create();
container.addServerRenderer({renderer: reactRenderer});
container.addServerRenderer({renderer: vueRenderer});

I opened a PR to update the docs

[eikowagenknecht]

I managed to get it work with a slight variation of that code. But it only works as long as there are no client:visible directives in the components used by my MDX files. This is the code I use:

import { getCollection } from "astro:content";
import { SITE } from "@/config";
import rss from "@astrojs/rss";
import sanitizeHtml from "sanitize-html";

// Container API
import { experimental_AstroContainer as AstroContainer } from "astro/container";
import reactRenderer from "@astrojs/react/server.js";
import mdxRenderer from "astro/jsx/server.js";

const container = await AstroContainer.create();
container.addServerRenderer({ renderer: mdxRenderer });
container.addServerRenderer({ renderer: reactRenderer });

export async function GET(context) {
  // console.log("rendering");
  const allPosts = await getCollection("posts");
  const sortedPosts = Object.values(allPosts).sort(
    (a, b) =>
      new Date(b.data.published).valueOf() -
      new Date(a.data.published).valueOf(),
  );
  return rss({
    title: SITE.title,
    description: SITE.description,
    site: context.site,
    items: await Promise.all(
      sortedPosts.map(async (post) => {
        const { Content } = await post.render();
        const postHtml = await container.renderToString(Content);
        const sanitizedHtml = sanitizeHtml(postHtml, {
          allowedTags: sanitizeHtml.defaults.allowedTags.concat(["img"]),
        });

        return {
          title: post.data.title,
          description: post.data.description,
          pubDate: post.data.published,
          link: `/posts/${post.slug}/`,
          content: sanitizedHtml,
        };
      }),
    ),
    customData: "<language>en-us</language>",
  });
}

I get the following error:

15:46:09 [ERROR] [Error] Error: invalid hydration directive "client:visible". Supported hydration methods: 
  Hint:
    This issue often occurs when your MDX component encounters runtime errors.
  Stack trace:
    at C:\Entwicklung\repos\eiko-wagenknecht.de\node_modules\astro\dist\runtime\server\hydration.js:51:19
    [...] See full stack trace in the browser, or rerun with --verbose.

Unfortunately it stops there and never states which hydration methods are supported. Is there any way to get it working with MDX files that contain client:visible directives (obviously without the interactive part)?

[bholmesdev]

I have also seen that hydration directive error. @ematipico has this been fixed in the past. Or is it being tracked?

[filipesmedeiros]

I have a tangential request: I have relatively complicated pages, rendered based on collections. They're not really complicated, they just have custom MDX components, some slots, etc.

I don't think this method being discussed here is valid for my use case. Hopefully I'm not the only one ahah

So my question is: could there be a way to "force" the RSS render to be the last thing to happen, and somehow have access to the final rendered HTML of the page? This would avoid repetition between stuff we do on the slugged pages and the RSS feed (right now we have to have the same rendering logic twice).

We could then parse and transform the HTML on a case by case (for example, I would want to remove everything except the <article> element and its children).

What do you think? :)

[delucis]

Sharing https://github.com/delucis/astro-blog-full-text-rss/ here as I saw some people mentioning RSS. Would still be nice to improve the DX here but that repo demonstrates rendering MDX files in an RSS feed with the container API.

[eikowagenknecht]

Sharing my current solution here as well (Astro 5). The one from @delucis did not work for me because node complained about the imports. So this uses slightly different imports, which work for me. Also I use sanitize-html instead of ultrahtml as that one left in way too many tags and attributes and the filter options were cumbersome (no reasonable defaults).

Besides that, it should be mostly the same. Some code and comments is also just straight up copied from there :-)

import { getCollection, render } from "astro:content";
import { SITE } from "@/config";
import mdxRenderer from "@astrojs/mdx/server.js";
import reactRenderer from "@astrojs/react/server.js";
import rss, { type RSSFeedItem } from "@astrojs/rss";
import type { APIContext } from "astro";
import { experimental_AstroContainer as AstroContainer } from "astro/container";
import sanitizeHtml from "sanitize-html";

export async function GET(context: APIContext) {
  // Get the URL to prepend to relative site links. Based on `site` in `astro.config.mjs`.
  let baseUrl = context.site?.href || "https://*****.de";
  if (baseUrl.at(-1) === "/") baseUrl = baseUrl.slice(0, -1);

  // Create a new Astro container that we can render components with.
  // See https://docs.astro.build/en/reference/container-reference/
  const container = await AstroContainer.create();

  // Load MDX and React renderer.
  // Other renderers for UI frameworks (e.g. Vue, etc.) would need adding here if you were using those.
  container.addServerRenderer({
    renderer: mdxRenderer,
  });
  container.addServerRenderer({
    renderer: reactRenderer,
  });
  container.addClientRenderer({
    name: "@astrojs/react",
    entrypoint: "@astrojs/react/client.js",
  });

  // Load the published posts to add to our RSS feed.
  const posts = (
    await getCollection("posts", ({ data }) => Boolean(data.published))
  ).sort((a, b) =>
    // Satisfy TypeScript by checking for `undefined` first.
    !a.data.published || !b.data.published
      ? -1 // Sort by published date, descending.
      : b.data.published > a.data.published
        ? 1
        : -1,
  );

  // Loop over blog posts to create feed items for each, including full content.
  const feedItems: RSSFeedItem[] = await Promise.all(
    posts.map(async (post) => {
      const feedItem: RSSFeedItem = {
        title: post.data.title,
        description: post.data.description,
        pubDate: post.data.published,
        link: `/posts/${post.id}/`,
      };

      // Get the `<Content/>` component for the current post.
      const { Content } = await render(post);

      // Use the Astro container to render the content to a string.
      const postHtml = await container.renderToString(Content);

      // Process and sanitize the raw content with sanitize-html.
      // Also make sure that relative links are converted to absolute links.
      const sanitizedHtml = sanitizeHtml(postHtml, {
        allowedTags: [...sanitizeHtml.defaults.allowedTags, "img"],
        transformTags: {
          a: (tagName, attribs) => ({
            tagName,
            attribs: {
              ...attribs,
              ...(attribs.href && {
                href: attribs.href.startsWith("/")
                  ? baseUrl + attribs.href
                  : attribs.href,
              }),
            },
          }),
          img: (tagName, attribs) => ({
            tagName,
            attribs: {
              ...attribs,
              ...(attribs.src && {
                src: attribs.src.startsWith("/")
                  ? baseUrl + attribs.src
                  : attribs.src,
              }),
              ...(attribs.href && {
                href: attribs.href.startsWith("/")
                  ? baseUrl + attribs.href
                  : attribs.href,
              }),
            },
          }),
        },
      });

      feedItem.content = sanitizedHtml;
      return feedItem;
    }),
  );

  return rss({
    title: SITE.title,
    description: SITE.description,
    site: baseUrl,
    items: feedItems,
    customData: "<language>en-us</language>",
    stylesheet: "/assets/styles/rss.xsl",
  });
}

[bholmesdev]

Thanks for sharing this! It sounds like we have some unfinished business documenting how the container API can be used to address RSS feed generation. I made an issue on our docs suggesting a "recipe" that walks through the setup process. If anyone's interested in contributing to our docs, give it a look!

[delucis]

Would also be happy to accept a PR updating my RSS example repo to v5. I haven't had time yet and that would be super helpful!