Help with MDX richtext editor

[konsumer]

I am working on a richtext editor for MDX here. My main goal is to use regular standard ASTs (instead of slate AST for example) and give users lots of control for overriding how stuff fits together, with reasonable & simple defaults and examples.

The main thing I am having trouble with is turning the HTML from the editor into MDX, and I'm a bit lost in the vast unified/syntax-tree/mdx ecosystem, having trouble figuring out even what libraries I should be using. I made a simplified sandbox that illustrates the problem, here.

The core of it is this:

// remark plugin to insert the editor UI for react-components
function remarkPluginSetupEditUi() {
  return (tree) =>
    visit(tree, (node) => {
      if (node.type === "mdxJsxFlowElement") {
        node.type = "html";
        node.value = `
<div data-tag=${JSON.stringify(node.name)} data-props="${encodeProps(node)}" class="slim-component flex gap-2 items-center justify-end">
  <div class="grow">${node.name}</div>
  <button onClick="alert('Here I would popup editor for ${
    node.name
  } in parent')">EDIT</button>
  <button onClick="alert('Here I would trigger delete in parent')">DELETE</button>
</div>
      `;
        delete node.children;
      }
    });
}

// rehype plugin to turn HTML from editor into mdast (for turning into MDX)
function rehypePluginEditUiToJsx() {
  return (tree) =>
    visit(tree, (node) => {
      if (
        node.type === "element" &&
        (node?.properties?.className || []).includes("slim-component")
      ) {
        const tag = node?.properties?.dataTag;
        const props = decodeProps(node);

        // What to do here to turn into JSX?
        console.log("found your component-editor", { tag, props });
      }
    });
}

// Turns MDX into HTML (for editor)
export const mdx2html = (input, options = {}) => {
  return String(
    unified()
      .use(remarkParse, { fragment: true })
      .use(remarkMdx)
      .use(remarkPluginSetupEditUi, options)
      .use(remarkHtml, { sanitize: false })
      .processSync(input)
  );
};

// Turns HTML (for editor) into MDX
// XXX: this is the one having trouble!
export const html2mdx = (input, options = {}) => {
  return String(
    unified()
      .use(rehypeParse)
      .use(rehypePluginEditUiToJsx, options)
      .use(rehypeRemark)
      .use(remarkStringify)
      .processSync(input)
  );
};

remarkPluginSetupEditUi outputs something that essentially looks like this:

<div data-tag="Tester" data-props="ENCODED_PROPS" class="slim-component">
  <div>Tester</div>
  <button>EDIT</button>
  <button>DELETE</button>
</div>

I may be going about it wrong, but I basically turn the MDX into HTML, with a div.slim-component to contain the UI for messing with the props of the react-element. In the editor it appears as buttons, and externally I can popup dialogs to allow editing the props and stuff. It encodes the AST props & children into data attribs, then pulls it back out later.

On the other side, I want to turn the RTE's HTML AST (with div.slim-component) back into MDX string, using the tag & props, but I'm not sure how to do this. Is there an easier way to do this? Do I need a plugin in html2mdx that understands JSX, so I can output that kind of AST in rehypePluginEditUiToJsx?

[wooorm]

Hi David!

MDX is about JSX. Not about HTML.
JSX has components. JSX has embedded JavaScript.
HTML doesn’t have that.
Sure, they might look similar: <span>a</span> is both valid HTML and JSX.
But they’re different: JSX is an extension to JavaScript.

You notice this with mdxJsxFlowElement already. That’s a JSX element. Change that. Don’t overwrite it with something else.

Also: don’t inject strings of HTML into an AST.
AST are for representing constructs as data structures, objects, nodes.
Strings are exactly what you don’t want.

overriding how stuff fits together

That’s what components can also do?

but I basically turn the MDX into HTML

Why do you want to turn MDX into HTML? I thought Slate has an AST

RTE's HTML AST

If you have an AST, transform that AST. Don’t serialize it as HTML. If you are working on an editor, you are wasting time by with all the parse/serialize/parse/serialize iterations

---

I recommend taking a step back and reformulating the question. What is it you want to do?

You might want to take a look at:

• https://github.com/blocks/blocks
• https://github.com/virtuoso-dev/lexical-mdx
• https://github.com/inokawa/remark-slate-transformer

[konsumer]

@wooorm I appreciate the help, but I think you might be misunderstanding my question. I will try to clarify.

Why do you want to turn MDX into HTML? I thought Slate has an AST

I am trying to make my own richtext editor, not use slate. I don't like how slate's config works, and it has a totally different, less standard AST. If you are using slate, it takes a great deal of configuration code to get it working in some way that isn't in the examples (even the examples take a lot of config.) Mine is not like that. It uses regular AST stuff (so you can use syntax-tree/unified plugins to do more with it, etc.) And config is very simple. To add or change the toolbar for example, you can just make a very simple react toolbar, and you're not locked into using their more rigid setup with lots of overrides and configuration.

MDX is about JSX. Not about HTML.
Also: don’t inject strings of HTML into an AST.

I am not really trying to make HTML, like as my final-form. I could do that easily, as a sidenote, with renderToStaticNodeStream or similar. It's very doable, but that is not what I am trying to do. I am trying to make HTML that represents the MDX in the RTE, since that is how richtext editors typically work (they are not JSX, or MDX, MD, or an AST, only editable HTML, represented by a regular DOM) and then convert that back to MDX string. If you look at the example, I don't have a problem doing that part. I can make a more complete example, if that is helpful, but have a look at the part that renders the HTML. This part is working fine, and I don't think there is any other way to output HTML other than strings (other HTML AST stuff is stripped, there) in that exact spot in the pipeline (when converting the original MDX source to HTML to be used in the RTE.) I've already got a working richtext editor that does all the parts except children of JSX elements, in the MDX. I am happy to accept that is the wrong way to make the HTML for the editor (as I said, I am no unified expert and although I have worked on a few RTE, it's not my area of expertise, either) but I think I would need to see an example of what you mean, since as I said, mine is working great.

There are 3 parts to the MDX RTE, in this solution:

• convert MDX string (yes, JSX too) into a HTML representation that can be edited in a richtext editor. When the user clicks on the inline element that represents the JSX, they can edit the props. I mark the things (divs with a couple buttons inside them) that represent inline JSX as "not editable"
• the UI for the actual editor, that lets you edit HTML text, etc. This is the regular RTE stuff, pretty much any HTML RTE works like this part
• convert the current HTML in the RTE back into MDX string

The last part is what I am needing help on, or I need to understand why this is the wrong approach. This seems like the only good way to do this to me, and slate itself works very similar, just without using rehype/remark.

[konsumer]

I dunno if this will clarify, but here is a fork of the sandbox that rebuilds the AST coming back from the RTE's HTML in the rehypePluginEditUiToJsx plugin. It appears to be working to build the AST it's supposed to, and injects a mdxJsxFlowElement from the original back into the output (in place of the div.)

The last piece I am missing is "how do I turn the mdast, which includes mdxJsxFlowElement back into a string of MDX?". I tried this, which does not work:

export const html2mdx = (input, options = {}) => {
  return String(
    unified()
      .use(rehypeParse)
      .use(rehypePluginEditUiToJsx, options) // my plugin that outputs mdxJsxFlowElement
      .use(rehypeRemark)
      .use(remarkMdx) // does this work here? I don't think it does.
      .use(remarkStringify)
      .processSync(input)
  );
};

It doesn't understand mdxJsxFlowElement, so just renders it's children without the JSX tag:

[konsumer]

I could be wrong, but I think my problem is that I am making a rehype plugin, but once it gets to rehypeRemark (to convert the hast to mdast) it strips out the unknown mdxJsxFlowElement. Maybe a rehype plugin that acts in hast-space would work better. Is there an equivalent to mdxJsxFlowElement in hast? I tried using rehype-remark, but it seems like it does something else, or maybe I was using it wrong.

[konsumer]

By using a handler for rehypeRemark I can parse the JSX back into a string with mdast-util-mdx-jsx, but I'm still not sure how to inject that back into the AST: https://codesandbox.io/p/sandbox/black-fast-ynxdnh I feel like I am very close, but still not quite getting how it fits together.

[wooorm]

[…] as a sidenote, with renderToStaticNodeStream or similar […] I am trying to make HTML that represents the MDX in the RTE

With the first (renderToStaticNodeStream) you are evaluating things.
With the second, you are not.

MDX includes JavaScript. Such as {1 + 1}. What do you want in your output?
2 or {1 + 1}? What about:

import data from './data.js'

<div id={data.id} />

How do you want to represent that in HTML?
And how do you want to convert that back?

You can completely do this. I am just not sure how you want to go about it.

I think my problem is that I am making a rehype plugin, but once it gets to rehypeRemark (to convert the hast to mdast) it strips out the unknown mdxJsxFlowElement.

You are correct!

I feel like I am very close, but still not quite getting how it fits together.

You are quite close:

1. Your remarkPluginSetupEditUi is not very good yet, you should not
   inject strings of HTML in an AST. As you’ve learned, there’s an AST for
   HTML: rehype/hast.
   Replace it with a rehype plugin.
   This means you need to pass through nodes from mdast to hast untouched.
   • https://github.com/remarkjs/remark-rehype#optionspassthrough
   • https://github.com/mdx-js/mdx/blob/e77be3a/packages/mdx/lib/node-types.js#L5
     You should use rehype-stringify instead of remark-html.
     Use the nodes you are given and that are expected:
   • https://github.com/syntax-tree/mdast-util-mdx-jsx#syntax-tree
   • https://github.com/syntax-tree/hast#nodes
     Learn how to replace nodes: do not use delete:
     https://unifiedjs.com/learn/recipe/remove-node/.
     Do not serialize the entire Node. The children are content again,
     which you can show instead. I would only serialize the attributes.
2. Now you can do the inverse when going from rehype to remark:
   1. Turn the divs you made into flow elements with a rehype plugin:
   • https://github.com/syntax-tree/mdast-util-mdx-jsx#syntax-tree
   • https://github.com/syntax-tree/hast#nodes
   2. Pass through the MDX nodes when going rehype-remark
   3. Profit!
3. At this point, you are doing a lot of AST work!
   I think you’d be better off to replace things with utilites:
   remark-rehype -> mdast-util-to-hast, and such.
4. As you are in a browser, with a DOM, that’s also sort of like an AST.
   So instead of going through HTML (slow), transform them:
   • https://github.com/syntax-tree/hast-util-from-dom
   • https://github.com/syntax-tree/hast-util-to-dom

[konsumer]

With the first (renderToStaticNodeStream) you are evaluating things.
With the second, you are not.

Yep, exactly. I just meant I can use proper JSX evaluation, if I just want JSX-to-HTML, but in my thing it's not really that since I am trying to turn the string source into another string that represents it, then later, back into a string more like the original. I think we are on same page, here.

You can completely do this. I am just not sure how you want to go about it.

A quick hack I did before (I made a plugin to handle props, where it handle most props ok, but not children) was to to just JSON-encode values and just pass anything that was not a direct JSON-able value, so plain strings and numbers worked ok. It wasn't perfect, but worked ok for our editor, mostly. It seems like passing the AST tree of it works well (and comes out as proper JSX on the other side with toMarkdown) and I think If I keep the AST but encode what the user enters in the same format, it should work well.

It may help to explain the usecase a bit more here. Since users can easily modify the toolbar and how it parses things, I don't really need to support every kind of JSX component, directly, I just need to pass the value to them in a reasonable form that they can use however they like. An example is if I have this I want to be able to insert:

<Button href="#contact">MORE INFO</Button>

I can make a specific "Button" button on the RTE toolbar that pops up a modal that asks about href and children. I don't need to actually understand the JSX of it really, just generate the string and parse the string into those known 2 props. I am trying to make a more general RTE that can handle whatever the user wants with a little react toolbar code, but acts sort of similar to TinaCMS's RTE, which has "Embed" stuff, without having to setup a whole sub-field schema to describe the type. Instead the user just makes their own button that has access to the value and some state-stuff from the editor, and they can do whatever they want. We offer our own field-widgets as exports, the user can build something simple, very easily, but also more complicated stuff we didn't consider. JSX just becomes a way to contain "some extended thing goes here" in the MDX.

On points 1&2, I totally I agree. It's a complete hack because I can't figure out how things are supposed to fit together. I like what you are saying about keeping it in hast, and this feels like the right answer to me, for sure. I will try this.

At this point, you are doing a lot of AST work!

Agreed! I think part of it is trying to figure out how it fits together. I will probably prototype this rehype hast-only flow with similar over-complicated chains, but I will totally refactor once I get it working to use utils, and simplify.

So instead of going through HTML (slow), transform them

I will try this! This also seems like a really good path. A lot of our demos are string-oriented to try to work out how it will look, but I definitely see the value of staying in DOM/AST if possible.