Question: Extending marked

[retani]

Is it possible to extend markdown with custom tokens?

marked has the marked.use() method for this (see https://marked.js.org/using_pro#use) but I cound't find it exposed. Or maybe somehow through the options object (see https://marked.js.org/using_pro#extensions)?

And if not, how would one go about implementing this into svelte-markdown?

[pablo-abc]

So, in the current implementation of this package we are only importing the Lexer from marked. Everything else gets tree-shaked out of the bundle. Skimming the docs it seems there's no way to extend it like this from just the Lexer. The options prop in <SvelteMarkdown /> is exactly the same one as Marked's to which you can pass a "tokenizer" or "renderer" property which might already do what you want? If it's not possible like that then I'd be open to add that option somehow.

[retani]

I've tried, but there are a few problems.

1. Extending tokenizer and renderer using marked.use(options) works, but when passing the options object directly marked(src, options), it seems that the tokenizer and renderer functions get overwritten instead of being merged (this.tokenizer.space is not a function)
2. I am not sure if it is also necessary to modify the Lexer, because I am not skilled enough in this area. I tried to do it with just using a tokenizer and a renderer, but it didn't work out well. Could be just a lack of knowledge.

BTW what I am trying to do it to implement footnotes. [fn] footnote [/fn] should turn into <sup title="footnote">[1] footnote</sup>

https://github.com/retani/marked-footnotes

[pablo-abc]

Thanks for the repo! I'll be tinkering with it. From what I see you're using Marked directly. I'll get back to you as soon as possible.

[retani]

Thanks for checking it out. I was using marked directly to see if I can extend it using just the options. The strangest bit is that it doesn't work without marked.use() at all.

I have found a workaround for my specific problem, but would be good to being able to extend it and knowing how. I realized that extending markdown is something for true regex experts.

[pablo-abc]

Sorry for the delay. I'll look into this. That is really weird. I've been debating on whether I should migrate this to remark instead of marked due to its plugins (and because React Markdown uses remark and I'm trying to recreate a similar API), but sadly I have not even had time to experiment with it.

[retani]

My experience with marked hasn't been great. remark's plugins are quite nice. That's basically all I know :)

[raulfdm]

Hey guys.. sorry to jump in in this conversation but I have a question regarding remark: Will that work client-side?

I'm asking that because I'm diving deep into a remote mdx/shortcodes solution to run client-side and it seems because unified rely on a lot of node apis and package, it won't work exactly like how marked works right now. 🤔

[pablo-abc]

Svelte Markdown is inspired by React Markdown which uses remark behind the scenes and works client side. The reason I used marked was that I wanted a working version ASAP and remark was giving me some issues with rollup.

[raulfdm]

Hmm... gotcha.

Maybe I have to read more about how React Markdown does that... Are you open for contributions?

I'm trying to find a good solution to fetch "mdx" content from a CMS and render truly Svelte components without rely on {@html}.

[pablo-abc]

@raulfdm I am definitely open to contributions. The only issue I see is that with mdx style syntax, Svelte components do need a build step. If you wanted to fetch it in the browser and process it there you'd need to bundle the Svelte compiler with your app which takes away most benefits of Svelte. In this scenario I'd usually just recommend mdsvex.

[raulfdm]

The problem I'm trying to solve is this one:

But I've encountered a lot of issues which I realized that's because of Vite SSR:

• ReferenceError: module is not defined sveltejs/kit#1961
• Vite SSR Dependency Problems? Let us know! sveltejs/kit#1895

mdsvex is indeed a really good candidate but it's truly focused on local files .svx

Unless I walk through ALL posts via node script and generate those files, It won't work. Even importing the compile method, it would throw the same SSR error.

[genericFJS]

@retani @pablo-abc I've found a solution which enables the use of extensions (but costs an import of marked).
See #38 .

[genericFJS]

You can pass extensions to SvelteMarkdown components indirectly by first passing your extensions to marked.use() and then reusing the marked.defaults as the options-parameter for the component.
If you want to set other options in addition to the extensions, you may use marked.setOptions() and use the marked.defaults afterwards (or just manipulate the marked.defaults object yourself).

Example:
File App.svelte

<script>
  import SvelteMarkdown from "svelte-markdown";
  import LatexMarkdown from "./LatexMarkdown.svelte";

  const latexTokenizerExtension = {
    name: "latex",
    level: "inline",
    start(src) {
      return src.match(/\[\[/)?.index;
    },
    tokenizer(src, tokens) {
      const rule = /^\[\[latex\s*(?:color="(.*)")?\]\]/;
      const match = rule.exec(src);
      if (match) {
        console.log(match[1]);
        return {
          type: "latex",
          raw: match[0],
          text: match[0],
          color: match[1],
        };
      }
    },
  };

  marked.use({ extensions: [latexTokenizerExtension] });
  // marked.setOptions(…)
  const options = marked.defaults;

  const renderers = {
    latex: LatexMarkdown,
  };

  const source = `
  # This is a header

This is a paragraph.

This is red [[latex color="red"]] text.

* This is a list
* With two items
  1. And a sublist
  2. That is ordered
    * With another
    * Sublist inside

| And this is | A table |
|-------------|---------|
| With two    | columns |`;
</script>

<SvelteMarkdown {source} {options} {renderers} />

Imported file LatexMarkdown.svelte

<script>
  export let color;
</script>

<span style={"color: " + color}>LaTeX</span>

[AshfordN]

Has this been tested? I'm trying to do something similar using marked-katex-extension, but I noticed you're passing the extension to marked.use(). Where is this defined? It appears that marked is undefined in your example as well.

[AshfordN]

So far I have this:

<script>
  import { marked } from 'marked';
  import katex from 'marked-katex-extension';
  import Markdown from 'svelte-markdown';

  marked.use(katex({ throwOnError: true }));
  const options = marked.defaults;

  let source = `# Markdown-Latex Example

Inline: $\\frac{4}{3}$

Block: 
$$c = \\pm\\sqrt{a^2 + b^2}$$`;
</script>

<svelte:head>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" integrity="sha384-bYdxxUwYipFNohQlHt0bjN/LCpueqWz13HufFEV1SUatKs1cm4L6fFgCi1jT643X" crossorigin="anonymous" />
</svelte:head>

<Markdown {source} {options} />

But the LaTeX portions are not displayed, as shown below:

I'm not sure what I'm missing

[AshfordN]

Sorry for the back-to-back posting, but the following code seems to work:

File: App.svelte

<script>
  import { marked } from 'marked';
  import katex from 'marked-katex-extension';
  import Markdown from 'svelte-markdown';
  import KatexRenderer from './KatexRenderer.svelte';

  marked.use(katex({ throwOnError: true }));
  const options = marked.defaults;
  const renderers = { blockKatex: KatexRenderer, inlineKatex: KatexRenderer };

  let source = `# Markdown-Latex Example

Inline: $\\frac{4}{3}$

Block: 
$$c = \\pm\\sqrt{a^2 + b^2}$$`;
</script>

<svelte:head>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" integrity="sha384-bYdxxUwYipFNohQlHt0bjN/LCpueqWz13HufFEV1SUatKs1cm4L6fFgCi1jT643X" crossorigin="anonymous" />
</svelte:head>

<Markdown {source} {options} {renderers} />

File: KatexRenderer.svelte

<script>
  import Katex from 'svelte-katex';

  export let text;
</script>

<Katex>{text}</Katex>

Something about this feels a bit inefficient though, so if anyone has any ideas for improvement, I'd appreciate the feedback.