@docusuarus/mdx-loader frontmatter removal prevents rehype vfile having correct line numbers

[vjpr]

docusaurus/packages/docusaurus-mdx-loader/src/index.js

Lines 24 to 58 in ae45b11

	module.exports = async function docusaurusMdxLoader(fileString) {
	const callback = this.async();
	const {data, content} = matter(fileString);
	const reqOptions = getOptions(this) || {};
	const options = {
	...reqOptions,
	remarkPlugins: [
	...(reqOptions.beforeDefaultRemarkPlugins || []),
	...DEFAULT_OPTIONS.remarkPlugins,
	[
	transformImage,
	{staticDir: reqOptions.staticDir, filePath: this.resourcePath},
	],
	[
	transformLinks,
	{staticDir: reqOptions.staticDir, filePath: this.resourcePath},
	],
	...(reqOptions.remarkPlugins || []),
	],
	rehypePlugins: [
	...(reqOptions.beforeDefaultRehypePlugins || []),
	...DEFAULT_OPTIONS.rehypePlugins,
	...(reqOptions.rehypePlugins || []),
	],
	filepath: this.resourcePath,
	};
	let result;
	try {
	result = await mdx(content, options);
	} catch (err) {
	return callback(err);
	}

I am writing an plugin that adds the source file + line number to every heading element.

Below you can see that the frontmatter is being parsed and then only the remaining markdown content is passed onwards.

const {data, content} = matter(fileString);
// ...
result = await mdx(content, options);

This means that in a rehype plugin the vfile.history[].position.start.line will not be correct, it not take into account the frontmatter lines.

I think a better approach would be to parse the frontmatter using a remark plugin so to retain the original line numbers. Like this: https://www.npmjs.com/package/remark-extract-frontmatter

This should also be corrected on https://mdxjs.com/guides/custom-loader#custom-loader page I think.

Workaround

I could use grey matter to parse the gray matter again in my plugin but that means a lot of duplicate file read operations.

Remark

In remark plugins, I am just using the first comment to apply config.

<!-- {tocMaxDepth: 1, tocMaxDepth: 4} -->

# foo

const plugin = (options = {}) => {
  const name = options.name || 'toc';

  const transformer = (node) => {

    const firstComment = node.children.find(node => node.type === 'comment')
    let config = {}
    if (firstComment) {
      config = JSON.parse(firstComment.value)
    }

	// stuff...

  }

}

[JPeer264]

I agree that it is cumbersome not to have the frontmatter inside rehype/remark plugins.

We have a different use case, we only want to check certain frontmatter values inside a remark plugin, but have no chance of doing that, without reading the file again.

It would make sense to put fileString instead of content into the processing here. But I think it is also cumbersome if the logic is duplicated to remove frontmatter and the contenttitle yet again from the endresult as plugin.

Btw this is the updated source:

docusaurus/packages/docusaurus-mdx-loader/src/loader.ts

Lines 161 to 165 in 7ab2bd3

	const {frontMatter, content: contentWithTitle} = parseFrontMatter(fileString);
	const {content, contentTitle} = parseMarkdownContentTitle(contentWithTitle, {
	removeContentTitle: reqOptions.removeContentTitle,
	});

and

docusaurus/packages/docusaurus-mdx-loader/src/loader.ts

Lines 209 to 214 in 7ab2bd3

	result = await compiler
	.process({
	contents: content,
	path: this.resourcePath,
	})
	.then((res) => res.toString());

[slorber]

@vjpr as users are going to upgrade to Docusaurus v3 and MDX v2 soon they will likely encounter compilation errors and it's important to show correct line numbers if possible.

In #9386 I'm now trying to avoid altering the original content before passing it to MDX, using remark-frontmatter and another new internal plugin to handle the contentTitle export. It seems to fix that line number problem nicely.

---

@JPeer264 the front matter becomes available in the AST as raw string, with nodes of type yaml/toml.

remark-frontmatter does not parse that string, and we don't need this in Docusaurus core so if you want to do something with those new AST nodes you are free to handle those yourself, or use a community plugin:

• https://github.com/mrzmmr/remark-extract-frontmatter
• https://github.com/vfile/vfile-matter
• https://github.com/remcohaszing/remark-mdx-frontmatter (not sure it's a good idea because we already add a frontMatter named export to the compiled output)

I understand it's not super convenient to have to parse multiple times the same front matter, but it should unlock your use-case more efficiently now.

In the future we'll try to optimize all this, because currently we already parse files at multiple layers (mdx loader and plugin code) and it's not super efficient. I'm not sure yet how to do that properly though.

[slorber]

Nevermind, found a way to expose the frontMatter parsed data we already have without reparsing it.

const vfile = new VFile({
  value: content,
  path: filePath,
  data: {
    frontMatter
  }
});

function remarkPlugin() {
  return async (root, vfile) => {
    console.log(vfile.data.frontMatter)
  };
}