Unknown Content Collection Error when content directory is a symlink

[openscript]

Astro Info

Astro                    v3.5.3
Node                     v20.6.1
System                   Linux (x64)
Package Manager          pnpm
Output                   static
Adapter                  none
Integrations             @astrojs/mdx
                         @astrojs/react

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

When ./src/content is a symlink and via getEntry data is retrieved from any collection, a Unknown Content Collection Error is thrown:

  Error reference:
    https://docs.astro.build/en/reference/errors/unknown-content-collection-error/
  File:
    /home/projects/github-tvxvbc/content/navigation/top.yml?astroDataCollectionEntry=true
  Stacktrace:
UnknownContentCollectionError
    at AstroError (/home/projects/github-tvxvbc/node_modules/astro/dist/core/errors/errors.js:33:5)
    at getEntryModuleBaseInfo (/home/projects/github-tvxvbc/node_modules/astro/dist/content/vite-plugin-content-imports.js:243:11)

What's the expected result?

It should be possible to retrieve collection data through symlinked content directories.

Link to Minimal Reproducible Example

https://stackblitz.com/edit/github-tvxvbc

Participation

• I am willing to submit a pull request for this issue.

Tasks

Preview Give feedback

No tasks being tracked yet.

[mikenikles]

Astro                    v4.1.1
Node                     v21.5.0
System                   macOS (arm64)
Package Manager          pnpm
Output                   static
Adapter                  none
Integrations             @astrojs/starlight
                         @astrojs/svelte

I have the same issue with the latest version of Astro & Starlight.

In astro.config.mjs, I added the following:

export default defineConfig({
  ...
  vite: {
    resolve: {
      preserveSymlinks: true
    }
  }
});

This results in the following error when I run pnpm dev

15:03:05 [ERROR] Cannot find module 'kleur/colors' imported from '/Users/m/webstone-education/app/node_modules/astro/dist/runtime/server/endpoint.js'
  Stack trace:
    at nodeImport (file:///Users/m/webstone-education/node_modules/.pnpm/[email protected]/node_modules/vite/dist/node/chunks/dep-R0I0XnyH.js:50537:25)
    at eval (/Users/m/webstone-education/app/node_modules/astro/dist/runtime/server/endpoint.js:3:50)

[natemoo-re]

This was marked as "has workaround" because you don't need to use symlinks, but it's not exactly the same thing. I recently dug into this myself and it seems fixable.

Some of the content collection code is explicitly blocking symlinks because we typically determine collections from the filepath. But IMO, symlinking the entire docs/ directory into the src/content/ directory should be supported.

I traced the error thrown to this part of the code, which might be a good starting place for anyone looking to dig into this.

https://github.com/withastro/astro/blob/main/packages/astro/src/content/utils.ts#L182-L197

[mikenikles]

Thanks for pointing me in that direction. I debugged it and it all looked ok. Despite symlinking src/content/docs to a directory outside the Astro Starlight app, that particular code worked fine.

I decided to manually install kleur to see what would happen. In the end, I added the following dependencies and now it all works 😮!

pnpm add kleur html-escaper clsx zod @astrojs/internal-helpers @pagefind/default-ui

I wonder if it's related to PNPM and the fact that the Astro Starlight app runs in packages/app – not sure that's really an issue though.

Update: I removed the above dependencies because it felt too hacky. What helped, but also isn't ideal, was solution 1 in the PNPM FAQs.

[Fryuni]

Some of the content collection code is explicitly blocking symlinks because we typically determine collections from the filepath.

Can't we use the symbolic path as if the symlink were not there, similar to what most shells do?
it would need some logic to check fail if it forms a cycle, but that is relatively simple to do.

[joshunrau]

I wrote a quick, isolated hacky vite plugin that solves this problem:

/** @typedef {NonNullable<import('astro').ViteUserConfig['plugins']>[number] } VitePluginOption */
/** @typedef {Extract<VitePluginOption, { name: string}>} VitePlugin */

/**
 * Workaround to allow Astro content collections to work with symlinks and pnpm

 * @param {Object} options
 * @param {Record<string, string>} options.collections - a mapping of collection names to directories, relative to the root defined in astro.config.js
 * @returns {VitePlugin}
 */
const symlinkPlugin = ({ collections }) => {
  /**
   * Finds the plugin object with the specified name
   * @param {VitePluginOption[] | null | undefined} plugins
   * @param {string} targetName
   * @returns {VitePlugin | null}
   */
  const resolveTargetPlugin = (plugins, targetName) => {
    for (const plugin of plugins ?? []) {
      if (!plugin || typeof plugin !== 'object' || plugin instanceof Promise) {
        continue;
      } else if (Array.isArray(plugin)) {
        const target = resolveTargetPlugin(plugin, targetName);
        if (target) return target;
      } else if (plugin.name === targetName) {
        return plugin;
      }
    }
    return null;
  };

  return {
    config: async ({ plugins, root }) => {
      if (!root) {
        throw new TypeError('Expected root to be defined in astro config');
      }
      const contentDir = path.resolve(root, './src/content');
      const collectionDirents = await fs.readdir(contentDir, { encoding: 'utf-8', withFileTypes: true });

      const targetName = 'astro:content-imports';
      const target = resolveTargetPlugin(plugins, targetName);
      const transform = target?.transform;
      if (!target) {
        throw new Error(`Failed to find target plugin: ${targetName}`);
      } else if (typeof transform !== 'function') {
        throw new Error(`Unexpected type of transform method: ${typeof transform}`);
      }

      /**
       * Attempt to resolve the absolute path to the symbolic link for the real directory 'id'
       * @param {string} id
       */
      const resolveSymbolicLink = (id) => {
        for (const [name, relpath] of Object.entries(collections)) {
          const targetPrefix = path.resolve(root, relpath);
          if (id.startsWith(targetPrefix)) {
            const dirent = collectionDirents.find((dirent) => dirent.name === name);
            if (!dirent) {
              throw new Error(`Expected collection '${name}' does not exist in directory: ${contentDir}`);
            } else if (!dirent.isSymbolicLink()) {
              throw new Error(`File is not a symbolic link: ${path.join(dirent.path, dirent.name)}`);
            }
            return id.replace(targetPrefix, path.join(dirent.path, dirent.name));
          }
        }
        throw new Error(`Failed to resolve symbolic link for ID: ${id}`);
      };

      target.transform = async function (code, id, options) {
        /** @type {ReturnType<Extract<VitePlugin['transform'], Function>>} */
        let result;
        try {
          result = await transform.call(this, code, id, options);
        } catch (err) {
          if (!(err instanceof Error && err.name === 'UnknownContentCollectionError')) {
            throw err;
          }
          id = resolveSymbolicLink(id);
          result = await transform.call(this, code, id, options);
        }
        return result;
      };
    },
    name: 'symlink-plugin'
  };
};

For example, if I have blog and docs at the root of my monorepo, with the astro app in apps:

astro.config.js

export default defineConfig({
  ...,
  vite: {
    plugins: [
      symlinkPlugin({
        collections: {
          blog: '../../blog',
          docs: '../../docs'
        }
      })
    ],
    resolve: {
      // the default behavior, but make sure this is not true as it is handled by plugin
      preserveSymlinks: false
    }
  }
})