Error when using a Reference and an inline Loader

[mollimoll]

Astro Info

Astro                    v4.15.9
Node                     v18.20.3
System                   Linux (x64)
Package Manager          npm
Output                   static
Adapter                  none
Integrations             none

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

I'm using an inline loader, combined with a schema reference to another collection. When I run npm run build I receive an error stating the "Collection does not exist or is empty."

I can reproduce this issue on StackBlitz with the experimental contentLayer flag, but it is also occurring in the Astro 5 beta.

What's the expected result?

I'd expect the build to pass, and that Astro does not fail if a collection is empty (if that is indeed the issue).

Link to Minimal Reproducible Example

https://stackblitz.com/edit/withastro-astro-mn5gwx?file=src%2Fcontent%2Fconfig.ts

Participation

• I am willing to submit a pull request for this issue.

[StevenDepa]

I was going to experiment with the new content layer because I really need this feature, but i saw this issue, and since it seems to be an use case I need, I tried to investigate. The issue appears to be the same in either dev or build, and both in Astro v4.15.9 and the last beta.
Looking at the code, the problem seems to be that the schema validation code tries to load the data store

astro/packages/astro/src/content/runtime.ts

Line 590 in 2594eb0

const store = await globalDataStore.get();

to find the referenced collection, but the store is loaded completely empty

astro/packages/astro/src/content/vite-plugin-content-virtual-mod.ts

Line 136 in 2594eb0

if (!fs.existsSync(dataStoreFile)) {

because the data store is not written to disk until all the collection have finished loading, and before that every collection tries to validate the schema.

astro/packages/astro/src/content/content-layer.ts

Line 172 in 2594eb0

schema = await schema();

I can see that the mutable data store tries so hard to save the file to disk

astro/packages/astro/src/content/mutable-data-store.ts

Line 188 in 2594eb0

this.writeToDisk(this.#file!);

but it's file path reference is empty until writeToDisk is called at the end of collection loading.

astro/packages/astro/src/content/content-layer.ts

Line 227 in 2594eb0

await this.#store.writeToDisk(cacheFile);

I may be missing something but I think that every collection "existence" should be initialized before trying to validate schemas, otherwise references will never work.

[carlosmonterocanabal]

I had the same problem with inline loader with Astro beta 5. I show an example of my use case:

const categories = defineCollection({
    loader: glob({ pattern: "**\/*.yml", base: "./src/data/categories/" }),
    schema: z.object({
        title: z.string(),
        description: z.string(),
    }),
});
const products = defineCollection({
    loader: async() => {
        const categories = await getCollection('categories'); // <= HERE is the error
        // call external API with id from each category
        return products.map((p) => ({
            id: p.id,
            ...p
        }));
    },
    schema: z.object({
        title: z.string(),
        description: z.string(),
        price: z.number(),
    }),
});

When I run npm run build I get

The collection "categories" does not exist or is empty. Ensure a collection directory with this name exists.

[ascorbic]

@mollimoll thanks for the report. I'm investigating this now.

@carlosmonterocanabal your issue is different: you shouldn't be trying to use getCollection inside the loader, because the order of execution is undefined and (as you see) you can't tell which collection will be loaded first. One collection can't refer to another in that way.

@StevenDepa Are you also trying to getCollection inside a loader? Can you share your code? The runtime is not supposed to be used before the loaders have run.

[ascorbic]

@mollimoll ok, I'm reproduced it and can see where the issue is. It's a race condition. I'll get a fix out. Thanks again for the report and great repro!