Content Layer

[FredKSchott]

Body

Summary

• Explore a new and improved content layer for Astro.
• Improve the current experience of loading/defining data into content collections
• Improve the current experience of querying data from content collections

Background & Motivation

Content Collections are a key primitive that brings people to Astro. Content Collections make it easy to work with local content (MD, MDX, Markdoc, etc) inside of your Astro project. They give you structure (src/content/[collection-name]/*), schema validation for frontmatter, and querying APIs.

Goals

• Explore a new and improved content layer for Astro.
• Improve the current experience of loading/defining data into content collections
• Improve the current experience of querying data from content collections

Example

See discussion below for examples.

[FredKSchott]

Defining collections of data that lives anywhere

It would be great if your content could live anywhere, and if content collections were not tied to any one folder structure. For example:

// A folder full of Markdown (MDX) files
defineCollection({
    name: 'blog',
    data: glob('./content/blog/*.mdx'),
  });
// A single file containing an array of objects
defineCollection({
    name: 'authors',
    data: file('./content/authors.json'),
});
// Remote data, loaded with a custom npm package
defineCollection({
    name: 'articles',
    data: storyblokLoader({startsWith: 'articles/posts'}),
});
// Custom data, loaded from anywhere you'd like
defineCollection({
    name: 'my-custom-collection',
    data: () => { /* ... */ },
});

@bholmesdev built a prototype earlier this year that seeded Astro DB from the file system, and supported watching the file system for updates. I believe that same basic API / concept could work here. Something like:

// Every function helper (glob(), file(), storyblok(), etc) 
// would implement this `data` API:
defineCollection({
    name: 'my-custom-collection',
    data: ({
      // db: insert/update/delete entries based off of the data you load here
      //     for example, fs.scandir() a directory, then db.insert() every file as an entry
      db, 
      // watcher: optionally watch the file system for changes, if needed/relevant
      //       for example, update a collection during development, as files change
      watcher,
    }) => {
      // 1. fetch your data
      // 2. insert it into the DB
    },
});

Open Questions:

How is data stored in content collections? This assumes db. I would love to see content collections powered by a SQLite DB (does not need to be Astro DB).

Is data stored in structured tables, or an unstructured KV JSON Document store? If stored as unstructured JSON, then we could still use Zod for schema validation, plus schema and data could be decoupled. AKA, a generic data function could be provided by an npm package, and you could provide the schema parser to define only the fields you care about, refined from the complete source object stored in the DB.

[stereobooster]

Similar concept https://github.com/stereobooster/braindb: it watches files in any folder and populates SQLite database that later can be used to query the content

[matthewp]

This proposal has been accepted to stage 2: #946

Please continue the discussion there.