page loaders

[mbostock]

A page loader is like a data loader, but for pages instead of files or assets: it allows you to generate Markdown dynamically (at build time) rather than being limited to static Markdown. Page loaders allow server-side rendering #931 and are particularly useful in conjunction with parameterized pages #1523.

(Unlike earlier versions of this PR, this implementation is still Markdown-specific. In the future, we can explore authoring pages in HTML instead of Markdown if desired, but this should be treated as an orthogonal problem.)

TODO

• Page loaders
• Adopt parameterized page loaders in place of generate-themes script
• Documentation
• Tests

Fixes #931.

[mbostock]

We could maybe call this “page generator” (and “generated pages”) instead.

[Fil]

Yay, I wanted to explore something like this too. In my idea (that I hadn't really tried to implement yet) parseMarkdown would have been one of the input helpers, instead of something that is applied by default to all (transformed) pages. In order to outline some design considerations:

my idea was something like this:

md  -> parseMarkdown -> parsed page
mdx -> parseMdx      -> (same)
js  -> runJs         -> (same)

Whereas this PR treats md as the pivot format:

js ---|
md ---|
mdx --|
       \-> md -> parseMarkdown -> parsed page

The only advantage to my approach is that the generated page wouldn't go through markdown at all if your input format was not md, thus making sure there are no surprises after your page generator has run. (However, outputting html with no line breaks should guarantee that too.)

This PR's approach is much better:

• well “contained” in terms of API, and simple to reason about if you want to write a new page format generator.
• You can even run the page generators independently from Framework with node home.page.js > home.md; like you do with data loaders.
• It does not require the page format generator to be aware of how Framework works: you could even use pandoc.

[mbostock]

Indeed that was my original plan which you can see in the first commit, and I’d still like to support that, too — particularly for JSX which is compiled to JavaScript that needs to be evaluated on the server and then serializes to HTML. I was originally thinking of generating JSON using the RenderPage interface, but that won’t quite work as-is because parseMarkdown calls parseJavaScript internally and I don’t want to serialize the entire AST for JavaScript; so I think I’ll pull the parseJavaScript out and have parseMarkdown simply return the unparsed source for code blocks. I also want to support HTML as an extension, perhaps with <script type="observablehq"> for code blocks? But not a high priority.

[mbostock]

Now that we’ve landed JSX and React separately #1429, I want to reboot this PR to focus just on page loaders.

I want to stick with Markdown as the serialized representation of the page for now. We do need a serialized representation because we want to be able to write page loaders in other languages (e.g., index.md.py for a Python page loader) just like we do with data loaders. I think we could also support HTML as a serialized representation (e.g., index.html.py) and even static HTML pages (e.g., index.html instead of index.md). But we’d need to design that representation (for example how do we represent reactive code blocks, probably <script type="observablehq"> elements?) and that can be a future enhancement.

[mbostock]

As part of this work, I refactored the data loaders implementation a bit. Instead of an Asset interface, I made a StaticLoader implementation of the Loader interface; this means that downstream consumers of loaders don’t have to care whether a file is static or generated dynamically by a loader. (No "load" in loader test.) I also changed loader.path so that it is relative to the source root rather than relative to the current working directory, which felt more consistent.

[mbostock]

Rebooted this PR. I marked this as ready for review since I’m planning on merging it into the parent parameterized routing PR and landing both together. Both are close to ready — mostly needing documentation and tests, and porting the theme previews over to parameterized page loaders.