Markdown: Execute markdown after content loaded by later middlewares

[jimjimovich]

Feature Request

Continue the work which was started in #1611 and #1649 to make the staticfiles file server
the only middleware that hits the disk and loads content by extending this feature to markdown. This would allow the markdown directive to be used with other middlewares such as proxy.

[jimjimovich]

I've started digging into this and am willing to give it a try. I'm sure I'll need help along the way!

[mholt]

Sounds good! Feel free to submit a PR at any time to track your progress.

[jimjimovich]

Having just looked at the v2 Markdown module, I understand why this issue has been ignored for so long. The new module looks so incredibly simple compared to the v1 plugin!

It seems like markdown "front matter" is ignored by the new module altogether. Is the idea to make a separate module for parsing front matter that could be run before the Markdown module? If so, I think this is a wonderful idea!

If there was a front matter module that parsed the front matter of a file into variables that are available to modules further down the chain, you could do something like this:

1. Proxy module
2. Front Matter module
3. Markdown Module
4. Template Module

or any combinations of modules. This would be amazingly powerful and solve the problems described in this issue as well as make my caddy-stencil plugin unnecessary.

[mholt]

Thanks Jim -- the v2 markdown module is still incomplete. I was thinking it at least needs a way to configure a header and footer to wrap the markdown content in. That will allow markdown pages to have styling, navigation, etc.

And/or should you be able to provide a single template file into which to render the markdown? It could be made available through a {{.MarkdownBody}} variable or something.

It seems like markdown "front matter" is ignored by the new module altogether. Is the idea to make a separate module for parsing front matter that could be run before the Markdown module? If so, I think this is a wonderful idea!

I hadn't actually thought of this, but it might not be a bad idea.

So the front matter middleware would simply make variables available to templates, no matter which module executes those templates?

[jimjimovich]

About 9 months ago, I really dug into the Templates and Markdown plugins. I mean, I printed them out and marked them up with all kinds of different colors of highlighter just to try to figure out what was going on! I ended up writing my own plugin to do what I needed that was sort of a combo of Templates and Markdown.

The main thing I needed was to be able to take any input (from file or proxy) and run it through templates. I also wanted the ability to parse front matter (and as a side effect and amazing bonus, parse JSON APIs into beautiful templates).

I want to build out a bit of a demo to show off the ideas I came up with, but you might be able to glean some of them from my plugin page https://github.com/jimjimovich/caddy-stencil

I've been thinking about the project again today and realized that it wold be completely unnecessary if there were 3 modules as follows (and in my case, I only need 1 and 3):

1. Front Matter - which could parse JSON, YAML, and TOML front matter on any type of text document, markdown or otherwise, and also parse stand-alone valid JSON (this is very easily done). The parsed data would be sent down the chain of middlewares and made available as template variables.

2. Markdown - This would quite simply, almost exactly as it is right now in the v2 module, parse Markdown and that's all.

3. Templates - Very much like the old templates plugin, this module would allow you to use go templates and could be placed after Markdown or any other module (for example Proxy).

This would give you a very flexible setup where you could use Front Matter (or chose not to), then parse Markdown (or replace it with any other type of markup parser you can dream up, or none at all) then pass the result to the Templates module for even more processing.

I've been using the caddy-stencil plugin I built in production on several sites to do some pretty amazing stuff. The main feature for me is to be able to have site-wide templates into which I can inject data from backend apps, APIs, etc. From the user's perspective, they see what looks like a unified web app, but in the background, Caddy is being used to tie together a lot of backend (micro) services and APIs. (Throw the JWT plugin in there, and you have a unified auth solution for your backend apps!!!)

Making the Front Matter parser a separate module would greatly simplify the Markdown module and make creating new parsers/middlewares extremely simple.

[jimjimovich]

Another, possibly better, approach would be to make the code that deals with front matter a separate go module (that is, not tied to Markdown) so that it could be easily reused in other Caddy Modules. That would remove complexity from the configuration for end users but still allow flexibility and code reuse for Caddy module developers.

[jimjimovich]

I put together this little demo that shows off the power of being able to use templates/markdown/stencil with data from the proxy directive as well as showing off the power of being able to process the front matter (or valid JSON, in this case) with or without markdown. It uses my Stencil plugin (basically a combo of the old Markdown plugin and the old Templates plugin).

https://stencil-demo.starryhope.com/
https://github.com/jimjimovich/stencil-demo

As a bonus, it also shows a nifty way of running Caddy in AWS Lambda with Apex Up (https://up.docs.apex.sh).

[mholt]

That's really cool, Jim. Thanks for putting that together! Does the demo use front matter? I wasn't able to find where that is relevant.

Still, I think this has a lot of potential in Caddy 2. I agree that a separate Go module/package might be best so that it can be used easily in Markdown and Templates, or maybe even as a separate middleware entirely. Do you want to try to hack the demo together in Caddy 2 and see what you like?

[jimjimovich]

When putting together the Stencil plugin, I realized that the Markdown plugin's parser for JSON would parse valid JSON without any actual markdown document body (since it parses JSON front matter). I tweaked the parser a bit to also allow for JSON arrays, but mostly it's the same as in the v1 Markdown plugin.

In the demo's case, it's using "front matter" but not a document body. It could easily have the same JSON with a document body following and be handled by the Markdown plugin (or other plugin).

In the real-world, I have used front matter to pass variables to the Caddy template from a Rails app. The output of the Rails app is html with JSON front matter. This allowed me to take an old Rails app, rip out all the templates, styling, and auth code, then connect it into a unified app that uses Caddy for most of the templating and auth.

At the end of the day, the JSON front matter part of the markdown plugin is really just a JSON parser that allows for a document body after the JSON and places that body into a template variable.

I'm totally willing to work on this, but I'm going to need to dig into the v2 code a bit more to know how to approach things.

Another thing to think about though, is that there's really not that big of a difference between the (v1) Templates plugin and Markdown plugin ... just the ability to parse markdown and front matter. Maybe the Markdown plugin could be combined into the Template plugin with options for rendering markdown and looking for front matter?

[jimjimovich]

When looking at the new Templates module, I see you have some mentions of markdown in there. Would it just be better to merge the Markdown module into Templates?

[mholt]

When looking at the new Templates module, I see you have some mentions of markdown in there.

That's because templates can render certain values as Markdown, but the difference with a separate markdown module is that it can render whole documents that are Markdown.

Would it just be better to merge the Markdown module into Templates?

Hmm, maybe. The question is, what if a document is wholly Markdown? Which styles get applied to it? A header and footer? Not everyone will use front matter or templates...

At the end of the day, the JSON front matter part of the markdown plugin is really just a JSON parser that allows for a document body after the JSON and places that body into a template variable.

This is an interesting idea though: just have a middleware which does this transformation. But then again, it would need to be used in conjunction with a template, so, I see why you might combine them.

Do you want to draft up a proposal for how a combined templates/markdown middleware would look and work in Caddy 2?

[jimjimovich]

Yes, I'll work on something. Where's the best place to do that?

[mholt]

You can open a new issue here, that would be fine! Just describe how it would look in configuration, how it would work functionally, etc.

[mholt]

I suppose I will close this issue then.