Merge Markdown Module with Templates Module in v2 #2736

jimjimovich · 2019-09-02T19:15:01Z

1. What would you like to have changed?

I think it would be useful to merge the functionality of the Markdown module with the Templates module.

The v1 Markdown module provides the following features, some of which overlap with the Templates module and others that would be useful for both Markdown and general templates.

It parses Markdown files.
This feature is also available via templates if the template uses the markdown function.
Parsed Front Matter. The v1 Markdown module parsed JSON, YAML and TOML front matter that is often found in Markdown files and provided the results for use in templates. This functionality is useful not only for Markdown files, but also for parsing other types of documents with Front Matter. It is also useful for parsing JSON APIs that do not contain a document "body" but only valid JSON.
Provided default template for Markdown files.
If a configuration did not specify a template, a default template was used. Along with that default template, users could specify a CSS file and a JS file to be inserted into the default template. Providing a default template for Markdown files in a combined Templates module would be easy and re-use the markdown action from the Templates module without the need for an entirely new module.

2. Why is this feature a useful, necessary, and/or important addition to this project?

Combining these two modules would make the resulting Templates module much more powerful, without losing any functionality from the Markdown module. Templates would gain the ability to parse Front Matter and JSON APIs (a side-effect of the JSON Front Matter parsing). It would be easier to maintain one module rather than two.

3. What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

Currently, my work around is maintaining a separate plugin for v1 to have this functionality. It would also be possible to keep the Markdown module similar to v1 but allow the feature to disable Markdown for documents that don't need/contain Markdown syntax.

4. Relevant Issues

#2139

jimjimovich · 2019-09-02T19:27:55Z

Sample configs might look like this

Parse markdown files with default template.

templates [<matcher>] {
        markdown {
		css somecssfile.css
		js somejsfile.js
	},
	mime "text/markdown"
	root /markdownfiles/
}

Include front-matter parsing even if not markdown.

templates [<matcher>] {
	front-matter json/yaml/toml
	mime "text/html"
	root /myproxiedpath/
}

mholt · 2019-09-10T04:01:39Z

Hey Jim! I did read this, just so you know -- I haven't been able to focus on it recently but will try to get around to commenting this week. I need to think about it. I'm glad you filed a detailed issue.

mholt · 2019-09-19T21:32:21Z

I think my primary question with this approach is how does the handler know whether to parse and execute the file as markdown, or as a template?

Can you explain precisely what this does:

templates [<matcher>] {
        markdown {
		css somecssfile.css
		js somejsfile.js
	},
	mime "text/markdown"
	root /markdownfiles/
}

And what are example files it would handle?

jimjimovich · 2019-09-21T17:00:02Z

Matt, Thanks for looking into this! The config you questioned above is essentially the same as the sample Markdown config from v1. The main difference is the use of mime types instead of file extensions. In the actual code (at least in v1), the Templates plugin looked at mime codes instead of file extensions because it could receive input from the file system or from something like Proxy. I don't know why I put mime in this example config instead of ext. It could just as easily be "ext .md .markdown" and then have default mime types for each of those extensions (that's what v1 does).

I can't say that I completely understand the new format for config files, but what I was trying to show is probably something more like this.

templates [<matcher>] {
        markdown {
		css somecssfile.css
		js somejsfile.js
                ext .md .markdown
	},
	root /markdownfiles/
}

Basically, it seems to me that the Markdown and Template modules are doing several things that are repetitive and that the same Markdown functionality from v1 could be integrated into Templates. As a side effect, Templates could take advantage of things like Frontmatter, which would be GREAT!

mholt · 2019-09-21T17:05:22Z

Remember that in v2, request matching (the process of deciding whether a handler will execute based on the request) is done using matchers now, not the handlers themselves. So giving the handlers a list of file extensions is probably not useful/necessary and just makes things needlessly complicated. If a request should be served by the templates/markdown handler, the matcher should decide it.

I believe though that the templates handler conditionally executes based on the Content-Type of the response in any case, to allow for more liberal matchers that might include images, etc, for ease of configuration.

jimjimovich · 2019-09-21T21:37:00Z

I really need to find time to play with v2 configs!

I believe though that the templates handler conditionally executes based on the Content-Type of the response in any case, to allow for more liberal matchers that might include images, etc, for ease of configuration.

Yes, this is powerful! I remember this from the v1 Templates code. It's one of the things that seems to me would make it super easy to use Templates for several types of content in a single directory.

mholt · 2019-09-22T17:52:12Z

Yeah so we basically need a way for the templates handler to decide whether to render the whole document as markdown (first?). Can a .md file be rendered as both markdown and templates? If so, which first?

Or should two separate templates handlers be configured using matchers to decide which handler should be executed, whether it be the templates or the markdown configuration?

mholt · 2019-12-23T19:35:08Z

@jimjimovich I've made progress on this, as a similar feature is needed for the Caddy 2 docs site.

So far, the way it works is you use the templates module. For our new docs site, I use a rewrite so that any requests in a certain docs path get served by an HTML file that is templated.

Then, in the top of the templated file, I do something like this:

{{$pathParts := splitList "/" .OriginalReq.URL.Path}}
{{$markdownFilename := default "index" (slice $pathParts 2 | join "/")}}
{{$markdownFilePath := printf "/docs/markdown/%s.md" $markdownFilename}}
{{$markdownFile := (include $markdownFilePath | splitFrontMatter)}}
<!DOCTYPE html>
<html>
...
	<title>{{default $markdownFilename $markdownFile.Meta.title}} &mdash; Caddy Documentation</title>
...
	<article>
		{{markdown $markdownFile.Body}}
	</article>
...
</html>

Make sense? What do you think?

(I've updated the template action docs here: https://github.com/caddyserver/caddy/wiki/v2:-Templates)

jimjimovich · 2019-12-23T19:46:01Z

@mholt I love it! So, basically, you can get access to the markdown file's front matter right inside the template? Would this work if the markdown (with front matter) is coming from any source (proxy, for example)? If it also works with "markdown" with no body (just front matter, like json, for example), it'd be even more flexible and amazing.

Thanks for all your amazing work on this! It really turns Caddy into much more than a web server and more of a development platform.

mholt · 2019-12-23T19:53:33Z

So, basically, you can get access to the markdown file's front matter right inside the template?

Yep.

Would this work if the markdown (with front matter) is coming from any source (proxy, for example)?

Yes. Templates will execute the same no matter whether the file came from a reverse proxy or a static file, etc.

If it also works with "markdown" with no body (just front matter, like json, for example), it'd be even more flexible and amazing.

Umm I'm not sure about this, but you basically just want your template to be able to parse JSON? Does it need to initiate the HTTP request to obtain the JSON too?

jimjimovich · 2019-12-23T19:57:05Z

If it also works with "markdown" with no body (just front matter, like json, for example), it'd be even more flexible and amazing.

Umm I'm not sure about this, but you basically just want your template to be able to parse JSON? Does it need to initiate the HTTP request to obtain the JSON too?

Yeah, it'd be nice to be able to get the JSON (or other front matter) from a proxy request to a backend service. The only real difference between a JSON file (unless it's a JSON array) and a markdown file with JSON front matter is the lack of a body on the JSON file. I've found this approach very useful for consuming JSON APIs on the server side and delivering HTML to the client after passing it through the template.

mholt · 2019-12-23T20:45:17Z

@jimjimovich Thanks for sponsoring! 💚

Can you give a specific example of this behavior with a JSON API? I don't quite have a firm grasp of it yet.

jimjimovich · 2019-12-23T20:53:23Z

@mholt No problem, wish I could do more!

I put together this example some time ago of using a weather API. Here's the demo and source code.

https://stencil-demo.starryhope.com/json/
https://github.com/jimjimovich/stencil-demo

It was using my stencil plugin, which it seems like your new Templates work should obsolete.
https://github.com/jimjimovich/caddy-stencil

mholt · 2019-12-23T21:04:37Z

Thanks, I will look at it soon. Happy holidays!

jimjimovich · 2019-12-27T19:16:45Z

Does it need to initiate the HTTP request to obtain the JSON too?

To answer this more correctly. No. The HTTP request in my example is handled by the Proxy directive.

jimjimovich · 2019-12-30T19:52:13Z

{{$markdownFile := (include $markdownFilePath | splitFrontMatter)}}

I just realized how powerful this is! You could use this to pull in JSON data that is on the server and dynamically generate content. This gives me so many ideas! Thanks!

jimjimovich · 2020-01-03T15:40:43Z

@mholt I was trying to put together the above demo using v2 templates and realized something that I guess has been escaping me since we started this discussion. The v1 markdown directive allowed you to set a default template into which the files were rendered. When I wrote caddy-stencil, I took this idea and used it as well. This allowed me to set a default template to use for rendering whatever comes from the file server or proxy directives. Would it be possible to add something like this to v2 Templates? Or perhaps this is already possible with some fancy configuration in v2?

mholt · 2020-01-03T16:36:59Z

@jimjimovich Default in case of what, though? The template handler gets its source from upstream the chain, like file server or reverse proxy. It's not hard-coded, so the lack of a hard-coded template file nixes the need for a default template file. Can you be more specific then? (I've looked through the stencil repo and your examples, which are really nice, but from my perspective it's difficult to put all the pieces together in terms of what needs to happen for v2 -- also I have a cold so I'm probably not thinking top-notch lately)

jimjimovich · 2020-01-03T17:00:27Z

@mholt In terms of Caddy v1, the default template was provided and used to render the markdown file that was returned from the file system. We had worked to rewrite the markdown module to use proxy or any other source as input instead of it being hard coded to the file system (similarly to the way Templates was upgraded in v1).

Stencil used a lot of Markdown for things like parsing the frontmatter, but it also took the input from proxy or file server and put it into the template (essentially, it was my reworked version of Markdown that never got accepted with actual markdown parsing removed since I didn't need that). The key is that it did essentially what the new splitFrontMatter does and made those variables available in the template that was defined.

In trying to recreate this in v2, it might look something like this:
source (file server, proxy) -> templates with default template where $source is a variable that can be used with splitFrontMatter or other template functions.

jimjimovich · 2020-01-03T17:06:22Z

Perhaps it's easier to think about it as a parent or master template.

mholt · 2020-01-03T18:38:49Z

Okay, so letsee...

source (file server, proxy) -> templates with default template

In which cases would the default template kick in, though?

jimjimovich · 2020-01-03T18:43:34Z

Okay, so letsee...

source (file server, proxy) -> templates with default template

In which cases would the default template kick in, though?

The default/master/parent template would be used if it was passed as part of the config. Otherwise, it would parse the document as normal. Allowing for this would also allow all of the functionality of the old Markdown module to be implemented easily using the Templates module.

jimjimovich · 2020-01-03T18:51:52Z

I did some research and the idea came to mind that if there was a default template, perhaps the source (from file server or reverse proxy) could be added as a second template called something like "source."

That'd allow you to have a default template that has your navigation, etc. and then include {{ template "source" }} where you want the content that is coming from your file or proxy.

mholt · 2020-01-03T18:55:10Z

I just rewrite all requests in a certain subpath to a specific file, and that becomes my template. Is that sufficient?

templates
rewrite /docs/* /docs/index.html # <-- has the "default" template
reverse_proxy /api/* localhost:4000

jimjimovich · 2020-01-03T19:02:44Z

I just rewrite all requests in a certain subpath to a specific file, and that becomes my template. Is that sufficient?
templates
rewrite /docs/* /docs/index.html # <-- has the "default" template
reverse_proxy /api/* localhost:4000

That's fine if you're only using local files, like in your example from several days ago. In that example, you're doing some templating magic to access the markdown files that are on your server based on the url path.

jimjimovich · 2020-01-03T19:05:08Z

I suppose a similar thing could be done with reverse_proxy and then using httpInclude in the "default" template ... I'll try messing around with that.

jimjimovich · 2020-01-04T02:46:46Z

@mholt I've done some testing and it seems like your rewrite trick might work in conjunction with httpInclude in the template. I've put together a very simple demo config and site.

https://github.com/jimjimovich/caddy-json-demo

Here, you can see that it's possible to proxy the www.metaweather.com api (at this point, it's hard coded in the template to a specific address).
https://caddy-json-demo.starryhope.com/weather/

However, because most APIs are minified and therefore on one single line, splitFrontMatter doesn't work here. It also wouldn't work for JSON arrays (that start and end with brackets [ ]).

Would you be willing to add a parseJSON function to Templates? It should be rather easy to implement and I would be willing to help (would most likely need help with tests). This would allow making very nice templates for JSON APIs and would be extremely useful.

mholt · 2020-01-04T20:03:52Z

@jimjimovich Thanks for the update, glad that is workable.

Would you be willing to add a parseJSON function to Templates?

Here you go: Masterminds/sprig#223

jimjimovich · 2020-01-05T01:02:40Z

@mholt Thanks! Seriously, you're amazing! That should be very useful!

gonzaloserrano · 2020-02-10T21:30:04Z

Sorry to disturb, but does anyone around here have a simple Caddyfile that's able to render a markdown file? Thanks!

mholt · 2020-02-10T22:08:08Z

@gonzaloserrano

localhost

templates
file_server

In the future, please ask questions in the forum, as this issue is now about the parseJSON template function being implemented.

jimjimovich · 2020-02-24T01:14:43Z

@mholt What do you think the chances of parseJSON making it into Caddy 2 are? It seems there isn't any movement on the sprig pull request. I have a project I'd love to use this for!

mholt · 2020-02-24T03:27:47Z

Yeah I dunno, you'll have to bug the maintainers I guess!

jimjimovich added the feature ⚙️ New feature or request label Sep 2, 2019

mholt added the v2 label Sep 10, 2019

mholt mentioned this issue Sep 19, 2019

v2: Caddyfile markdown directive #2757

Closed

mholt mentioned this issue Nov 4, 2019

v2 Markdown Testing and Feedback #2861

Closed

mholt mentioned this issue Jan 4, 2020

Add fromJson and mustFromJson funcs Masterminds/sprig#223

Merged

mholt closed this as completed Mar 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge Markdown Module with Templates Module in v2 #2736

Merge Markdown Module with Templates Module in v2 #2736

jimjimovich commented Sep 2, 2019

jimjimovich commented Sep 2, 2019

mholt commented Sep 10, 2019 •

edited

Loading

mholt commented Sep 19, 2019

jimjimovich commented Sep 21, 2019

mholt commented Sep 21, 2019

jimjimovich commented Sep 21, 2019

mholt commented Sep 22, 2019

mholt commented Dec 23, 2019 •

edited

Loading

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 27, 2019

jimjimovich commented Dec 30, 2019

jimjimovich commented Jan 3, 2020 •

edited

Loading

mholt commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

mholt commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

mholt commented Jan 3, 2020 •

edited

Loading

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 4, 2020

mholt commented Jan 4, 2020

jimjimovich commented Jan 5, 2020

gonzaloserrano commented Feb 10, 2020

mholt commented Feb 10, 2020 •

edited

Loading

jimjimovich commented Feb 24, 2020

mholt commented Feb 24, 2020

Merge Markdown Module with Templates Module in v2 #2736

Merge Markdown Module with Templates Module in v2 #2736

Comments

jimjimovich commented Sep 2, 2019

1. What would you like to have changed?

2. Why is this feature a useful, necessary, and/or important addition to this project?

3. What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

4. Relevant Issues

jimjimovich commented Sep 2, 2019

Sample configs might look like this

mholt commented Sep 10, 2019 • edited Loading

mholt commented Sep 19, 2019

jimjimovich commented Sep 21, 2019

mholt commented Sep 21, 2019

jimjimovich commented Sep 21, 2019

mholt commented Sep 22, 2019

mholt commented Dec 23, 2019 • edited Loading

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 23, 2019

mholt commented Dec 23, 2019

jimjimovich commented Dec 27, 2019

jimjimovich commented Dec 30, 2019

jimjimovich commented Jan 3, 2020 • edited Loading

mholt commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

mholt commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

mholt commented Jan 3, 2020 • edited Loading

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 3, 2020

jimjimovich commented Jan 4, 2020

mholt commented Jan 4, 2020

jimjimovich commented Jan 5, 2020

gonzaloserrano commented Feb 10, 2020

mholt commented Feb 10, 2020 • edited Loading

jimjimovich commented Feb 24, 2020

mholt commented Feb 24, 2020

mholt commented Sep 10, 2019 •

edited

Loading

mholt commented Dec 23, 2019 •

edited

Loading

jimjimovich commented Jan 3, 2020 •

edited

Loading

mholt commented Jan 3, 2020 •

edited

Loading

mholt commented Feb 10, 2020 •

edited

Loading