Future of static mustache

[agentgt]

Hi @sviperll !

A long time ago I started a similar project for handlebars. I never got it off the ground partly because handlebars is far more complicated (and dynamic). I recently decided I needed a type safe mustache and thus used your project as a starting point.

Our fork is here: https://github.com/snaphop/static-mustache

I have changed the project greatly (while still retaining most of the namespace/structure) and have allowed it to basically implement v1.3 of the Mustache Spec (actually runs a test suite against it) which allows template inheritance among other things. I have also updated the project jdk 17 and allowed more types like Optional and Map to be used.

All is good however there are changes I need to do now that will require a great amount of restructuring, reformatting as well as renaming.

Thus I will be hosting a new repository with a different name. I will retain as much of your copyright notice on the files that still resemble your work and the project will continue to use the BSD3 license.

EDIT: I will leave the old fork for some time if you would prefer hosting/owning your fork and just want to merge my changes into yours.

I will also create an organization that will own the new project that you can join if you would like to participate in continued development with me (and whoever else).

Anyway I want to greatly thank you for providing the excellent skeleton to an excellent idea.

Cheers

-Adam

[sviperll]

Hello Adam,

I'm glad to hear that you've found static-mustache useful. I've actually thought that that was very useful project and was a little disappointed because of the lack of interest from the community.

I think the project was started at the time of Java 6-7 and I'm sure an update to JDK 17 will allow to get rid of lots of crude elements. I think there is a split between XDefinition interface and X class implementing XDefinition interface that is supposed to be fixed with the Java 8 default methods (single X interface with some default methods). And I see that you've already used lots of case-expressions.

Thinking about the project from all this years I have two thoughts to share:

1. I think there is a marketing problem with this project. For people to actually notice something like this you should provide complete documented Spring-intergration. Something like snaphop-static-mustache-spring-boot thing that will automagically configure the thing to work with spring. But such integration contradicts one of the goals of the original project to have errors at compilation time. One of the way forward may be to try to integrate with some new "microservices" frameworks (Quarkus, Helidon, Micranauts). All of them are based on the build-time dependency injection and so, you may potentially generate classes with annotation processor and then have these classes injected to the use-site. I think without some kind of usage guide for major Web-frameworks, there may be no users at all.

2. From the passed years, I still think that the project is very valuable, but it make sense to extend it a little with something like your Formatter interface and have more use-cases, I would personally try to aim at least 3 use-cases:

   1. Generating HTML from some "view"-entities (HTML-escaping is required).
   2. Generating Markdown from some "view"-entities (some Markdown/HTML-escaping is required).
   3. Internationalization, you should probably be able to reference text-strings that can be translations to different languages, this can be implemented as a value-mixins, like .json or .property file that you compile along with the .mustache-template and have it's values as a fallback when there is no such property in the context Java-object. This way you can generate multiple Renderables, each for different language, by providing same .mustache-template, but different .json value-mixins.
   4. Generating complex reporting SQL before executing it (Singular values need to be inserted as parameters to a compiled PreparedStatement and not to be directly "formatted").

   To cover these use-cases, I think the template-compiler should generate some intermediate representation, not a straight-forward code to write text. I was thinking about returning to this project to extend it to cover SQL-case.

Additionally I need to mention that when I've started this, what I wanted is to get as much compile-time checks as possible. The result is that only a limited set of types can be referenced as values. This is quite different from other tools, that usually render everything by just calling .toString(). I think this is a valuable property that needs to be retained for this project to positively differentiate from the competition.

If you plan to create an organization, please write a set of "values" that you want to retain or aim for for the project, so that I or anybody else can agree or disagree before joining.

Lastly, anyway I'm glad that somebody has found this project valuable and can use it. Thank you, Adam, for reaching out and good luck.

[agentgt]

I'll answer just a few questions quickly and try to add more later but the TLDR; is that I agree with all your sentiments.

Some not well organized things:

• The number one goal of the project is type safety above almost all else.
  • Compliance to the mustache spec is a minor secondary goal but not all of that can be achieved
• I too am concerned with the marketing. The new project name and org is jstachio (a word play on mustachio... we could have a pistachio with a mustache as a mascot).
  • Thus I bought the hopefully catchy and short domain name of "jstach.io"
  • And there is now a github org called jstachio.
  • Regardless having "static" anywhere in a module name (module-info) gives a compiler warning.
• It was very painful not having an intermediary AST. I pushed the single pass nature of the original compiler design pretty much to the limit.
  • That being said I only know of one place where I truly need an AST which is for block scoping.

Actually it would be helpful what your thoughts are on inheritance block scoping. The fundamental problem is that parents can use blocks in multiple places and change the context of those blocks.

To get a better understanding here is the spec test:

https://github.com/mustache/spec/blob/5d3b58ea35ae309c40d7a8111bfedc4c5bcd43a6/specs/~inheritance.yml#L228-L237

I have been back and forth on this with @jgonggrijp as I expect "I say apples". The parent shouldn't be able to change the context. That is the block should be evaluated eagerly.

The reason is that parent could do something like this:

{{$block}}{{/block}} {{#nested}}{{$block}}You say {{fruit}}.{{/block}}{{/nested}}

In compliant implementations you would get:

I say apples. I say bananas.

In our implementation because the block is evaluated in the child context we get:

I say apples. I say apples.

From a type safe perspective going with the current interpretation of the spec "block" is now two different types. I could probably implement this dynamic scoping duck/structural typing but when there is a mismatch the errors it will be very very confusing.

[refacktor]

Long-time lurker here. I've been following static-mustache for a long time and salivating over its proposition of strong-typing at compile time, but I have never used it in any of my projects for one reason: the mustache notation requires writing of invalid HTML mark-up. I would much prefer if the tags could be placed as directives inside of HTML attributes, similar to Thymeleaf or Zope TAL, so that the mark-up would remain "valid at rest." I am glad to see the energy around this new fork, and hope this could be an opportunity to revisit the syntax to allow attribute-based directives (the key to enabling always-valid mark-up). I would volunteer as a beta-tester if this goes in that direction!

[agentgt]

If the compiler was better separated from the annotation analysis and code generation then it might be possible however I will say HTML 5 is fairly nontrivial to guarantee valid especially given that HTML 5 is no longer XML like XHTML was. I would be surprised if thymeleaf makes guarantees on valid HTML.

The only library that I know that does this in the Java world is HtmlFlow. I implemented something similar but much simpler a decade ago and given my experience I would not recommend going that route.

However unless thymeleaf makes guarantees I assume you mean problems like:

<ul>
{{#items}} {{/items}}
</ul>

Where you cannot get rid of the outer tag easily if items is empty. Yes that is an annoying problem. Obviously you can do things to fix that depending on implementation like:

{{#items.size}}
<ul>
{{#items}}{{/items}}
</ul>
{{/items.size}}

The other issue is HTML tag attributes where you really do not want the attribute to exist if some condition is not met.

However if an empty attribute means missing then it is possible to have valid HTML if you just embed the logic in the attribute (e.g. the class attribute).

<div class="{{#items.size}}show{{/items.size}}"></div>

Do you have more examples where mustache generates invalid HTML?

In my more than decades real world experience working with others writing mustache/handlebars templates generating valid HTML seems to matter very little compared to other problems like making it render correctly across browsers which validity does not help much (and in some cases you have to write invalid HTML to make it work).

[sviperll]

what your thoughts are on inheritance block scoping

I think this is a question of who control's the scope. I think you imply that child template controls the scope and this seems reasonable, because child "knows better", but the mustache spec says that parent control the scope.

I would say that for child to control the scope should be natural, but mustache has to specify parent as an owner of the scope, because parent has tools to narrow the scope for substituted parts by using conventional mustasche-blocks. Child doesn't have any such tools, this requires either some tools outside specification or some new syntax.

So I would say that it's better to go with mustache spec... or to leave out the feature completely :)

[sviperll]

requires writing of invalid HTML mark-up

@refacktor can you please expand a little what do you mean by invalid HTML mark-up. My personal experience is that mustache allows you to write valid HTML mark-up most of the time, with some caveats that @agentgt described. I feel like maybe you mean something entirely different and I don't understand what do you mean.

[agentgt]

Child doesn't have any such tools, this requires either some tools outside specification or some new syntax.

So I would say that it's better to go with mustache spec... or to leave out the feature completely :)

Yeah possibly a better goal would be to implement special lambdas (and then possibly implement mustache/spec#135 if that every comes out).

See they kind of left an extension hole in the mustache spec because arity 1 lambdas can have almost anything in them and thus a lambda implementation can interpret the contents however it likes. I mentioned that here: mustache/spec#135 (comment)

Actually that is what I was trying to explain to @jgonggrijp is that inheritance could be implemented with lambdas.

The only issue is the spec does not allow lambdas to get the context but many implementations do.

Thus I really should have focused on getting lambdas working. I plan on having an implementation soon.

It looks something like:

public interface LambdaMixin {
    public record Model(String name) {}
    
    @TemplateLambda(template="""
            <b>Hello {{name}}</b>
            """)
    default Model hello(String name) {
        return new Model(name);
    }
}

You add the above interface to your root model object.

@GenerateRenderer("page.mustache")
public record Page() implements LambdaMixin {}

page.mustache

{{#hello}}Adam{{/hello}}

[refacktor]

Thanks for both responses.

As a matter of principle, one reason I like statically typed compiled languages is the ability to detect a large variety of programming errors prior to runtime. Under this principle,

Do you have more examples where mustache generates invalid HTML?

is not the right question to ask. The use-case which violates the principle is when mustache template itself, at rest as source code, is invalid HTML. I'm also extending the concept of "valid" a bit, to say it includes visually reasonable rendering in the browser. This last property is what the Thymeleaf documentation refers to as being "natural", but the term is not universal and there are other frameworks that claim "natural templating" but do not share this characteristic.

Here are two examples:

<tr>
{{#items}}<td> .... </td>{{/items}}
</tr>

The template text between tr and td is not valid. Alternative proposal:

<tr>
<td data-mustache-loop="{{#items/}}"> .... </td>
</tr>

(Thymeleaf goes a step further and defines a "th:" namespace for its attributes. That's fine too)

The other example is when you want dynamic attributes. This is invalid HTML:

<div {{#items.size}}class="show"{{/items.size}}></div>

Something like this, would be valid HTML at rest:

<div data-mustache-attr="{{items.size:class=show}}"></div>

I admit this is completely different from current Mustache syntax. But since there is talk of building an AST, perhaps there could be extensibility for supporting additional parsers.

[agentgt]

To formalize it a little more

public @interface TemplateLambda {
    
    String name() default ""; // make the lambda name different than the physical method name
    String template() default ""; // inline template
    String path() default ""; // resource path to template
    boolean prerender() default false; // not allowed in the spec
}

public OUTPUT lamdaPhysicalName(String textInLamdaBlock, CONTEXT object); 

Where OUTPUT can be:

[String|Renderable|Consumer<Appendable>|SomeObject]

And CONTEXT:

[Current immediate type checked context object | RenderSupport object like original mustache spec]

BTW the original lambda support allowed the lamda to take two arguments. This was retconned but tons of implementations do it.

[agentgt]

@refacktor I will certainly at some point generate an AST as it makes testing much easier. Yes and I did indeed mean at rest and not generate (I'm not sure why I said that).

I don't know if I have the cycles to go off and generate a natural thymeleaf like templating even if uses a similar/same context based AST as the mustache one.

If I did I would use Jericho to parse. I have used it in the past and it is by far the best sort-of valid HTML parser.

Finally mustache is not ideal for the old school design where you have a template designer use an editor to make a design with static HTML. That is not done that much these days. Once the design is done it gets marked up with template logic and thus the developer/designer is no longer creating static HTML. Instead the template designer these days often work with actual data and thus are always seeing templates filled with some level of data. An example even for static tools would be jekyll or hugo... the template author is no longer looking at un processed templates but rather processed ones.

Mustache has a huge advantage in this regard as there are tons of tools to process the template with inputted json or yaml.

Also the Mustache spec had tons of tests that I didn't have to write.

Oh and mustache btw has a plugin in almost every editor/ide. In some very loose ways the problem you are describing is actually more of an IDE or editor problem.

For example your complaint of

<tr>
{{#items}}{{/items}}
</tr>

Of where content shouldn't be there the editor or IDE would be smart enough to know to ignore those tags (and indeed the VS code plugin last time I checked is).

If we create a new syntax we will have no plugins.

[jgonggrijp]

Thanks @agentgt for mentioning me. I am pretty much an outsider to this discussion, but it was still interesting for me to read. I have a few loose remarks, I hope I will not distract you too much from the core discussion.

• It was very painful not having an intermediary AST. I pushed the single pass nature of the original compiler design pretty much to the limit.

I am seriously impressed that you managed to get so much of the spec working, without having an AST.

@refacktor

the mustache notation requires writing of invalid HTML mark-up

There is a solution that has not been mentioned yet in this thread: you can change Mustache's tag delimiters. Mustache has a tag that lets you change the delimiters "live" in the template, which you could hide away in a JavaScript code comment near the top of the template in order to keep the tag invisible and the HTML valid:

<script>
    // {{=<!--\ -->=}}
</script>

Many Mustache implementations also provide a way to override the delimiters by passing an option to the compiler, so you don't need to pull this trick in every individual template. @agentgt can probably tell you whether (and how) this is possible with static mustache/jstachio.

Either way, this would allow you to use HTML comments as Mustache tags:

<!--\ escaped.variable -->
<!--\& unescaped.variable -->
<!--\# section -->
<!--\^ negative.section -->
<!--\/ end.section -->
<!--\! comment (not retained in rendered HTML) -->
<!--\> partial -->
<!--\< parent -->
<!--\$ block -->

Of course, there are some limitations:

• You can place these tags either entirely outside of HTML tags, or entirely inside a quoted attribute; making an attribute="value" conditional as a whole is still not possible (at least not if you want valid HTML).
• You cannot have "normal" HTML comments anymore.

Nevertheless, this is a way to write Mustache templates that are also valid HTML, which is available to you right now and which is even portable across Mustache engines for many platforms.

(Side note: this uses the "change delimiters" feature for roughly the opposite of its intended purpose. Changing delimiters is intended for keeping the Mustache syntax distinct from the target syntax.)

I also subscribe to @agentgt's comment that introducing an entirely new syntax has disadvantages of its own.

@agentgt

Yeah possibly a better goal would be to implement special lambdas (and then possibly implement mustache/spec#135 if that every comes out).

By all means, do not wait for the spec and implement the feature whenever it suits you. Specs should ideally be based on existing practice, rather than the other way round. I already implemented mustache/spec#131, which has not been merged yet. Also, there is at least one implementation (not mine) that already does something close to power lambdas.

BTW the original lambda support allowed the lamda to take two arguments. This was retconned but tons of implementations do it.

I presume you mean this example that appears in the outdated-but-prominent version of the mustache(5) manpage (by the way, an updated version is here):

Template:

{{#wrapped}}
  {{name}} is awesome.
{{/wrapped}}

Hash:

{
  "name": "Willy",
  "wrapped": function() {
    return function(text, render) {
      return "<b>" + render(text) + "</b>"
    }
  }
}

Output:

<b>Willy is awesome.</b>

This behavior was never in the spec, and in that sense, I would not way that it was "retconned". Rather, this is just how the original Ruby implementation did it initially. Lambdas were entirely responsible for the output of their section, so the render argument was needed so that it was still possible to output some form of the original section contents. When people decided to spec the language, they realized this was putting too much responsibility with the lambda. Note that the lambda still could not access the contents of the context.

[agentgt]

@sviperll I added lambda support. I am stunned how well it works.

Any method in the context that has the @TemplateLambda annotation will be considered a lambda.

    @TemplateLambda
    default String listProps(String body, Map<String, String> props) {
        return props.entrySet().stream().map(e -> e.getKey() + " : " + e.getValue())
                .collect(Collectors.joining("\n"));
    }

{{#props}}{{#listProps}}{{/listProps}}{{/props}}

Anyway there are two forms.

The one above where the first argument is the unrendered string body (which would be empty in this case) and the second argument is the context object at the stop of the stack.

The second one is below:

    @TemplateLambda
    SomeModel lambda(ContextObjectOnTopOfStack props) {  
       return new SomeModel("Adam"); // maybe I do something to props as well  
    }
    
    public record SomeModel(String name) {}

{{#context}}{{#lambda}}{{name}}{{/lambda}}{{/context}}

The above would render my name.

The above is essentially inline partials with the lambda producing its own context.

It gets all compiled and type checked.

Now that I have the major features implemented I will move it to the new org and repo. Thanks Again!