Syntax with_prototype recursion issue

[scuml]

Description of the bug

Using

    - match: ""
      push: "Packages/HTML/HTML.sublime-syntax"
      with_prototype:

as recommended in the docs is now throwing a

Apparent recursion within a with_prototype action: 25000 context sanity limit hit error.

Steps to reproduce

Start Sublime
Use Djaneiro Plugin
Error appears in html file.

Expected behavior

No error

Actual behavior

Apparent recursion within a with_prototype action: 25000 context sanity limit hit error.

Sublime Text build number

4186

Operating system & version

macOS 14.7.1

(Linux) Desktop environment and/or window manager

No response

Additional information

No response

OpenGL context information

No response

[deathaxe]

Docs are out-dated.

This strategy to extend existing syntax definitions is out-dated and has been replaced by syntax inheritance (via extends: keyword) for ST4 exactly for the mentioned reason in OP.

The way push...with_prototype works has limitations with regards to detecting recursions and might easily hit those sanity limits even without them. Demand on accuracy and features of syntax definitions, which do not just highlighting but also need to drive all sorts of other features in ST has grown far beyond of what push...with_prototype machinary can handle.

To learn more about that, check out packages like Astro, Jinja2, Laravel Blade Highlighter, Liquid/Jeykyll, MDX, Mustache, Ngx HTML, Twig, Svelte, Vue all of which using syntax inheritance.

Even builtin syntax definitions such as ASP, ERB, Go Templates, JSP, PHP have been updated to use inheritance.

Other examples are Less, SCSS (Sass package) and PostCSS, all of which extend ST's CSS.

[deathaxe]

Otherwise duplicate of #4061.

[deathaxe]

With regards to Djaneiro:

Jinja2 and Django syntax definitions should be covered by Jinja2 package.

Same applies for Django Syntax package.

[scuml]

Took a look at those packages and still struggling with how to override previous contexts without with_prototype. I've got mismatched double-quote errors and django comments placed in javascript blocks that I can't get right.

Reference file is here: https://github.com/squ1b3r/Djaneiro/blob/304a85adcf9320c54b121623784b3a576f4ec4b3/Syntaxes/HTML%20(Django).sublime-syntax

[deathaxe]

One major difference between with_prototype and extend (inheriting) is:

with_prototype recursively injecting patterns into ALL contexts of a pushed syntax without control about which ones are extended. This may cause those patterns be injected in unwanted locations, causing syntaxes to break or at least bloats whole compiled syntax massively.

Extending HTML just puts those patterns into the extended syntax, but not in one of its "children" recursively. Each such child syntax needs to be extended from its base to add desired additions (template tags). Unfortunatelly this requires a bunch of boilerplate code at the moment. See HTML (Jinja) for instance.

We still bet and hope for #3787 to be able to avoid that by just providing some scope variable to be replaced by extending syntaxes.

---

That said,

... as Jinj2 package already provides good and up-to-date syntax definitions, I wonder why re-invent the wheel? What does Djaneiro's syntax offer beyound Jinja2? I had a look at Django Syntax, which doesn't do except for the {% comment %} block, which is easy to also add to Jinja2.

Wouldn't it be more effective to maintain a single syntax definition and re-use it?

[LoneBoco]

I've been running into this issue with LoneBoco/razor-sublime#3. It looks like the recursion protection introduced in build 4075 had a regression in build 4181. The last working build was 4178.

[LoneBoco]

@deathaxe Should I create a new issue for the regression in the recursion protection? It seems this issue is focusing more on documentation.

[deathaxe]

That's not a regression of ST. We just hit the limits of its original recursion protection capabilities by adding HTML syntax highlighting support to JavaScript template strings.

see: sublimehq/Packages#4019

The way how HTML is extended using with_prototype causes all embed statements to be converted in push...with_prototype internally.

ST seems to be not able to detect recursions of such converted statements.

However, even if it was, the way with_prototype works causes huge syntax trees and is for various other reasons no longer an appropriate way to extend complex syntaxes such as HTML, which include/embed other complex syntaxes like JS/CSS/... , because many of them push/set/pop many small contexts which often rely on nothing uncontrolled being injected.

It would at least require with_prototype to respect meta_include_prototype to get it right. But as that was a braeking change, it's unlikely to happen.

I can only recommend to follow the already long list of template syntaxes by extending HTML. It requires a bit more boilerblate, sure, but it has various other advantages.

We've for instance reduced PHP's size by about 70-75% and significantly reduced its compile time and increased parsing performance heavily using that new approach.

[LoneBoco]

The problem is that I can't use embed in C# Razor. HTML can include C# and C# can include HTML. You can just switch between syntax at will. Many years ago I had requested a feature to limit how many recursions occur (like a token that counts how many times it gets hit and stops after a set number of includes), but instead build 4075 included recursion detection so that wasn't needed.

[scuml]

Thanks for your help @deathaxe

The Djaneiro package goes beyond just the django templating language and include python snippets that I've memorized the shortcuts for, as well as a URL indexer that autocompletes {% url %} view names that is a giant productivity booster.

[deathaxe]

Sure, I saw its extends.

The question was about how to avoid duplicate Jinja syntax definitions.