Template variable support

[yuri-karadzhov]

Is it possible to add support of template variables like $x (variable symbol can be configurable) to work with shader templates. At the moment parser returns syntax error on symbol $

[xinaesthete]

@yuri-karadzhov Surely your 'template variables' are something that lies outside the scope of standard GLSL? So it would seem a bit out of scope, or I suppose there could be some way of allowing the user to define extensions to the normal grammar... but I think it's out of scope, realistically. Whatever custom pre-processor interprets $x should likely come before glslx.

It's possible that what you're talking about could be done more 'legally' with something like #define statements... however, glslx also doesn't handle those right, which is a bona-fide bug. Strictly speaking maybe that should probably be filed as an issue on this repo, but there's already this evanw/glslx-vscode#1

[yuri-karadzhov]

@xinaesthete Actually as shaders for webgl compiled in run time that could not be preperocessed before glslx.

[xinaesthete]

(you're wrong, you have control at runtime of what happens to the strings you feed to WebGL, which could include some pre-processing, and if you're using $x then it must. All I mean by preprocessor is something that processes or transforms code before passing it on to another process like a compiler. The fact that shaders in webgl are compiled at runtime only if anything introduces another place where a preprocess stage could be present)

[yuri-karadzhov]

Let me explain my thought. Let's say I want to pass $x to my template at runtime (just before shader compilation), but I want to check and minify template during compilation (compilation of web project, not shaders, when shader text turn into template function or string) then there is no way to do it with glslx. Probably I will create yet another webgl parser, because no one seems to be able to support templates.

[xinaesthete]

As I understand it, your pipeline should be something like

[non-standard GLSL code with templates to be substituted]
             |
[template pre-processor]
             |
[standard GLSL code]
             |
 [glslx says it's ok?]
     y /           n\        
 [webgl]     [error message]

Actually, my apologies; so the thing is that you want to serve code which is minified, but checked... I'll update in a sec...

[yuri-karadzhov]

Sure it will like this, but I need to have a template in runtime to compile shaders dynamically.

So on the client side I will have a shader factory, that I can pas template value to and it will generate shader code to me, that I compile and use in a program.

[non-standard GLSL code with templates to be substituted]
             |
 [glslx says it's ok?]
     y /                           n\        
 [optimised templates]     [error message]
             | compilation
 [template function]
             | js runtime
      [glsl code]
             |
       [webgl]

[xinaesthete]

You could always write a post-processor to re-introduce templates to the minified code. It seems to me that it's either that, or the parser needs to have some kind of way of specifying custom extensions to the grammar.

Actually, I seem to remember noticing commit comments along the lines of adding support for #include to glslx, which is fairly widely used but also not part of the glsl standard. Is your template syntax something that you specified yourself, or from some library that you use?

[xinaesthete]

Come to think of it, maybe $x could be legal syntax with an appropriate #define (After a quick go in shadertoy, I don't think so)... one way or another, #define should definitely be supported. I guess doing so with full support for macros etc does rather complicate the parser.

[xinaesthete]

ok... thinking again, I guess you mean templates as in JS template literals ${x}?

[yuri-karadzhov]

It's from my loder for webpack https://github.com/yuri-karadzhov/glsl-template-loader, template $ symbol is configurable. For now I use glsl man, the kinda ignore template variables and parse and check code despite of them, but they can't do any optimization or minification.

Yeah, it seems like it's possible to define a type of $x on top and use all the optimizations.

[yuri-karadzhov]

Yeah, it could be a function or just template string. The transpiller does the job to create one or another. In source code I use $x because it is valid for glslman and some linters (the symbol is configurable thought)

[xinaesthete]

You could maybe keep a record of what all the transformed vars are and transform them back as needed... so on the webpack side you give each substituted template a unique id, which you save as metadata in a dictionary where that id is key and your desired template string is the value, so that you can then on the client side substitute each id back with the template string, and then with the value that you want to substitute at runtime before finally passing to the compiler.

I think my logic is more-or-less right, but that might be hard to parse (for you as a human).

[yuri-karadzhov]

I can transform them to template function with the dictionary immediately after glslx check, but what if glslx inline some vars or change the name during minification? What initial values should I gave this variable to make them wotk with glslx?

I mean let's say I have a template

attribute vec2 a_Position;
attribute vec3 a_Color;

varying vec3 v_Color;

void main(void) {
  v_Color = a_Color * $mult;
  gl_Position = vec4(a_Position, 0.0, 1.0);
}

Loader will transform it to a function that takes a value and returns a shader code like this:

const vCode = vertShader({mult: 0.5});

Where should I move the definition of $mult so glslx will be able to work with it?

[xinaesthete]

This is turning a bit off-topic really, but never mind. I hope it's somewhat helpful, I've spent rather longer thinking about it than I mean to... probably helps myself think about some related stuff anyway...

One approach that could be sound would be to turn $x into something legal like t_x while at the same time inserting (temporary) lines of code with uniform declarations for each of those... you'll have to later delete those from the minified code (should be doable; you'll know what strings you're looking for... could easily be a place for bugs though).

You raise a good point that glslx could inline vars or change names... I had feared that, to quote The Big Lebowski again, "You're entering a world of pain". Although, looking at glslx output, I see that it gives a 'renaming' value similar to the dictionary I described. So that's not such a show-stopper at all.

// --- prepended template declarations, to be deleted when re-substituted(?) -- //
// making them uniforms at least stops them from being inlined...
uniform float t_mult;

attribute vec2 a_Position;
attribute vec3 a_Color;

varying vec3 v_Color;
void main(void) {
  v_Color = a_Color * t_mult;
  gl_Position = vec4(a_Position, 0.0, 1.0);
}

glslx will turn that into

{
  "shaders": [
    {
      "name": "main",
      "contents": "uniform float a;attribute vec2 b;attribute vec3 c;varying vec3 d;void main(){d=c*a,gl_Position=vec4(b,0.,1.);}"
    }
  ],
  "renaming": {
    "t_mult": "a",
    "a_Position": "b",
    "a_Color": "c"
  }
}

Maybe if you make your template symbol be something like t_ in the first place it might marginally reduce the complexity of the implementation...

I'm afraid the devil is in the details and of course I won't be able to give you a full solution... one concern is how you deal with different glsl types... actually, in your example $mult could be float or vec3. Let's suppose that all template substitutions are floats for now (as long as they are in places where floats are legal, they could still mostly be replaceable with other types, ie const vCode = vertShader({mult: 'vec3(0.5)'}); would work, but there'd be plenty of edge cases and it'd be hard to realistically do stuff like that totally safely).

I'm really not intimately familiar with glslx; I just tried it and saw that it choked on #define, making it a bit of a non-starter for me just now.