Allow passing custom vertex data to canvas_item shaders using a custom vertex array

[Giwayume]

Describe the project you are working on

Complex 2D shaders that need a separate interpolation layer from UV (as UV is already in-use for texture mapping) for calculating implicit functions across the face of a triangle.

Describe the problem or limitation you are having in your project

In Godot 3, I could use vertex colors in canvas item shaders as uncompressed float values that are interpolated across a triangle.

Now, in Godot 4, vertex colors are forced to be 8 bit integers, and there is no replacement for this kind of extra data for canvas item shaders.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Allow at least one of the 4 CUSTOM0-3 arrays to be accessed from canvas item shaders.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

https://docs.godotengine.org/en/4.0/tutorials/shaders/shader_reference/spatial_shader.html

https://docs.godotengine.org/en/4.0/tutorials/shaders/shader_reference/canvas_item_shader.html

Add CUSTOM0 to vertex shader of canvas item.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No workarounds.

Is there a reason why this should be core and not an add-on in the asset library?

It cannot be done as an add-on.

[bluenote10]

I had raised something similar before, but it has been re-interpreted as back-porting Godot 4 behavior to Godot 3. I really wish there was a better solution in Godot 4. The 4 CUSTOM0-3 arrays feel hacky, and are also limited in type.

[Calinou]

The way mesh rendering is implemented in 3D is very different from 2D (especially for sprites or polygons), so the implementation can't just be copy-pasted from 3D to 2D. You would also need a way to define the custom vertex data somehow.

I had raised something similar before, but it has been re-interpreted as back-porting Godot 4 behavior to Godot 3. I really wish there was a better solution in Godot 4. The 4 CUSTOM0-3 arrays feel hacky, and are also limited in type.

This is an unrelated subject. Please open a separate proposal for this, but note that what you're requesting may not be feasible or desired. Using an unified name for custom data fields makes the shader compiler and autocompletion's job easier.

Adding support for more datatypes to the custom array data should be tracked in its own proposal.

[Giwayume]

The way mesh rendering is implemented in 3D is very different from 2D (especially for sprites or polygons), so the implementation can't just be copy-pasted from 3D to 2D. You would also need a way to define the custom vertex data somehow.

MeshInstance2D + ArrayMesh already allows defining The custom properties, my understanding is the CanvasItem shader just needs them to be passed. I'm testing this out.

I'm not interested right now in Sprite2D or Polygon2D classes. I'm just trying to open up a way for plugins to be able to use this data in a 2D rendering context (MeshInstance2D). Sprite2D and Polygon2D UI implementations can come later.

[clayjohn]

This shouldn't be too hard to implement in the RD renderer as a lot of the mesh management is shared between 2D and 3D.

Where this will be difficult is in the gl_compatibility backend where all the mesh attributes are already used. The way batching works in the gl_compatibility renderer is that we pass the per-instance data in as per-instance mesh attributes. You would need to carve out a path where a fallback is used instead to free up enough attribute slots

[Giwayume]

You're right RD is a fairly simple change, but digging through all this complex rendering code that I have not seen before, is not so simple.

func _ready():
	var mesh = ArrayMesh.new()
	var surface = []
	surface.resize(ArrayMesh.ARRAY_MAX)
	var vertices = PackedVector2Array()
	var custom0 = PackedFloat32Array()
	
	vertices.push_back(Vector2(0.0, 0.0))
	custom0.append_array([0.0, 0.0, 0.0, 0.0])
	vertices.push_back(Vector2(400.0, 0.0))
	custom0.append_array([1.0, 0.0, 0.0, 0.0])
	vertices.push_back(Vector2(0.0, 400.0))
	custom0.append_array([0.0, 1.0, 0.0, 0.0])
	
	surface[ArrayMesh.ARRAY_VERTEX] = vertices
	surface[ArrayMesh.ARRAY_CUSTOM0] = custom0
	mesh.add_surface_from_arrays(Mesh.PRIMITIVE_TRIANGLES, surface, [], {}, Mesh.ARRAY_CUSTOM_RGBA_FLOAT << Mesh.ARRAY_FORMAT_CUSTOM0_SHIFT)
	
	var mesh_instance = MeshInstance2D.new()
	mesh_instance.mesh = mesh
	mesh_instance.material = ShaderMaterial.new()
	mesh_instance.material.shader = custom_shader
	add_child(mesh_instance)

Got it working for RD

[Giwayume]

@clayjohn

all the mesh attributes are already used.

GL_MAX_VARYING_COMPONENTS is supposed to be a minimum of 60, and the minimum number of layout slots is supposed to be 1/4 of that number, so 15. It looks like slot 14 is unused in canvas.glsl. Am I wrong in saying we have 1 free slot to meet minimum spec?

Would it be an acceptable limitation to say Vulkan gets CUSTOM0-3 and GL gets CUSTOM0?

Or we could say MultiMesh/Particles don't get access to custom vertex attributes in GL and reuse those 3 slots?

[Giwayume]

If I do something like this in the gl3 shader,

#ifdef USE_INSTANCING

layout(location = 1) in highp vec4 instance_xform0;
layout(location = 2) in highp vec4 instance_xform1;
layout(location = 5) in highp uvec4 instance_color_custom_data; // Color packed into xy, custom_data packed into zw for compatibility with 3D

#else

#if defined(CUSTOM0_USED)
layout(location = 1) in highp vec4 custom0_attrib;
#endif

#if defined(CUSTOM1_USED)
layout(location = 2) in highp vec4 custom1_attrib;
#endif

#if defined(CUSTOM2_USED)
layout(location = 5) in highp vec4 custom2_attrib;
#endif

#endif // USE_INSTANCING

That means MeshStorage::_mesh_surface_generate_version_for_input_mask would need to be passed either an indicator of whether or not it should remap based on whether or not it's a multi-mesh instance, or a map of old to new indices. But since the order changes, that also changes the stride/offset values. Seems like a very messy solution.

Whereas if I try to use the new "standard" locations for mesh custom data,

layout(location = 6) in highp vec4 attrib_A;
layout(location = 7) in highp vec4 attrib_B;
layout(location = 8) in highp vec4 attrib_C;
layout(location = 9) in highp vec4 attrib_D;

This is populated by the RasterizerCanvasGLES3::_enable_attributes function. It seems this InstanceData struct is required for regular MeshInstance2D display. Moving these attributes could also be very messy since they are also populated in a sequential order.

Assuming I can make use of layout(location = 14), theoretically all of those slots can shift down one and make room for a CUSTOM0 vec4 in gles3. Which would be a fairly clean solution. It meets my needs, but obviously not as great on the GLES3 side for shader customization.

Any thoughts?