Add hint_layer_texture for screen-reading shaders

[eddieataberk]

Describe the project you are working on

A 3D game with complex shaders.

Describe the problem or limitation you are having in your project

It is not possible to limit screen reading shaders to affect only certain objects.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add a sampler hint to read the visual instance layers.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

smilar to other sceen texture hints such as hint_screen_texture, hint_depth_texture, and hint_normal_roughness_texture add a new sampler hint called hint_layer_texture reading the visual instance layer of the objects behind the shader material. then we can access to this texture with a code like this:

uniform sampler2D layer_texture : hint_layer_texture, repeat_disable, filter_nearest;

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

This is about core shaders.

[Calinou]

This would require rendering the entire scene another time to get this layer texture, so I don't see how this can be viable from a performance standpoint. screen_texture and depth_texture just retrieve what was rendered this frame – they don't compute anything specifically for it, other than the mipmaps (if requested).

[eddieataberk]

This can be a setting under the "Rendering->Textures->Render Layer Texture". And only the people needed it can enable it without causing any performance hindrance to the rest of the users. Especially for us moved from other engines this would be a great addition. Stencil Texture had been the feature I missed the most from Unreal Engine.

[Calinou]

This can be a setting under the "Rendering->Textures->Render Layer Texture". And only the people needed it can enable it without causing any performance hindrance to the rest of the users.

Even if you expose a setting, it takes a lot of time to render the same scene twice instead of just once. This would roughly mean that if your system can hold 60 FPS in a given situation, it would barely manage to reach 30 FPS after enabling this.

In practice, I expect the added render time to make for a slowdown between 1.8× and 2×. This is too expensive for most users to consider using just for a single special effect. If you have extra CPU/GPU time available, you can do a lot of other things with it, such as increasing resolution, graphics settings, using better antialiasing and so on.

Stencil Texture had been the feature I missed the most from Unreal Engine.

For this, #7174 is a more suitable solution.