Expose an intuitive subset of stencil operations

[QbieShay]

Describe the project you are working on

Godot engine

Describe the problem or limitation you are having in your project

1. No stencil buffer support which makes a number of important effects impossible to achieve without waste of GPU resources
2. Stencil API is extremely convoluted, counterintuitive and brain-breaking to understand
3. People requested (5 years now) stencil support. People need stencil.

After gathering feedback in #3373
After testing with godotengine/godot#78542
After a significant amount of hours spent by clay and myself to wrap our head around this

Considering we don't want to expose the full complexity of stencil outside of lower level API (more on this in the future)
Considering we don't want to 100% copy any other engine in their implementation (expect tutorials from Unity to not apply)

Read along for Godot's stencil proposal ^^
We've tried to think of the use cases brought up in #3373 and they all seem possible with this API.

This will be an iterative process and more features will come little by little. This is huge work. There's a lot of things to consider. Please be patient

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Supersedes #3373

Stencil operations are needed for a wide range of 3D FX. They can also be very useful to optimize a lot of other 3D FX.

Currently, Godot allocates a stencil buffer when allocating the 3D renderbuffers (combined depth + stencil), but does not make use of it.

Stencil operations as exposed by the OpenGL and Vulkan standards are cumbersome, confusing, and exposes a lot of meaningless combinations of similar-sounding settings. We want to implement something that can be used by most users and not just by users with a background in advanced tech art or 3D graphics APIs.

At the same time, there is a huge demand to have some control over stencil operations to do more than just the most common effects (masking and xray).

Masking works like the depth buffer. In other words, at some point in time you render to the stencil buffer and place a value. At another time (or at the same time) you read from the stencil buffer, compare the value and reject based on a set condition (equals, not equals, greater than, etc.). Masking allows users to implement outlines, complex masks, and performance optimizations.

Xray allows users to add overlays that persist through scene geometry. I.e. show an enemy through a wall etc. Xray requires writing a stencil value at one point (i.e. when you first draw the enemy), but only when the depth test passes. In a later pass the "xray" effect is drawn and rejects any pixels that match the xray stencil value.

The important difference between the two is that Masking writes regardless of the depth test.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Shader API

Similar to render_mode, add the keyword stencil_mode

Mode (enum):

WRITE_DEPTH_FAIL // stencil value is only written if the depth test fails

READ_ONLY // stencil value is never written, but it is tested // must pass stencil and depth // depth testing can be disabled in the depth settings. Stencil testing can be disabled by not using stencil

READ_WRITE // stencil value is written if ref passes stencil test and depth test

WRITE_ONLY // stencil value is written if it passes depth test

DISABLED

Compare mode (enum):

COMPARE_LESS

COMPARE_EQUAL

COMPARE_LESS_OR_EQUAL

COMPARE_GREATER

COMPARE_NOT_EQUAL

COMPARE_GREATER_OR_EQUAL

Ref (an integer 0-255)

These could be implemented as rendering modes or with a new token stencil_mode i.e.:

stencil_mode read_only, compare_less, 1;

Shader

The stencil value should be exposed in the fragment shader by reading from the stencil buffer with a STENCIL_ID keyword.

Pipeline details

There's a lot of work needed to make this fully usable not only in term of exposing the relevant keywords to shaders, but to also make sure that those operations are placed in a sensible manner in the rendering pipeline. For example WRITE_DEPTH_FAIL cannot be run at the depth prepass. Similarly, stencil operations need to run in a consistent manner and to run either in depth prepass or in subsequent passes, but ideally not both, which causes effects to misbehave.

Considerations on how depth sorting buckets are managed for opaque objects must be taken into consideration, possibly supporting appropriate ordering of subsequent draw passes specified via the "next pass API" which currently doesn't seem to sort objects reliably, causing flickering and completely breaking stencil operation.

Tentative step-by-step

1. First implementation will add the above keywords. It will probably be very finnicky on the render ordering and require to be run in some combination of transparent and opaque. How to solve this will be decided in the future once enough feedback on this work is gathered. Solving all the ordering usecases is out of scope for step one. Step one should grant at least the possibility to create
2. masks and impossible geometry (like antichamber's room)
3. outlines
4. xray shaders
   this is the absolute minimal subset of features that should reliably work out of the box. Windwaker style lights have been tested but they don't work reliably due to sorting issues on draw passes mentioned above. they can be however achieved just by putting separate nodes with the same sphere and using sorting offset.
   This needs to be iterative work. There will be back and forth.
5. second pass should address sorting issues and any usecase that's isn't covered in first pass and add presets:
6. StandardMaterial3D

We want users to be able to implement stencil outlines and stencil Xray effects without understanding how stencils work (the same way that users can implement transparent effects without understanding all the depth settings). Accordingly, we propose to expose a high level API in the StandardMaterial3D that makes these much easier.

stencil_mode (enum):

OUTLINE. When selected, exposes:

color

thickness

ID

Internally this would:

set mode to WRITE_ONLY

set ref to ID

Add a next_pass with a basic opaque material set to unshaded and with albedo_color set to the specified color and with the grow property set to thickness and with stencil mode set to READ_ONLY, compare mode set to NOT_EQUAL and ref set to ID

XRAY. When selected, exposes:

color

ID

Internally this would:

Set mode to WRITE_DEPTH_FAIL

set ref to ID

Add a next_pass with a basic transparent material set to unshaded and with albedo_color set to the specified color and with stencil mode set to READ_ONLY, compare mode set to NOT_EQUAL and ref set to ID

CUSTOM. When selected, exposes:

read_write_mode,

compare_mode (only visible when read_write_mode includes reading), and

ref

In parallel stencil explorations for 2D can be made. I have no idea how to do that or even if it's possible. I have exhausted my energy on this so proposals for 2D stencil are welcome. Feel free to experiment with the original PR I opened to then make a proposal!

Alternatives

To the people inevitably unsatisfied with this solution: we know. We hear you. We have decided to not compromise entirely on usability for this work, but it doesn't mean that it will be forever this way. There's the intention to extend the rendering API and to eventually offer low level access to the renderer and to the whole stencil API via the lower level interface. This is out of scope for this initial work. This proposal is a weighted compromise between:

1. maintaining the user friendliness which is at the core of Godot's development
2. keeping the work manageable and the maintenance cost contained: exposing everything now would make it much harder to tweak things in the future. API cannot be unexposed once exposed.
3. getting some form of support for stencil finally in the engine with an amount of work that's feasible

Conclusion

Thank you for reading until here if you have, and thanks to everyone that participated to this discussion and provided their input. Special thanks to @apples whose initial work made it possible at all to finally reach a consensus on how we want this API to look like.

Thank you all <3

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

No

[QbieShay]

It would be super useful and great if someone picks up this work code-wise. Ideally we'd be able to test WIP builds and give feedback in an iterative manner with the community. I do not have the capacity to implement this.

[RedMser]

Thank you for your work on this proposal!

Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material? Is it for ease of implementation? I don't think this API will find use if color is all you can specify.
Technically an outline/xray material could be realized by godot automatically tweaking the stencil settings of the material sub-resource when it is used for outline/xray. If the inspector allows for it, disable or hide any options which are forced to a fixed value by the parent material.

Note that stencil stuff is all way above my head, so I probably wouldn't be able to benefit from even the mid-level API, let alone anything low-level.

[QbieShay]

Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material?

The idea is that if you want a custom one you can make a shader yourself, but if you want something quick it's just a matter of a few clicks. Note that it auto-adds a second pass to the standard material.

[fire]

Is it possible to support "half". So instead of an integer 0-255, it's a 16 bit integer? Not sure about the internals of allowing 32bit integers.

[clayjohn]

@fire No, OpenGL and Vulkan only support an 8-bit mask/reference value

[apples]

I'll be starting work on implementing this shortly!

No idea how long it might take. I'll be trying to implement the proposal as written, I don't think there will be any problems with it.

[QbieShay]

Thank you @apples !

[apples]

@QbieShay I'm wondering about the OUTLINE and XRAY modes adding a next_pass.

What specifically should happen for this? If the regular next_pass property is used, it will appear in editor, and may lead to confusion for users, and being able to edit that material might lead to problems. And what if the user has a next pass of their own?

Wondering what your thoughts are on this.

[QbieShay]

Hmm. I'd say for now leave it to the side, we'll have a chat about it in rocket chat. I have thought about it a bit, and I don't quite know.

[clayjohn]

I think maybe the following would work:

If user does not have a next_pass material
Create and set a next_pass material with the right stencil settings

f user has a next_pass material
Emit a warning in an editor toast (or do a proper popup) that says next_pass already exists, this effect requires the next pass to work.

[QbieShay]

@clayjohn I was thinking perhaps it can be together in that mesh menu where you can generate collission and outline mesh instead?

[apples]

A design question regarding the reference value and comparison operators:

With this design, the stencil reference value is exposed, but not the associated compare/write masks. But in both OpenGL and Vulkan, there are compare/write masks which are used to control which bits of the stencil values are considered.

To be useful without being able to specify the masks, the assumption must be that the masks will be equal to the reference value. For example, with a reference value of 4, both the compare mask and write mask should also be 4.

This works fine for simple "layer" based effects, but it brings into question what the comparison operators are for. The operators EQUAL, NOT_EQUAL, and ALWAYS make perfect sense to include, but when you think about the operators LESS or GREATER, those make less sense. GREATER especially wouldn't make sense, as it would be equivalent to NEVER.

So my suggestion is: Either we must allow the two masks to be specified in addition to the reference value, or we should remove the LESS and GREATER operators from the list.

[apples]

Also, was there any thought to integrate godotengine/godot#73527 into this? Sometimes depth functions are needed for certain stencil effects, e.g. Wind Waker style lights, which need to use depth function GREATER.

Edit for clarity: The linked PR godotengine/godot#78542 does include my depth function changes, but these are not part of this design as-written.

[clayjohn]

@apples IMO the masks should always be 255. Specifying masks is way too confusing for most people and only useful in very limited scenarios. This proposal intentionally ignores masks.

[apples]

masks should always be 255

But then how would we support multiple different effects using different bits of the stencil value? If you have an outline effect using ref 1, and a light effect using ref 2, with a mask of 255 they would interfere with each other.

[clayjohn]

Remember, the goal is not to support every single possible effect with stencils. The goal is to provide something to the user that is easy to use and useful. If we expose everything, stencils will be too confusing for anyone to actually use. Plus it contributes to a huge amount of bloat in the material (which hurts usability even more).

Many popular effects that require stencil won't work with a StandardMaterial anyway as they require fine grained control over culling, render order, blending, and depth testing. In order to make them work, we would have to not only expose every feature of stenciling, but massively overhaul how the StandardMaterial works. The cumulative effect would be turning it into a tool that is slightly more flexible for very advanced users, but impossible to manage for regular users.

We want all users to be able to benefit from stencil, which is why this proposal is aimed at covering 99% of use-cases. For advanced users we have plans to expose a new way of authoring materials that will provide access to all features from the graphics API. See also https://docs.godotengine.org/en/latest/contributing/development/best_practices_for_engine_contributors.html#cater-to-common-use-cases-leave-the-door-open-for-the-rare-ones

[apples]

@clayjohn Maybe there's a miscommunication here. I understand all that. I'm trying to simplify the design, not expand it.

I think using a mask of 255 is a bad idea, because it prevents multiple stencil effects from working together. As in my example, you couldn't have a character with an outline also be lit by a wind waker style light, since they would be overwriting each other's stencil buffer values.

I agree that exposing the masks to the user is a bad idea. That's why my alternative suggestion is to always use the ref value as the masks (and remove the LESS and GREATER operators since they become confusing at that point). This makes the ref value behave much more like a layer selection.

[QbieShay]

Isn't the power of the masks that you can use different read and write masks? If we don't support it then I'm not sure it's better than a single ref value, since ref can be always cleared afterwards (that windwaker thing can have a last stencil reset pass)

[QbieShay]

@apples Since I didn't see any further conversation on ref vs mask, I think in general if you believe there's things with mask that can't be achieved with ref only and not vice-versa, it's fine for me to add (please @clayjohn confirm), but then it should be given a name that makes is possible without breaking compat to add ref later (so, giving it a different name than ref, just mask or layers. i think i prefer layers)

[apples]

@QbieShay Just to clarify: I don't think we need to expose the mask to the users. I just think that using 255 as the masks internally will result in different stencil effects colliding with each other, and confuse users.

I think it would be simpler for users if we set the mask to be equal to ref, so that different stencil effects don't interfere with each other in the first place. This way there's no need for an extra "clear" pass (which is impossible to do in certain scenarios). Though, this does limit users to the 8 individual bits.

For advanced users, we can consider exposing the masks to the stencil_mode in the shaders only (not in the material properties), but I think it should be left out for now.

[QbieShay]

I think that it would be quite confusing then to use greater and lesser, if the ref is also the mask (for example 1 wouldn't be lesser than 2). If i understood correctly. If we use masks like this, i think it should be 100% clear that it's only 8 layers so that people know how to use it

[eddieataberk]

any updates on this?

[Calinou]

any updates on this?

To my knowledge, nobody is currently working on implementing this. It's kind of depending on #7916 either way, which isn't planned before 4.4 at the earliest.

[EnlightenedOne]

Out of curiosity in the tech wishlist the stencil operations are urgency high, importance high but the render compositor is urgency medium and importance high. If the compositor blocks the stencil solution should its urgency not match that of its highest dependency?

[Glorax]

Out of curiosity in the tech wishlist the stencil operations are urgency high, importance high but the render compositor is urgency medium and importance high. If the compositor blocks the stencil solution should its urgency not match that of its highest dependency?

@QbieShay @clayjohn Could we have clarification on this point?

[clayjohn]

Out of curiosity in the tech wishlist the stencil operations are urgency high, importance high but the render compositor is urgency medium and importance high. If the compositor blocks the stencil solution should its urgency not match that of its highest dependency?

@QbieShay @clayjohn Could we have clarification on this point?

Sure, those could be updated to be more consistent. We are also discussing ways of implemented stencils without having the full rendering compositor. Ideally we would merge the compositor first, but no one seems interested in working on it, so we don't know how long it will be until its implemented. It may be best to merge a very restricted form of stencils first so that users have something.

[Glorax]

@clayjohn I see this proposal is marked as "in progress". Does this mean it it's on the verge of being implemented in a 4.4 snapshot in the near future? It doesn't say "current cycle" or "backlog".

[fire]

As far as I know, there isn't a ready-to-review and test pr in 4.4, so I expect a later release of godot, like 4.5+ for stencil operations.