Add a project setting to use hard cutoffs for shadow map rendering

[Calinou]

• Related to Make a crisp shadow rendering effect similar to old classic games #1930.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Shadow rendering in Godot can only result in pixelated shadows with bilinear filtering between pixels, or soft shadows. There is no way to draw hard shadows.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Smooth cutoff (current)	Hard cutoff (new option)

Hard cutoff shadows in action:

truck_town_hard_shadows.webm

Compared to stencil shadows, this approach has many advantages:

• No restrictions on polygon count. With stencil shadows, you need to be careful to avoid having too many polygons in your shadow meshes, as high polygon counts can be very slow when using stencil shadows, even on high-end GPUs. Shadow maps don't have this issue and can end up being faster than stencil shadows. This makes the approach versatile, from lowpoly retro-styled projects with small-scale levels to modern stylized open world games.
• Alpha scissor, opaque pre-pass and alpha hash material shadows are supported. In comparison, stencil shadows can't support these by design (other than casting solid shadows).
• This approach supports colored shadow maps with per-pixel colors if these are implemented in the future (Implement dithered shadows for semi-transparent objects #3276).

It also has some drawbacks compared to stencil shadows:

• Shadows are less stable in motion. While increasing the shadow resolution and/or filter quality helps, stencil shadows remain unbeatable on this aspect.
• Shadow edges look somewhat rounded. This is less noticeable at higher shadow resolutions (or at lower viewport resolutions).
• Shadows can exhibit the usual shadow acne and peter-panning issues that shadow maps have, although hard shadows are generally less subject to this since they have a distinct lack of blurring. This lets you get away with lower bias values than you could otherwise use. In contrast, stencil shadows generally don't have to deal with shadow acne and peter-panning.

Nonetheless, it's a viable approach for many games and is much easier to implement in the engine compared to stencil shadows (which would require stencil support to be implemented first anyway).

Performance between both approaches is expected to be roughly identical. A hard cutoff allows lower-resolution shadows to look sharper, at the cost of looking less stable in motion. Shadow filter quality can be increased to reduce stairstepping artifacts in the shadow and improve stability in motion at the same time. This approach scales up very well to high shadow resolutions and high filter quality, allowing for near stencil-like shadows if desired.

Default shadow resolution	Low shadow resolution	Maximum shadow resolution

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The easiest way to implement a hard shadow cutoff is to use a conditional in the various sample_shadow() shader functions. This conditional returns 1.0 for values above 0.5 and 0.0 for values below 0.5.
Of course, there are more advanced approaches which allow for the hard cutoff to remain slightly smooth, which is useful to antialias the result without needing FXAA or TAA. I haven't started exploring these yet, so let me know in the comments if you have experience with this.¹

We could choose to expose a shadow hardness factor that goes from 0.0 (current behavior) to 1.0, either as a per-light property or a project setting for directional and positional shadows respectively. If this is a project setting, this can be sent to the shader as a specialization constant to avoid wasting performance on unused codepaths.

I have a proof of concept branch here: https://github.com/Calinou/godot/tree/add-hard-cutoff-shadows-option
It works with all rendering methods (Forward+, Mobile, Compatibility). Setting the shadow filter quality to High or Ultra in the project settings is recommended to improve shadow stability in motion.

We could further improve quality in Forward+/Mobile by implementing a traditional PCF shadow filtering codepath (as done in the Compatibility rendering method), which seems better suited to hard shadow cutoffs than the Vogel disk rotation method. This method is suited to soft dithered shadows, but it's suboptimal for this particular use case. This would also be helpful for godotengine/godot#53961.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Shader templates would make it possible to implement this as an add-on, assuming they give enough low-level shader access to override the shadow sampling options. However, it might be argued that games with a stylized art direction are common enough nowadays for a hard shadow cutoff option to be beneficial in core.

Additionally, it's not known yet whether shader templates will be supported when using the Compatibility rendering method. In comparison, the proof of concept branch linked in this proposal already supports all rendering methods.

Footnotes

1. MSAA can't make shadow maps look smoother, unlike stencil shadows. ↩