Implement LightmapGI shadowmasks

[BlueCube3310]

Salvages #77284
Closes: godotengine/godot-proposals#2354

Adds support for baked shadowmasks to LightmapGI. The shadowmasks are saved as compressed images and applied on top of the dynamic shadowmaps.

Progress:

• Support baking shadowmasks,
• Support Forward+ renderer,
• Support Mobile renderer,
• Support Compatibility renderer,
• (Bugfix) Allow lightmaps without shadowmasks,
• Dilate,
• Fix blending artifacts, Edit: Caused by DXT1 Compression,
• Document everything properly,
• Clean up and optimize the code,

To consider:

• Consider using specialization constants,
• Consider blending seams.

To do in the future (important):

• Compress grayscale textures as RGTC_R, Edit: done
• Implement and use bicubic sampling (See Implement bicubic sampling for lightmaps #89919) Edit: done,
• Further optimizations and cleanup,
• Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.
• Remind the user when lighting needs to be rebuilt,

[AThousandShips]

Since this is salvaged from a previous PR, unless it doesn't contain any of their code, please add them as co-author

[BlueCube3310]

Did I do it correctly? Sorry, I'm still new to github.

[BlueCube3310]

I think it's ready for review now. I'm still not certain whether the documentation is clear enough, or if the seams are noticeable enough to require blending them, though.

[Calinou]

Filter the shadowmasks in the Lightmapper,

Reimplementing support for bicubic sampling could resolve this. If this doesn't suffice, we could add an option that uses Image::resize() to halve the shadowmask resolution, then use Image::resize() on the small image to bring it back to the original size with cubic resampling. This will effectively blur the shadowmask without requiring an expensive filter.

Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.

This is a clever idea I hadn't thought of before. It makes a lot of sense if you need sharp distant shadows but don't need indirect lighting data to be very precise 🙂

That said, in practice, you don't want distant shadows to be too precise as you run the risk of having aliasing visible in the distance (lightmaps and shadowmasks aren't mipmapped).

[Calinou]

Tested locally, it mostly works as expected (I tested Forward+ and Mobile).

Testing project: test_lightmapgi_mobile.zip

The way shadowmask interacts with real-time shadows looks strange. I don't recall this happening with previous PRs, but maybe I started noticing this issue just now. This is most noticeable with Directional Soft Shadow Filter Quality set to Ultra, but it occurs with any setting:

I'd expect the final shadow color to be min(shadowmask, real_time_shadowmap). Right now, it seems to be real_time_shadowmap > 0.0 ? real_time_shadowmap : shadowmask. This issue affects both Forward+ and Mobile.

If you set Directional Soft Shadow Filter Quality to Hard, notice how shadowed area transitions look much sharper than intended:

Regarding fixing seams, I'd say it's worth the effort as right now, seams can be quite noticeable in the distance:

Real-time shadow	Shadowmask

[SpockBauru]

Tested my MRP on Forward+, Mobile and Compatibility renderers. Shadowmasks is working as intended without any visual issues. Baked the LightmapGI on each renderer and everything is working.

Also made a quick test on my (halted) personal project. Out of the box, it works as expected without any issues. Far shadows are now present:

Shadowmaps None	Shadowmaps Replace

With this developers can make an option to reduce shadowmaps resolution and distance, improving the performance on weak systems.

I didn't find any issue related to this PR and believe that is ready to merge.

[clayjohn]

Let's go ahead and merge this!

It looks great and while I think there is room for iteration, I think it's time to get this in the hands of users so we can get some feedback

[akien-mga]

Thanks!

[Calinou]

@BlueCube3310 I noticed shadowmasks are saved as PNGs, but saving them as lossless WebPs can make them much smaller on disk. Example with a 4096×4096 shadowmask texture:

321k test_shadow.png
151k test_shadow.webp

This does not affect memory usage, but it'll reduce file sizes in version control, which is helpful as lightmap data is often iterated upon. In the codebase, it should be as simple as replacing save_png() to save_webp() (which is lossless by default).

The downside is that we can only do that for images that are 16383×16383 or smaller, so shadowmasks that are larger would have to be resized down. In practice, only 16384×16384 can be an issue here¹, so we could downsize it to 16383×16383 upon saving (Godot's VRAM compression will pad the last pixel automatically).

Footnotes

1. The maximum lightmap image size that can be saved by Godot is 16384×16384. ↩

[michaelharmonart]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

[clayjohn]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

That's an existing issue in LightmapGI. #98684

[michaelharmonart]

ahh ok. l never noticed since there was never a way to blend between real-time and baked lighting like this before now

I noticed now only since there was a clear transition line between realtime and baked shadows only since there was a mismatch in which objects had shadows.

[IsaacMarovitz]

In 4.4.dev7 despite having shadowmask mode set to Replace, baking does not seem to build the shadowmasks.

I can't tell if I'm using it wrong, or if this is a bug.

[SpockBauru]

baking does not seem to build the shadowmasks.

@IsaacMarovitz It's working here, did you remembered to set shadowmasks to replace before baking?

[IsaacMarovitz]

Nvm, I was using it wrong, my directional light was set to static and I missed the warn in the console. The DirectionalLight Bake Mode documentation needs to be updated to reflect this change.

[unfa]

Hey, where can I find a good explanation of this feature? I was looking around and couldn't really find anything.

[SpockBauru]

Officially there's an explanation on the "latest" version of the docs, not the "stable" one since 4.4 stable was not released yet. Be aware that it's an experimental feature so there's no much explanation yet:

• https://docs.godotengine.org/en/latest/classes/class_lightmapgi.html#class-lightmapgi-property-shadowmask-mode
• https://docs.godotengine.org/en/latest/classes/class_lightmapgidata.html#enum-lightmapgidata-shadowmaskmode

I'm also answering people on this Reddit thread: https://www.reddit.com/r/godot/comments/1hcr73g

[Calinou]

Hey, where can I find a good explanation of this feature? I was looking around and couldn't really find anything.

I've opened a PR to document shadowmasks: godotengine/godot-docs#10482

[GustJc]

Not sure if something changed.
But using the latest master (v4.4.dev.gh [24d7451]) and trying to bake the lightmapGI using the testproject from (#85653 (comment)) crashes Godot.
With a lot of "Couldn't map PC to fn name" errors.

But testing it using a simple scene on a new projected or using the test_lightmapgi_mobile project (#85653 (review)) worked fine.

shadowmap_bug.mp4

Here is the start of the crash:
(On Windows 10, Foward+, Nvidea 1070)

[SpockBauru]

Compiled the master branch locally and can confirm that the demo is not working anymore. First time I baked, it got my PC unresponsive for a while and I had to make a hard reset. Second time it crashed Godot.

My system: Godot v4.4.dev (24d7451) - Windows 11 (build 22631) - Multi-window, 1 monitor - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3080 (NVIDIA; 32.0.15.6614) - 12th Gen Intel(R) Core(TM) i7-12700KF (20 threads)

@GustJc Would you kindly open a new issue?

@Geometror
EDIT: Tested some some PR's and is crashing since #99538 (Transparency PR). Something broke between Dec 12 and Dec 19.

EDIT2: It seems like if I insist and try again it manage to bake, but taking double the time and the computer becomes unresponsive in the process.

[FeralBytes]

@clayjohn

Personally, I suspect that we can get away with making the shadowmask a binary selection "SHADOW_MASK_ENABLED", "SHADOW_MASK_DISABLED" with a boolean that pops up when enabled to turn on/off blending. But I would like to get some more user feedback about what they need before we go down the rabbit hole of trying to decide on users problems for them.

I would like to provide some user feedback. In my limited testing overlay seemed to work better for me on the web than replace, because the default dynamic shadow distance was so low in compatibility mode on the web, the shadow would begin to hide itself before it could be replaced for objects that were tall, such as a hill nearby. So for the web, which really needs such a performant feature, I think overlay may be a better option than replace.

That also makes me wonder if shadow mask ONLY would do even better. But it does really come down to the needs of the game and the artist designing the game. I personally, am in favor of keeping all of the modes exposed, because Godot is a general solution game engine. People have many potential use cases and we cannot know them all right now, so early in the release of this new feature.

Thank you for your time.