Implement Shadow Meshes

[mrjustaguy]

Supersedes #2600

Describe the project you are working on

Nothing Specific.

Describe the problem or limitation you are having in your project

Most of the Complexity of a Mesh isn't visible when using shadows, meaning a very complex mesh is being used to render shadows.

Current Workaround is to have 2 Meshes, with the 2nd mesh being a Simplified version of the original, and having the original mesh have Cast Shadows set to off, while the shadow mesh has Cast Shadows set to shadows only. This has an issue with Mesh 1 being shadowed by Mesh 2 in places where it shouldn't be shadowed, as the Shadow Mesh has a Very simple geometry compared to the Original Mesh, and the Original Mesh is very concave, in a Way the Shadow Mesh just cannot be due to the lower number of polygons.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Allowing for Shadow Meshes would improve performance (significantly in some cases) because way fewer Triangles would have to be processed by the Shadow Rendering for a nearly identical outcome.

However, to Avoid the issue of the Workaround described above, there needs to be some additional work, to prevent Inaccurate Self Shadows, while keeping The desired Self Shadowing.
Example of what I mean:
Imagine a Human in a T-Pose, with a light placed just left of his left leg. The left Leg should shadow the Right leg and prevent it from getting that Light. However, due to simplification of the Leg, the Simplified Mesh of the Leg is actually poking the original Leg mesh, and Without mechanisms in place, would end up shadowing The Foot that is Simplfied into a box, as that box almost entirely contains the Original Foot and as such occludes it. If on the other hand There is no self shadowing, the Right leg will be lit despite being occluded by the left leg.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

How the Shadow Mesh would work:

1. General World (Not the Original Mesh the Shadow Mesh is created from) - Shadows it, Original Mesh is Not casting Shadows, No Corner Cases.
2. Self Shadowing [Corner Case Warning] (Original Mesh must get shadowed By the Shadow Mesh Properly) - This is the Tricky part, as there are 2 Types of Self Shadowing, The Desired (Original Mesh would shadow itself in the given conditions) and The Undesired (Original Mesh wouldn't shadow itself in the given conditions)
   The separation of the two types of Self Shadows would have to be dealt with, Along the lines of:
   Shadow Mesh Self Shadows, meaning Original Mesh would Self Shadow (99% chance really, unless one of the poking parts is self shadowing, however this can't be mitigated probably, and is a part of the expected Slight Quality Loss)
   Shadow Mesh Shadow should as such be transferred from the Shadow Mesh to the Original Mesh, and the best way to do it would probably be some sort of Shadow Texture Transplant from the Shadow Mesh to the Original Mesh.
   The issue of doing this would be, what if The Original Mesh inside the Shadowed triangle is concave, and has other triangles in the concave area, as this would lead to Shadows on Random parts of the Original Mesh, which is also Unwanted. The Way Around it would be to create a Special UV for the Shadow Mesh->Original Mesh, which would essentially boil down to doing this:
   For ShadowMeshTriangles Place OriginalMeshTriangles that are projected onto the Shadow Mesh Triangle with Depth Information, to see figure out which triangles that are Inside the Shadow Mesh UV Triangle are the ones that actually matter. See the Picture Below for clarification of what the Loop above is intended for creating.
   

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Rendering is Core.

Note: Godot can Auto-Generate Shadow Meshes, as an LOD of the Original Mesh, and is the easiest part to implement and as such is taken for granted in the Proposal above..

[clayjohn]

Godot already uses simplified meshes for Shadow Meshes. See godotengine/godot#45452

There is also already a proposal for using a lower LOD automatically for Shadow Meshes #2600

Can you clarify what this proposal adds that is different?

[mrjustaguy]

Sure.. 1st off, This isn't Related to the PR's simplification, as that isn't using a LOD, This is.

The reason being for Going over #2600 is that it isn't a Proper proposal, so Aimed at Making a Proper Proposal for adding the Feature that is Wanted with that Proposal, and also Solving A problem of The "workaround" in #2600.
(which is using 2 Mesh Instances, with one of them being the Mesh with cast shadows off, and in it's full Triangle Glory, and the other being Shadow Caster only, with only a fraction of the Triangles, so an LOD of the 1st mesh.)

The Problem I'm talking about is the fact that Lower LODs will often enough, be poking outside the Meshes with More Triangles, Which causes Undesired Self Shadowing, (See SS.zip, and please note, This Project Example of the effect, this Isn't Proper Self Shadowing, as LODs of the Primitives Never have Poking Triangles, because with primitives an LOD of a given Primitive will always be contained within the Original Shape, something that isn't the case with Concave Shape LODs)
and currently the only way to prevent such an issue with self shadowing would be to not allow the Shadow Mesh to shadow the Original, Higher Poly Mesh, which again isn't desirable as you still probably want Some Self Shadowing.

The Proposed solution to the Self Shadowing Problem is to basically Map the Shadow Mesh Shadow Onto the Visible Mesh (aka the Original) because the Shadow Mesh can Self Shadow Accurately, just can't do so accurately for a Higher Poly Model that has Any Vertices inside the Lower Poly Shadow Model which cause the Undesired type of Self Shadowing.

The Proposed solution to the Self Shadowing Problem however Might be impossible to do, but I don't know if it is or isn't, as I don't know How Rendering works well enough to be able to determine that, and really if the SS Problem solution isn't doable, it doesn't add too much to #2600 aside from the Structure...

[clayjohn]

Thanks for the clarification. It sounds like you aren't proposes a new feature, rather you are suggesting some implementation details for #2600 to avoid a problem that might arise if #2600 is implemented.

Is that fair to say?

If so, I might just merge this proposal with that one so all discussion about using LODs for Shadow Meshes can be tracked in one place.

[mrjustaguy]

It's fair to say that, so yea Merging would make sense.

[Calinou]

Closing in favor of #2600. Please continue the discussion there.