Weird issue with one area of texture not being painted

[iatenothingbutriceforthreedays]

Hey thank you so much for creating this package and publishing the source!
I am trying to use it for painting bullet holes on a humanoid ragdoll and it is working great, except for this strange thing where the decals don't apply to one segment of the model (made the decals huge here to make it obvious):

As you can see there's this portion of the model along the figure's left side that doesn't get painted at all. The other side of the model gets painted fine.

Here's the DecalColorMap where you can clearly see that weird hard edge in the lower right:

It looks like that edge aligns with edges of tris in the model:

But I have no idea why that particular region wouldn't be painted.

I set up everything according to the instructions (I'm using the built-in pipeline), I switched on lightmap generation to generate the UV2, and the decal textures are set to 'clamp' and mipmaps disabled. I thought there might be some problem with the auto-generated lightmap UV, so I also tried adjusting the shader to use the original UV1 for the decal normal map but it has exactly the same behavior, even though that hard edge doesn't correspond to a seam in the UV map.

I've also tried messing with the position and rotation of the decal projection (e.g. projecting it straight-on to those faces), but it seems to make no difference.

Do you have any idea what might be causing this? Any idea what I could try to debug it further? I've run out of ideas and I'm not good enough with shader programming to dig into the DecalProjector.shader code.

Thank you, any help is much appreciated!

[naelstrof]

Oh that's really weird, I have a few guesses:

• Decals won't project there if the uvs don't allow for it (if they're outside the 0 to 1 plane for example). It might be worth adjusting the UV2s. (Mess with the lightmap generation settings on the model.)
• If backface culling is enabled on any of the examples, they'll fail to paint on backfaces in the projection. Doesn't really seem like the problem here though.
• The projection depth of the decal could just be clipping it right there. Again doesn't really seem to be the problem because it's failing right on the edge.

Out of all these guesses, I'd wager that the UV2 has something wrong with it. You mentioned that you tried switching it to uv1 and it had the same problem, though uv1s can be overlapping and off the 0 to 1 UV plane which can cause lots of problems.

[naelstrof]

To investigate further, I'd probably need to get my hands on the model!

[iatenothingbutriceforthreedays]

Hey, I don't want to upload the model directly but if you do want to take a look it's a free asset on the asset store and I didn't make any modification except to switch on lightmap uvs, and add a DecalableLit material. But I also tried with a completely different model and had basically the same problem still. Maybe I'm doing something wrong with the way I'm using the decal system? Or might be something specific to the built-in render pipeline? I'll see if I can reproduce this in HDRP, once i figure out how to set that up.

Asset: https://assetstore.unity.com/packages/3d/characters/creatures/character-monster-1-76640

[iatenothingbutriceforthreedays]

Oh yeah, and I tried messing with the lightmap settings like you suggested, but no luck. Tried changing the projection depth as well and no result.

[naelstrof]

Could I see the code that you use to draw the decal?

[iatenothingbutriceforthreedays]

Sure:

OnCollisionEnter(Collision collision) {
            Vector3 impactPos = collision.contacts[0].point;
            Vector3 impactNormal = collision.contacts[0].normal;
            Quaternion rotation = Quaternion.LookRotation(impactNormal);
            
            // Create bullet hole
            decalProjector.SetTexture("_Decal", bulletHoleFleshColor);

            SkinnedMeshDecals.PaintDecal.RenderDecal(
                bodyRenderer, // the SkinnedMeshRenderer attached to the ragdoll
                decalProjector, // a Material with the "Naelstrof/DecalProjector" shader
                impactPos,
                rotation,
                decalSize,
                decalDepth,
                "_DecalColorMap"
            );
}

The collision is between a SphereCollider (bullet) and a BoxCollider (body).
Thanks a lot for helping out :)

[iatenothingbutriceforthreedays]

OK, I modified your SkinnedMeshDecalExample a little to work with the built-in render pipeline, and i tried adding my model to that scene and it works fine, except for some visible artifacts along the seams of the lightmap UV:

That might be something to do with my graphics card, idk. Anyway it doesn't have the problem with not painting a whole section of the mesh. So I guess the problem is somewhere with my decal projecting code. I'll keep looking into it.

[naelstrof]

Seems to work on my end too. My other guess is the fact that they have lots of meshes to create the mouth and face, depending on how they unwrapped they can overlap and mess with other parts. Actually I'm unsure about this. Ideally this would all be one mesh anyway.

As for the seams, there's a variety of solutions that can be made for it, a very detailed video that has a very similar approach has good solutions for you here: https://youtu.be/c7HBxBfCsas?t=163

I already use conservative rasterization when possible, though even with it-- there seems to always be a seam or two.

[iatenothingbutriceforthreedays]

OK, I made a little progress on figuring this out. In my original scene the camera view was being rendered into a rendertexture which was then being rendered on a different camera (if that makes sense). That seems to be what's causing the problem, I assume due to some kind of screen-space calculations in DecalProjector? I don't really know.
I still haven't figured out why that's happening, or how to work around it, but I'll keep messing around. If you have any ideas let me know!

[iatenothingbutriceforthreedays]

Actually for my case I can just work around it by avoiding rendering into a rendertexture. But would be cool if there was a way to do use rendertextures without breaking the decal system.

[naelstrof]

That sounds very strange! The decal system automatically creates and manages its own render textures. It also will magically delete them when memory constraints are reached.

It also manually renders to them using command buffers, so no cameras should actually be involved. I'm very confused

[iatenothingbutriceforthreedays]

Hey I don't think I explained very well - I mean in my original setup I actually had two cameras: the first camera is pointed at the ragdoll character, and that camera doesn't render to the screen, it renders to a RenderTexture:

And then there's a second camera which is pointed at a quad which displays the render texture:

And that camera renders to the screen.

If I just render the first camera directly to the screen then the decal issues go away.
So something about my rendertexture setup seems to be causing problems for the decal system, I still don't know exactly why though.

[naelstrof]

Ooh I see. That's really strange that it causes a problem. I would think it wouldn't since the command buffers should be all self contained. Since it does cause a problem I must be using a variable in the shader that unintentionally comes from an active camera. I'll try to look into it.

However, a quick fix on your end would be for you to do some proper post-processing/procedurals/command buffers rather than pointing cameras at quads. Though I understand that's a big ask. Sorry!