[Bullet] Concave polygon collision shapes are completely broken in Bullet on master when built with LLVM

[Exxion]

Godot version:

Most recently tested on 49b5776 but has been present for a while (definitely at least 11 days but I'm pretty sure it was at least a few days longer than that). Presumably introduced sometime after #42881 was opened because it would have made testing part of that PR impossible. Could be arbitrarily old assuming that PR was not tested with LLVM.

OS/device including version:

Windows 10 Home 64-bit

Issue description:

On Bullet physics, when built with LLVM, rigid bodies do not interact with static bodies with concave polygon colliders. Convex polygon colliders work fine, and concave ones also work fine on Godot 3D physics, but not Bullet.

Steps to reproduce:

1. Build master with LLVM.
2. Use a mesh to create a staticbody with trimesh collider, or use a mesh to create a trimesh collider on an existing static body.
3. Create a rigid body that should collide with it.
4. Watch as it does not.

Minimal reproduction project:

MeshCollide.zip

[slapin]

That is strange as it is rather common case. Can't reproduce this with my own project or starting from scratch, using 8f265e8. Recent breakage?

[slapin]

if 8f265e8 works for you it can be bisected then

[akien-mga]

if 8f265e8 works for you it can be bisected then

Note that this is a 3.2 commit while OP's 49b5776 is in master, so it can't be directly bisected as the branches diverged, it would have to be between 3.2-stable and 49b5776, which would take a while.

But it's likely that the regression is more recent than that and specific to master.

[slapin]

Sorry for confusion, it is better to find good commit in master and bisect from there.

[pouleyKetchoupp]

@Exxion Please add the project you've updated to 4.0 to make testing easier.

[Exxion]

@pouleyKetchoupp I just made a new one instead. I updated the OP with it.

[pouleyKetchoupp]

Can't reproduce on 4a0abe7. There's a little overlap on collision with both Bullet and Godot Physics, but it does collide. Could you try again on latest and make sure you don't have custom changes that could affect the behavior?

[Exxion]

Still happens for me on latest. The box just falls right through the mesh collider. I don't have any custom code, though I do compile with LLVM. It would be very weird for this to be compiler-dependent, though. I'll try to get MinGW to work and, if successful, try it again with that.

[pouleyKetchoupp]

Another thing to try could be to clean compile the project, just in case something went wrong with the build system somehow.

[Exxion]

I finally managed to find a MinGW toolchain that could actually build Godot, and it does indeed appear to somehow be compiler-dependent. I wouldn't exactly call the collision on this build good, but it does collide.
I also cleaned the build before running it with that toolchain, so it could have been that, but I'm pretty sure I had already tried that. Not certain though. I'll try LLVM again just to make sure.

EDIT: I apparently can't test it again without reinstalling some garbage because the linker from this toolchain doesn't seem to play nice with LLVM, or something
Too bad despite this bug because LLVM is way faster, at least on my machine

[Exxion]

And now I finally managed to get LLVM to work again and it doesn't happen anymore. I'm not sure whether it was because I cleaned the build, or because I broke something in a header file somewhere when getting LLVM to work the first time, or because I'm using a slightly newer version of LLVM now, or just something mysterious. Whatever the reason, it wasn't actually on Godot's end. Sorry for the trouble.