Add support for subdivision surfaces using OpenSubdiv

[pouleyKetchoupp]

Describe the project you are working on:
Sport game with 3D animation

Describe the problem or limitation you are having in your project:
We're using Subdivision Surface Modifier in Blender when creating our models because it allows modelling and animating low-poly models.
https://docs.blender.org/manual/en/latest/modeling/modifiers/generate/subdivision_surface.html

This produces perfectly smooth models in Blender, but once they are exported into Godot, some glitches appear during animation:
(note: expected results are generated from my work in progress integration of OpenSubdiv)

Expected result	Glitchy result

This is due to the following limitation:

• Export from Blender applies the modifier and generate new vertices with bone weights averaged from the base vertices.
• Godot applies skinning on all vertices from the high-poly model based on the average weights calculated during export, but these weights don't produce a smooth surface when bending.

The solution is to apply the subdivision modifier after the skinning in Godot, but this is not possible at the moment because all we have is the final subdivided mesh.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
This problem can be solved by generating a subdivision mesh in Godot, and interpolate extra vertices after the skinning process to keep surfaces smooth.

We can use OpenSubdiv to help with this process. It's and industry standard, open source and high performance:
http://graphics.pixar.com/opensubdiv

Then we can simply export the low-poly mesh from Blender or other modeling software, apply the subdivision in Godot and get smooth animated meshes:

Level 0	Level 1	Level 2

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
I've made a first basic implementation based on Godot 3.2 for my project:
nekomatata/godot@8f9674f

OpenSubdiv is added as a module to generate subdivision data out of Mesh and apply skinning when provided a Skeleton.

When the mesh is subdivided, the skinning is done in the OpenSubdiv module directly in order to add a the step to interpolate the extra vertices, so only skinning on CPU is supported in this case.

I've added a property in MeshInstance to set a subdivision level, which triggers the generation of the subdivision data and the high-poly rendered mesh.

Here are the next steps for integrating this module into Godot:

• Port to Godot 4.0 / Vulkan
  There's not much to do since the whole process is independent from low-level rendering, but updating vertex arrays dynamically is not supported at the moment on master, so that would require fixing before we can integrate it into Godot 4.0. This fix is needed for soft bodies as well.
• Handle normals & UVs
  I'm using flat shading in my project so I didn't implement normals and UVs generation, but it shouldn't be too difficult to add.
• Handle blend shapes
  Blend shapes are also not supported at the moment. It would require the module to apply blend shapes along with skinning before interpolating the extra vertices from the subdivision.
• Extra optimization
  CPU: The skinning process on CPU could be easily optimized a little.
  Memory: Share the same vertex array between the different surfaces of a mesh.
  Memory: Share subdivision data between instances of the same mesh.

And some extra functionalities would be worth being added in the future:

• Add a subdivision resource
  It would allow to optionally generate and store subdivision data offline to trade memory for loading time, and could have a few properties to allow more variation in how to generate the subdivision.
• Support for skinning on GPU
  I'm not sure if it's possible and how it would work in the current rendering system, but if the scene shader could access subdivision data we would be able to do the whole skinning/interpolation process on GPU instead of CPU.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
This enhancement can be useful for any artist who wants to use subdivision surfaces. It's too complex to be done with just a few lines of script.

Is there a reason why this should be core and not an add-on in the asset library?:
This feature would be best in core, because it needs to take place within the rendering process (after skinning and before rendering) and it requires to be integrated in an optimized way to get good performance.

[bojidar-bg]

Question: Is it possible to change the export process, so that the weights of the subdivided vertices are computed correctly?

[pouleyKetchoupp]

@bojidar-bg No configuration of weights is going to give you the expected results. Subdivision needs to be applied after the skinning to make sure the surfaces are smoothed correctly.

If you have Blender available, you can easily apply the subdivision surface modifier before or after the armature modifier to see the problem.


Here's my test file:
subdiv-test-new.zip

In this example, even if you apply the subdivision modifier destructively and re-weight high poly vertices (bending the joint to the right) by hand, you would still have problems when bending the joint to the left.

[fire]

Can you describe more about how to implement normals, UVs and blend shapes?

[pouleyKetchoupp]

I didn't investigate much in detail how to do the next steps, so after our discussion by dm I'm just leaving some links to find information for anyone interested.

OpenSubdiv documentation:
http://graphics.pixar.com/opensubdiv/docs/api_overview.html

Samples on the OpenSubdiv repo:
https://github.com/PixarAnimationStudios/OpenSubdiv

[logzero]

Imho your primary problem is with your vertex weights, not sure if godot is the right place to fix them.

Your low poly mesh in subdiv-test-new has 5 vertex loops from top to bottom and the weights are essentially:
(1, 0) (1, 0) (1, 1) (0, 1) (0, 1)

To get better subdivision/interpolation results change them to;
(1, 0) (0.75, 0.25) (0.5, 0.5) (0.25, 0.75) (0, 1)

You can also check/adjust the weights after baking/applying the subdivision.

[pouleyKetchoupp]

@logzero Thanks for your suggestion, I've just tried in Blender to see how it helps. These weight values you're providing allow different interpolation results, but the skinning doesn't look the same. This is not the correct result for joints like character's elbows and knees.

Original weights (post-skinning)	Suggested weights (pre-skinning)

edit: I've updated the screenshots from Blender to accurately reflect what you get in Godot. "Preserve Volume" must be turned off in the armature modifier.

Once imported in Godot, it does work without the need for subdivision, but the result is that it deforms the whole shape while bending, so it's not what we're looking for:

Original weights (post-skinning)	Suggested weights (pre-skinning)

The idea in this proposal is to provide a way for artists to work on the low-poly models, and get the exact same result in Godot. I imagine lots of artists are dealing with workflows where they have to re-adjust weights on the high-poly model. This takes a lot of time, and they can't get the exact same result as with surface subdivision. That's exactly why we've decided to handle subdivision in the game engine, upon request from the artist in our team :)

[pouleyKetchoupp]

Also, for anybody interested I'd like to share this GDC presentation about subdivision that @fire mentioned on IRC when we were discussing this proposal:
http://twvideo01.ubm-us.net/o1/vault/GDC2014/Presentations/Brainerd_Wade_Tessellation_in_Call.pdf

It explains why they started to use subdivision in Call of Duty : Ghost to help with the art pipeline, with lots of technical details.

[fire]

I'm not able to do much with opensubdiv, but here are some videos of the branch. UVs, materials and normals were added. Skinning is the same as the original.

[DrAlta]

I'm going to try to hire someone to add render time tessellation as a stretch goal for my MMO Crowd funder. So if anyone how has the skills and the opening in their scheduled to work on this.

Godot Forum post for the job

[Calinou]

Now that GDExtension is here, is this feature still candidate for being added in core? Or would it be better created as an add-on?

[fire]

I remember having to modify the mesh apis in the RenderServer. In the middle of something so am unable to do an investigation.

[Ansraer]

Another major benefit of applying the subdiv modifier in godot is that the shipped 3d models can be significantly smaller.
An exported low poly mesh from blender is 4-6 times smaller than the same mesh after a single subdiv iteration was applied. For a single more detailed character this can quickly add up to hundreds of MBs.

EDIT: My hero character with all the shapekeys and only the most basic animations at a single subdiv iteration is currently 750MB. Since I plan to add quite a few more shapekeys & animations I imagine this will increase a lot. Admittedly, some of the suggestions discussed in #24 might help with decreasing that size, but it's still far larger than I want it to be.

[Ansraer]

@fire Could you link your subdiv 3.x branch? I could really use this for the project I am currently working on. If I find some time (and manage to finally fix my 4.0 display scaling PR first) I might even look into upgrading it to 4.x.

[fire]

Here is my non-working master https://github.com/V-Sekai/godot/tree/subdivision.

I had some changes, but these are my older branches.

• https://github.com/godot-extended-libraries/godot-fire/tree/feature/3.2/opensubdiv-next
• https://github.com/godot-extended-libraries/godot-fire/tree/feature/3.2/opensubdiv

[tefusion]

Now that GDExtension is here, is this feature still candidate for being added in core? Or would it be better created as an add-on?

I think an addon is better here, an attempt of mine: https://github.com/tefusion/godot-subdiv

I personally think this would be a pain to actually implement nicely in core. Subdivisions and gltf/the way godot stores mesh data fit together like pineapple and pizza. The easiest way to show this is to export the default cube as glb and reimport it. What you will get is a bunch of disconnected triangles (exception is when UV's are together). OpenSubdiv needs an index array which stores face connections though. The way fire solved this and what I also use in my addon is to just assume that if two vertices are at the same position they were original the same vertex and change the index array that way to get connected Faces.
Afterwards comes the actual subdivision. Maybe I just didn't find the right settings, but in a quick port to Godot 4 from opensubdiv-next Loop subdivision looked clunky to me and was also quite slow on larger meshes. I didn't try for long, so might've been my mistake.

Now how to get quads to be able to use the more common Catmull Clark subdivision modifier : I found some stuff back from 2019 on the gltf github which would add ngons, but afaik that never really gained that much traction and would require a custom format which wouldn't work with programs that flag is not implemented in. Another way is when you use the assumption of vert positions you just also try and merge quads with the assumption that two neighbouring triangles are a quad . I've only tried blender glb exports so far, but that solution actually seems to always be working and that's also the way my addon currently imports meshes.

[fire]

Opensubdiv has a catmark loop operator that works on trimeshes. The blog noted it was a new feature. https://graphics.pixar.com/opensubdiv/docs/release_34.html#triangular-patches-for-loop-subdivision

[fire]

https://github.com/tefusion/godot-subdiv

2022-10-14.15-17-54.mp4

• tri mesh -> opensubdiv -> baked into meshinstance3d
• works with skinning (evidence above)
• works with blend shapes (no video)
• requires subdivision bevels and no creases
• works with lod generation

[Calinou]

Subdividing on import is an interesting idea, but it's not compatible with procedural generation. It also still causes the file size space in the PCK to increase. Ideally, subdivision should happen on the fly (with caching) so that you don't have to include subdivided models in the PCK. This also allows making subdivision optional, so that you can disable it on low-end machines without having to ship separate models.

[fire]

I didn’t want to ship the opensubdiv gdextension so I pushed for import baking, but your feature of runtime subdiv level has been working for months now.

[fire]

No gpu compute shader yet, theres a metal shader we can port.