Make 3D Bullet physics multi-threaded

[starwolfy]

Describe the project you are working on:

A multiplayer game involving dedicated servers hosting higher player counts.
My server project simulates physics at a low tickrate and the clients interpolate the results. This gives each physics tick more time to process and allows for the creation of more physics entities, ideal for a server which hosts a larger amount of physics entities including players.

Describe the problem or limitation you are having in your project:

The 3D Bullet physics engine of Godot is single-threaded, my server project starts lagging behind when a certain amount of physics entities are spawned in and the profiler shows that this is caused by the native physics engine of Godot doing calculations. (disabled _phyics_process code for this case so nothing else holds it back).
At a certain amount of physics entities, the "Physics Time" takes too long to process each physics tick and the server becomes unplayable while my 8 threaded CPU only has 1 thread working at max capacity on Godot physics. Causing my server to have a physics bottleneck at a total of only 20% CPU usage.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

Allowing the Bullet physics engine to utilize all available threads on the given CPU would increase the maximum physics entity count by up to 4-6 times on high-end gaming processors!
There are server hosting solutions out there which allow for even more than 8 CPU cores, which are ideal for hosting large game servers, if 3D Godot physics were to allow multi-threading these servers would allow for many, many physics entities. Imagine game servers hosting worlds that contain up to a hundred players with all kinds of entities they can create and interact with! This would allow for HUGE dedicated server type game projects.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

@fire has a pull request in the making which unlocks multi-threading Bullet physics: godotengine/godot#40073
I am not knowledgeable enough to explain how the multi-threaded bullet physics feature is implemented in the back-end/core of Godot, but the aforementioned pull request should do a great job at showing how it is done!
By the looks of it, it is a change in the core physics code, not much would have to be done on the Godot game developer's end apart from enabling multi-threaded (physics) operations in the project settings.

So with this implementation, the total number of physics bodies that can be put in the same physical space that can interact with one another could be huge, especially for machines with high-end CPUs like servers.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

No, it cannot be worked around easily as this is a real performance limitation to 3D Godot physics, we are limited to one CPU thread. One could use 3D PhysicsServers for multi-threaded access on physics operations, but the 3D PhysicsServer instances still run on the same main thread which does not unlock the power of the additional CPU cores that are potentially idling, as mentioned in #483 (a great proposal to push the 3D physics boundaries as well).

Is there a reason why this should be core and not an add-on in the asset library?:

As far as I'm concerned Godot Bullet physics makes up a core engine part and cannot be changed with an add-on in the asset library in a performant way.

[starwolfy]

Here is another well-known game demonstration as to why properly multi-threading game physics in one big space state can benefit large online games and how keeping them locked to one thread becomes a bottleneck for dedicated servers:
https://youtu.be/MBpN679o5Yk
Minecraft servers face limitations because they are designed to mainly run on one thread. This is why Minecraft servers don't go over 200 players and with recent updates even fewer players.

[jonbonazza]

I think that multithreaded physics is a good idea, but i feel that all physics backends should be multithreaded, which is a fundamental change to the engine architecture. I think that this should absolutely br considered for godot 4.0

[Calinou]

There are plans to rework GodotPhysics so it can optionally make use of the GPU using compute shaders 🙂

Edit: Not for 4.0, but possibly for a future 4.x release.

[starwolfy]

@Calinou That sounds like a really cool feature actually, this would benefit a lot of projects.
Unfortunately, it doesn't work for my case as my game will rely on community-hosted servers and it would be too much to have GPU servers as a requirement for server owners as they are really pricey to rent.

[WilliamTambellini]

As @Classy11 , I would also vote for a CPU MT physics acceleration first, then GPU as a second phase.
Refs: CPU multithreaded bullet implemented in 2018:
bulletphysics/bullet3#126
and the MT demo:
https://github.com/bulletphysics/bullet3/tree/master/examples/MultiThreadedDemo

[jonbonazza]

I also agree that a CPU implementation should be more of a priority. We are also using godot server builds for our online game, and our instances are running on EKS, where gpu enabled servers are not an option.

[iBoonie]

Having GPU based physics would be fine on the client side, but if you are going to be hosting dedicated servers, 99% of the time you are not going to be getting any GPU compute (and if you do need it, you are going to be paying a hefty premium for it). Just for that case alone there is a need for a multi-threaded CPU based implementation.

[Jimmio92]

I feel like I should mention... A game I play called Trailmakers ran into this problem with Unity. They solved it by doing interprocess communication between two separate Unity instances.... it took many many updates before it played well, multiple reworks of the "physics baking" that happens when you finish building a vehicle, etc.

Don't be like Unity, do it right for 4.0! Multi-threaded on-CPU Bullet!

[Calinou]

Closing, as the Bullet module has been removed from the master branch by godotengine/godot#58946 (leaving GodotPhysics as the only option in core). Since multi-threading Bullet won't be done in an official capacity for 3.x, this effectively means that readding Bullet support with GDExtension and making it multi-threaded will have to be third-party efforts.

It's still possible to make GodotPhysics more multi-threaded, but this probably will only be done for a future 4.x release, not 4.0: #483