3D Physics not utilizing all CPU cores

[starwolfy]

Godot version:
Godot Engine v3.2.2.stable.mono.official

OS/device including version:
Windows 10 64bit, i7 7700k (4 cores, 8 threads), GLES3

Issue description:
I am benchmarking my game project which acts as a server (not actual server build, but no camera and 0x0 resolution) which does physics calculations for each player that is connected. It includes 4 areas and one kinematic body per player, my physics tickrate is 16tick/s.
When the player count reaches about 36 players the server starts lagging to the point where it becomes unplayable, because each frame takes too long to keep up with the 16tick/s physics rate. The Godot profiling tool shows that this is caused by the physics calculations being done by the Godot engine itself, actually my _physics_process scripts also slow it down some more but I have tested this without the scripts attached to the physics processes and only two threads are working at a 100% capacity when the server already starts lagging behind with physics (1 thread at 100% capacity is doing a worker thread logic so its not related to physics).
I have 1 thread reserved with the Thread class in GDScript for a different function, but that still leaves plenty of threads which are unused as seen in this screenshot:

First of all, ignore the total 120ms frame time, this does not add up because it is caused by screen capture software that I launched at the time of taking the screenshot. But as you can see, the "Physics Time" goes up way too high (50ms-70ms and higher the more players connect) and causes the target framerate (62.5ms) to no longer be achieved while my CPU only has 2 very busy threads while the remaining threads are not doing much (1 busy thread is from the Thread API). Ultimately resulting in only a 20-30% total CPU usage while my server is already lagging behind and becomes unplayable.

I am really wondering if this is the intended way multi-threading works with Godot physics or if there is a way to unlock more CPU cores for the (Bullet) physics engine to use?
This is not a Linux server build, which I do intend on using to really properly benchmark this project to the maximum but am running into problems as referenced in #38268 . Perhaps not all CPU cores/threads are utilised because some are dedicated towards processing graphics?

Steps to reproduce:
Put lots of areas and kinematic bodies in a Godot project and move/slide them around with code at run-time, also let them monitor collisions, overlaps etc.

Minimal reproduction project:
None.

[fire]

I have a commit for this. https://github.com/godot-extended-libraries/godot-fire/commit/d6327709b568594949cbb3e98eb97126af67b17c.patch but it crashes on my test scene.

godotengine/godot-demo-projects#483 has a minimum reproduction project.

@AndreaCatania is looking at direct bullet physics queries for increased performance.

[starwolfy]

@fire I will try out your patch and report back here when I have tested it with my project. Also, it is great to see that attention and time are already being spent on this multi-threading physics issue.

If you guys can get the Bullet physics engine to utilize all threads that are assigned this engine can handle A LOT of stuff and would serve as an amazing basis for game server frameworks that allow for very large player counts!

In my case 32 players at 16tick/s are already being handled with 1 core, imagine if that CPU utilization can go up close to 100%. We are talking about a potential 100 player+ server hosted on a single 8-core machine with accurate enough physics for many many games. Not to mention that 16tick/s could be lowered in my case and that there are server solutions out there that offer more than 8 CPU cores.

Keep up the good work!

[pouleyKetchoupp]

@fire Do you have a PR for this? It looks like it's a change that could be part of core, at least as an option for optimization.

[fire]

I can make one for 3.2, but it's not currently on master.

[fire]

@Classy11 can you write a proposal for this in https://github.com/godotengine/godot-proposals ?

[starwolfy]

@fire Yessir, tomorrow I will post my proposal and notify the rest about it here as well.

[starwolfy]

Here is the proposal:
godotengine/godot-proposals#1155

[lawnjelly]

36 players at 16 ticks per second doesn't sound that much, it is the equivalent to running 3.6 players at 160 ticks per second, which shouldn't be an insurmountable load.

While the threading physics proposal may or may not be desirable as an option for physics, it may well be a wild goose chase in your particular situation (it sounds a lot like displacement activity). If it were me, I would be investigating why physics is taking so long even single threaded. Perhaps the physics world is too complex, or the shape of colliders - that is if physics is actually the bottleneck. Profiling with an external profiler will help pin this down, the Godot profiler will only give you rough info.

If you aren't able to do this, then a minimum reproduction project may help.

[starwolfy]

@lawnjelly you are a 100% correct. The past two days I've been reorganising my code and physics bodies entirely. I now also have as little code as possible in the _physics_process and this has reduced the physics tick time tremendously.
I have misinterpreted the performance stats before and edited my issue accordingly last evening. The max amount of physics bodies (including players) is much, much higher now. Now the project is capable of having over a hundred physics entities. So that includes over a hundred players as well.
This actually resolves the physics performance problem I've run into, despite this my case on 3D multi-threaded physics remains for a more long term vision of my project, but this is not a direct problem I'm struggling with at the moment anymore.
When a certain amount of physics entities are reached the physics thread will be at max capacity. Spawning 3D PhysicsServers won't be able to help much as they run on the same thread as well, this does put the maximum amount of allowed physics entities on the performance of a single thread.
However I am thankful that the single threaded physics performance has now proven to be very fast already. I also understand that my use case is on the extreme end: a godot server supporting a 100 players with an amount of game entities that come with it. In which case being limited to one thread for all the physics will become a game limit. Godot has an epic Linux server build already, that combined with performant multi threaded physics would be so powerful as one could host game servers on epic scales.

[Zylann]

Here is another proposal that is related, albeit more complex as it'd require managing multiple 3D PhysicsServers: godotengine/godot-proposals#483

I don't see how you get to this conclusion. This proposal basically asks for multithreading by being able to run the server in a thread, or at least have thread safety, not having multiple ones (see how it's done with other servers), only with a particular focus on body creation.
Moving physics to be solely multithread might not be wanted either.

[starwolfy]

@Zylann I have misunderstood it then, I have edited my comment!

[NathanLovato]

What's the status of this issue? Is it related to documentation?

[starwolfy]

@NathanLovato So the Godot performance monitor includes the processing time of all _physics_process scripts and adds that to the general Physics Time duration. Also, working with many larger 3d Areas that are also moving around seems to really impact the physics duration too.

After having improved my _physics_process code and after having gotten rid of 3d areas; I personally am not in need of multi-threaded Godot physics anymore. On my end, this issue has been resolved and was more a misinterpretation done by me.

I thought that the Godot physics engine was bottlenecking my project and that is why I posted single-threaded physics as an issue, but it turned out that making a different approach on physics in my project has resolved the physics bottleneck for now.

There has been a proposal written for this issue and it has received some comments of people that are interested so there are some others who do seem to run into this bottleneck.
I would say that this issue is relevant for those who get bottlenecked by the physics engine and have also created efficient physics setups and scripts ontop of that, so they cannot push it anymore. But to me, the OP, this issue is not relevant anymore.

[NathanLovato]

Thanks, I'll close the ticket then.