Remote Scene Tree is unusably slow in large scenes

[TokisanGames]

(Update: 4.1 and 4.2 have gotten progressively worse. Now, the editor is entirely useless, even without the remote scene tree open. See comments below)

Godot version

Godot 4.1.beta3

System information

Windows 11/64, RTX 3070, Vulkan

Issue description

My largest scene has 80,000 nodes in it and we have countless more to add. Foliage, rocks, and houses made up of individual components. Running the game, editing the SceneTree, and debugging line by line is fine, but viewing the remote scene tree is unusably slow.

Click on something in the remote scene tree, even expanding/collapsing nodes with few children, clicking anything, or typing into the search filter, or stopping the program, and wait 10-20 seconds before you get a response.

Even if the remote scene tree is not active, the editor menu is very sluggish, though not as bad. Maybe 1s per response.

I wrote an MRP that generates meshes in the scene tree. Browsing the large tree in the editor is no problem. Running the objects in game is no problem. Only the remote tree and editor menus when running the game has a problem.

I found that even starting around 3000 the remote tree starts to get sluggish and gets progressively worse from there. 90k is unusable. Interestingly at around 100k nodes in my running scene (108,935) I started getting this error message repeated on the console and the remote scene tree operated at normal speed, though with maybe 70k nodes in the remote tree instead of the total.

ERROR: Condition "err != OK || size > out_buf.size() - 4" is true. Continuing.
   at: _write_out (core/debugger/remote_debugger_peer.cpp:106)

I've been struggling with this issue for a long time, all of the GD4 builds, and maybe before. Still present in 4.1.beta3. I thought it was caused by #68232, but no. This MRP has only the automatic material and MeshInstance3D. It's not a custom shader issue, nor physics, gpuparticles, nor rendering. It's slow even if the objects are invisible.

Steps to reproduce

• Run the MRP
• Set per_side to one of:
  • 30 for 28k objects (3s/click)
  • 45 for 91k objects (painful - this is my life)
  • 47 for 106k to overflow the buffer and it works at normal speed
• Generate. Wait 20-30 seconds.
• Expand/collapse nodes in the tree and observe it is normal speed
• Run the game
• Switch back to the editor, enable the remote tree and see its lag. With remote tree up, click on the menus and see how it kills everything. Good luck doing anything for 10+ seconds. You can click back to local for a usable editor.

Minimal reproduction project

4.1b3 project file, but everything works in 4.0 if you set the message limit to 102400.

test_slow_remote_tree.zip

[451507056]

I also ran into this problem, I have 125k nodes, click on the remote scene can not see the scene tree. 4.1.1.stable

[Calinou]

This is likely due to the same cause as #82105.

[TokisanGames]

This has gotten worse. When I reported it, the editor was functional while running a game until you enabled the remote scene tree. Now in 4.1.2 while running my game, with a scene w/ 80k nodes open, the editor is half useless. The code editor, local scene tree and inspector are ok. All of the menus, open window, and loading/closing a scene are slow. The visual profiler stops every few seconds anytime you hover over something while the tooltip loads.

I built 4.1.2 with module_text_server_fb_enabled=yes module_text_server_adv_enabled=no and now the editor interface is normal with a game running. The menus popup and the profilers don't halt when loading the tooltip. Opening and closing scenes is normal.

Then I enabled the remote scene tree and the editor performance fell off a cliff. Back to 20-30s per click in the tree or editor menus. So the advanced text server is not the cause of this. #82105 might indeed have the same cause. But the advanced text server is a whole other separate problem.

[Raymonf]

I've got a project that dynamically generates nodes from a binary file - the result is around 22,000 nodes according to what's obtained from the debugging server. Assuming the issue is the same, this seems to be caused by the fact that the (local state) remote tree gets cleared and recreated every single time the remote tree is updated.

godot/editor/debugger/editor_debugger_node.cpp

Lines 627 to 633 in d443f12

	void EditorDebuggerNode::_remote_tree_updated(int p_debugger) {
	if (p_debugger != tabs->get_current_tab()) {
	return;
	}
	remote_scene_tree->clear();
	remote_scene_tree->update_scene_tree(get_current_debugger()->get_remote_tree(), p_debugger);
	}

The create_item call in the loop that recreates the tree is a significant hotspot in large scenes, to perhaps nobody's surprise:

godot/editor/debugger/editor_debugger_tree.cpp

Lines 157 to 160 in d443f12

	// Add this node.
	const SceneDebuggerTree::RemoteNode &node = p_tree->nodes[i];
	TreeItem *item = create_item(parent);
	item->set_text(0, node.name);

I assume a way to do this that would be faster in large scenes would be to only send changed nodes or even the changed attributes, rather than the entire tree. But it's not clear to me how or if this could be implemented in Godot due to my lack of experience with the codebase.

[ghost]

I'm not sure if this is the exact same thing, it sounds like it. It seems to happen from having too many Node3D open when I go to remote. The entire editor, once in remote, will not respond every 10ish seconds for a few seconds. If I click to expand, or just do anything, I have to pray and hope it will respond to my click in a few seconds, then repeat the process again if I have to do another thing. Dragging a slider or anything works in the same way, click...delay...drag...nothing...then it moves. Any amount of real debugging the scene in remote is nearly impossible because of this.

This is happening on 4.2beta6 and 4.2 RC1. The Node3D count is about 570 nodes. Removing the nodes from the scene fixes the issue.

[lordhaywire]

Does anyone have a work around for this issue, or is there any word when it will be addressed? If this is the same issue as #82105 can they merge them? Or should I post on both issues. I don't know the etiquette. I also have a project I can upload which only generates 5000 nodes and makes the remote unusable.

[ghost]

Does anyone have a work around for this issue,

I have not discovered any workaround. I needed to position objects so I did it in a copy of my game with just a few objects. Not ideal at all.

[svipal]

Still an issue on current 4.2.1. The main scene in my current project is 40k nodes (most of whom are dynamically generated). clicking on the remote tree tab; and once in it, expanding/collapsing nodes etc .. freezes the whole editor for several seconds, making it sadly unusable .

[Calinou]

We could probably work around this by making the remote scene tree something you update manually by clicking a button in the editor, or by pressing a global¹ keyboard shortcut (as opposed to having it update once every second).

While this is less convenient, this would also allow revisiting #18998 without performance worries due to the remote scene tree.

Footnotes

1. By global keyboard shortcut, I mean one that's also defined in the project so that it works if you're currently focused on the project window. This is how F8 works when the project is focused. ↩

[TokisanGames]

At the moment it's entirely unusable for most of my game scenes. The tree is not changing frequently, once loaded. The only changes occur if an enemy is killed, or we load a different scene. It could update with only changes rather than getting all, every second. But a manual update would be a positive step forward towards usability.

Once a node is selected, we still need the inspector to update continually so I can monitor values there, change them, and see their updates in game.

[Stalker2106]

Just for the record, I'm experiencing the exact same behaviour, very slow remote tree view, around 20s between each clicks for a scene of just about 16400 nodes, that are absolutely static.
Running on 4.2.2 stable.
I recommend putting a manual refresh button as suggested, and a disable automatic refresh in editor options.
The scene run smoothly performance-wise though.

[YYF233333]

I did some profiling, and the main issue with this problem is not that the remote scene tree rebuilds every time it's modified, but that during the rebuild process, EditorNode::get_class_icon is called, which check the existence of path remap file every time. After commenting out this part, the remote scene tree's performance returned to normal.

Here's a simple workaround build (node icons are missing). If anyone is still suffering from this issue, feel free to give it a try.
https://drive.google.com/file/d/1wObYh0WXGf-4PIDFwoNvXm0ankDFwTdL/view?usp=sharing

And based on what I find in #100109, this is likely to be a Windows specific problem. Has anybody tested on Linux or MacOS?

[HolonProduction]

Just want to cross link this with #98666, which is also caused by the changes which #88427 made to get_class_icon.

My comment from over there:

Maybe a lightweight check on the script name before calling ResourceLoader.exists could prevent this. But personally I don't think get_class_icon should handle script paths at all, from its name I would assume it only handles class_name and native class names.

[YYF233333]

#88427 was merged in October but this issue is opened in 2023, seems there are bunches of code needs to be optimized, not only that one. 😂

[HolonProduction]

I based this on the profiling you did. But yeah this is probably not easily traceable to just one thing.

[YYF233333]

#101435 should resolve regression introduced by #88427 completely. Now bottleneck moves to Tree operations, we may have to adopt something like #99700 to tackle this.

[samuel-walker]

To add more data, I encountered this error on Mac with Godot v4.4.beta3.

E 0:01:17:0547   _write_out: Condition "err != OK || size > out_buf.size() - 4" is true. Continuing.
  <C++ Source>   core/debugger/remote_debugger_peer.cpp:108 @ _write_out()

This error fires once a second while the Remote tree is open. The Remote tree is blank. It has 84,787 mostly static nodes in it. For scenes with fewer nodes, the Remote tree is viewable.

Godot v4.4.beta3 - macOS Sonoma (14.6.1) - Multi-window, 1 monitor - Metal (Forward+) - integrated Apple M1 Pro (Apple7) - Apple M1 Pro (10 threads)

Thanks.