One thing that could be a reason (for sure not the only one): allInspectorNodes add a "self" node:

allInspectorNodes
	"Answer a list of attributes as nodes"

	^ { StInspectorSelfNode hostObject: self }, 
		self inspectorNodes

this then is used in StInspectorNode>>#children, while the debugger even filters it out again in some cases: StDebuggerContext>>#receiverNodes

receiverNodes

	^ self context receiver allInspectorNodes reject: [ :node | 
		  node label = 'self' ]

For speed, this just doubles the amount of nodes that are created...

I just tested myself. If we remove the selection remembrance mechanism in the object inspector,it gets 2 times faster.

With the inspector selection remembrance mechanism:

Without the inspector selection remembering mechanism:

The time spent in the inspector update is thus divided by 3.

Now, most of the time is due to the update of the model of the inspector (makes up for 40% of the update time). I will investigate to see if it can be fastened.

Regarding, the inspector selection remembrance mechanism, I think we could remove it. I don't think it is useful and I'm not convinced with its implementation. It seems to put in cache the same information twice, under different keys that aren't even accurate.

One thing that could be a reason (for sure not the only one): allInspectorNodes add a "self" node:

allInspectorNodes
	"Answer a list of attributes as nodes"

	^ { StInspectorSelfNode hostObject: self }, 
		self inspectorNodes

this then is used in StInspectorNode>>#children, while the debugger even filters it out again in some cases: StDebuggerContext>>#receiverNodes

receiverNodes

	^ self context receiver allInspectorNodes reject: [ :node | 
		  node label = 'self' ]

For speed, this just doubles the amount of nodes that are created...

Regarding this, the debugger rejects the self node by default and adds one whose host object is the receiver of the home context.
However, isn't the receiver of a context always identical to the receiver of its home context?
Anyway, if the self node is rejected, then I think #receiverNodes should simply call #inspectorNodes instead of #allInspectorNodes