Node with physical bones spams rotref_posscale_decomposition: Condition "!m.is_diagonal()" error when it's added to or removed from scene

[Miziziziz]

Godot version: 3.3.2 stable

OS/device including version: Windows 10

Issue description:

This error is spammed when I run the scene, or add or remove nodes containing physical bones from the scene:

E 0:00:00.908   rotref_posscale_decomposition: Condition "!m.is_diagonal()" is true. Returned: Vector3()
  <C++ Source>  core/math/basis.cpp:300 @ rotref_posscale_decomposition()

It only happens with certain skeleton setups though, my humanoid skeletons don't throw this error. Maybe because I have IK bones included in the skeletons of the non humanoid characters?
Probably related to #24968

Minimal reproduction project:
RotRefError.zip
open the main scene and you'll get errors, you'll also get them when you close the scene