Matrix3x2.Invert doesn't actually have tolerance

[lcsondes]

runtime/src/libraries/System.Private.CoreLib/src/System/Numerics/Matrix3x2.cs

Line 542 in 22b8d9a

if (MathF.Abs(det) < float.Epsilon)

Due to the definition of float.Epsilon this if will only work for ±0.0f (possibly not even that on ARM with no denormals), but the way the code was written suggests that there's some tolerance involved.

See also https://docs.microsoft.com/en-us/dotnet/api/system.single.epsilon#remarks

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

---

runtime/src/libraries/System.Private.CoreLib/src/System/Numerics/Matrix3x2.cs

Line 542 in 22b8d9a

if (MathF.Abs(det) < float.Epsilon)

Due to the definition of float.Epsilon this if will only work for ±0.0f (possibly not even that on ARM with no denormals), but the way the code was written suggests that there's some tolerance involved.

See also https://docs.microsoft.com/en-us/dotnet/api/system.single.epsilon#remarks

Author:	lcsondes
Assignees:	-
Labels:	area-System.Numerics, untriaged
Milestone:	-

[PranavSenthilnathan]

The check looks like it's just to make sure the determinant isn't zero. Another instance of this pattern actually has a comment:

runtime/src/libraries/System.Private.CoreLib/src/System/Numerics/Matrix4x4.Impl.cs

Lines 1276 to 1277 in 5d69e2d

	// Check determinate is not zero
	if (float.Abs(det) < float.Epsilon)

@tannergooding Is there any downside to just changing this to if (MathF.Abs(det) == 0.0f)?

[tannergooding]

Its functionally the same as the current code, but that is itself potentially a bug/issue and likely not the "right" fix.

What the code is "meant" to be doing is finding cases where the matrix is "not invertible" and while for strict math that is determinant != 0, it doesn't always work out the same with IEEE 754 floating-point due to the precision limitations.

A "correct" implementation would find the appropriate cutoff and treat values below that threshold as "not invertible", where that threshold can sometimes differ based on the matrix. However, doing that is far too expensive for the use-cases that this type is designed for. Given the cases the type is designed for, more ideally this API would have a signature closer to what GLM, DirectX Math, HLSL, and GLSL provide which includes the result being generally undefined if the value is not inversible, such that it isn't returning a bool at all.

However, since we need to work with what we have the more correct thing would be to define a general epsilon that works for most typical scenarios. A better check would be if (float.IsSubnormal(det)) as that includes everything that is very close to zero. But a more appropriate check is likely 0.00001f, which is the "epsilon" value used by many other matrix functions in the core libraries I listed above. 0.0000001f is likely the smallest value that would be "acceptable" as it produces a reciprocal of 10000000 (which has a log2 of 23.2 and pushes up against the precision limitations of float, which has 23+1 mantissa bits).

[PranavSenthilnathan]

Matrix4x4 can use the same change: here