Generalized dot test sensitve to numerical precision

[martinholters]

The tests at

Compat.jl/test/runtests.jl

Lines 171 to 173 in e667af7

	@test dot(x, A, y) ≈ dot(A'x, y) ≈ *(x', A, y) ≈ (x'A)*y
	@test dot(x, A', y) ≈ dot(A*x, y) ≈ *(x', A', y) ≈ (x'A')*y
	elty <: Real && @test dot(x, transpose(A), y) ≈ dot(x, transpose(A)*y) ≈ *(x', transpose(A), y) ≈ (x'*transpose(A))*y

sometimes fail for Float32 eltype with an error that can probably be attributed to numerical precision issues, e.g.

  Expression: dot(x, transpose(A), y) ≈ dot(x, transpose(A) * y) ≈ x' * transpose(A) * y ≈ (x' * transpose(A)) * y
   Evaluated: -0.0029978678f0 ≈ -0.0029971108f0 ≈ -0.0029982424f0 ≈ -0.0029982424f0

The same tests appear in the LinearAlgebra tests:
https://github.com/JuliaLang/julia/blob/29826c2c0893488dce0d4ee92b457f7f1a586dfb/stdlib/LinearAlgebra/test/generic.jl#L472-L474
So Julia is probably effected as well.

The probability of failing seems to be quite low (around 0.02% in a brief experiment), but it would still be nice to sort this out. CC @dkarrasch who added these tests in julia and @mcabbott who brought them here.

[dkarrasch]

Perhaps we should set a random seed somewhere close to the tests, and change that in case RNGs start to produce bad numbers.

[mcabbott]

I thought testsets did set a consistent seed, although the result isn't fixed across versions.

For example, this passes for me every time on Julia 1.5.0-rc1, and fails every time on 1.4.2:

using Test
@testset "stuff" begin
    @show rand()
    @test rand()>0.5
end

[stevengj]

Any dot product test will be badly conditioned (very sensitive to roundoff error and hence likely to fail the ≈ test of relative error) if you pick two vectors that happen to be nearly orthogonal.

Fortunately, this is a low-probability event, so it should suffice to set the seed. (Even if Julia changes the random-number generator in a future version, there is a low probability that we will need to change the seed.)

(Similarly for pretty much any ≈ test that involves computation with random numbers.)

@mcabbott, my understanding is that @testset sets the seed to the global seed, but this does not result in a deterministic test across multiple runs unless you call Random.seed! explicitly in the global scope first.

[mcabbott]

Thanks, sadly I did not get as far as actually reading the docs...

Should be simple to fix the seed though, will make a PR unless someone beats me to it.