Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) #72

DilumAluthge · 2020-12-19T02:36:49Z

Fixes #61

Once these tests are working on Julia nightly, we should reenable them.

I've opened #73 so that we don't forget to reenable these tests later.

Please note: we only skip these tests on CI. When you run the tests locally, these tests will not be skipped.

DilumAluthge · 2020-12-19T08:29:42Z

@mcabbott Looks like it's not just the KernelAbstractions tests that are timing out on Julia nightly. Group 3 is also timing out.

mcabbott · 2020-12-19T08:40:15Z

Yes, I saw that LoopVectorization tests in #71 timed out, on nightly, with lots of "WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead)." I presume this is some interaction with JuliaLang/julia#38136? Cc @chriselrod

DilumAluthge · 2020-12-19T09:28:54Z

@mcabbott Any idea what is causing the test failure in https://github.com/mcabbott/Tullio.jl/pull/72/checks?check_run_id=1581043675 (Julia 1.4, one thread, test group 2)?

┌ Info: Beginning to run test file
│   i = 2
└   file_i = "/home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl"
KernelAbstractions + gradients: Test Failed at /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79
  Expression: g3 ≈ (_gradient((x->(sum(#= /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79 =# @tullio($(Expr(:(:=), :(y[ind3[i]]), :(i ^ 2 * x[i])))));)), ones(size(ind3))))[1]
   Evaluated: [1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 0.0, 64.0, 81.0, 100.0, 121.0] ≈ [1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0, 100.0, 121.0] (tracked)
Stacktrace:
 [1] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79
 [2] include(::String) at ./client.jl:439
 [3] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:18
 [4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:12
Test Summary:                  | Pass  Fail  Broken  Total
KernelAbstractions + gradients |   77     1       1     79
  @inferred                    |    4                    4
  from TensorTrace             |   16                   16
  min/max                      |   18                   18
  mod, clamp, pad              |    3                    3
  finalisers                   |    4                    4
ERROR: LoadError: LoadError: Some tests did not pass: 77 passed, 1 failed, 0 errored, 1 broken.
in expression starting at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:11
in expression starting at /home/runner/work/Tullio.jl/Tullio.jl/test/runtests.jl:35
ERROR: Package Tullio errored during testing
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1510
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:316
 [4] #test#68 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [5] #test#65 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:294 [inlined]
 [6] top-level scope at none:1

DilumAluthge · 2020-12-19T09:34:31Z

Seems like, for some reason, the ForwardDiff gradient (g3) has a 0.0 where the Tracker gradient has a 49.0.

DilumAluthge · 2020-12-19T09:39:59Z

Also, all of the group 5 tests are failing (1.4 and 1.5, one thread and two threads). E.g. https://github.com/mcabbott/Tullio.jl/pull/72/checks?check_run_id=1581043810:

┌ Info: Beginning to run test file
│   i = 5
└   file_i = "/home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl"
Test Summary:                         | Pass  Broken  Total
gradients: Tracker + TensorOperations |   62       1     63
Test Summary:                        | Pass  Broken  Total
gradients: Zygote + TensorOperations |   64       2     66
analytic: Error During Test at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30
  Test threw exception
  Expression: g1 ≈ (_gradient((x->(real(sum(#= /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30 =# @tullio($(Expr(:(:=), :(y[i, j]), :(x[i, k] * x[k, j]))))));)), x0))[1]
  MethodError: no method matching Strided.StridedView(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}})
  Closest candidates are:
    Strided.StridedView(!Matched::Array{S,N} where N) where S at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    Strided.StridedView(!Matched::Array{S,N} where N, !Matched::Tuple{Vararg{Int64,N}}) where {S, N} at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    Strided.StridedView(!Matched::Array{S,N} where N, !Matched::Tuple{Vararg{Int64,N}}, !Matched::Tuple{Vararg{Int64,N}}) where {S, N} at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    ...
  Stacktrace:
   [1] _native_contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:426
   [2] contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{Symbol,Symbol,Symbol}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:322
   [3] contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{}, ::Tuple{Symbol,Symbol,Symbol}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:89
   [4] (::var"#∇ℳ𝒶𝓀ℯ#332")(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Array{Complex{Int64},2}, ::Array{Complex{Int64},2}) at /home/runner/work/Tullio.jl/Tullio.jl/src/tensor.jl:157
   [5] (::Tullio.var"#233#234"{Tullio.Eval{var"#ℳ𝒶𝓀ℯ#331",var"#∇ℳ𝒶𝓀ℯ#332"},Tuple{Array{Complex{Int64},2}},Array{Complex{Int64},2}})(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /home/runner/work/Tullio.jl/Tullio.jl/src/grad/zygote.jl:8
   [6] (::Tullio.var"#2462#back#235"{Tullio.var"#233#234"{Tullio.Eval{var"#ℳ𝒶𝓀ℯ#331",var"#∇ℳ𝒶𝓀ℯ#332"},Tuple{Array{Complex{Int64},2}},Array{Complex{Int64},2}}})(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /home/runner/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
   [7] #322 at /home/runner/work/Tullio.jl/Tullio.jl/src/tensor.jl:88 [inlined]
   [8] (::typeof(∂(#322)))(::Int64) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface2.jl:0
   [9] (::Zygote.var"#41#42"{typeof(∂(#322))})(::Int64) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface.jl:40
   [10] gradient(::Function, ::Array{Complex{Int64},2}) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface.jl:49
   [11] _gradient(::Function, ::Vararg{Any,N} where N) at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:17
   [12] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30
   [13] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
   [14] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:28
   [15] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
   [16] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:24

mcabbott · 2020-12-19T09:40:28Z

The bug in "/test/gradients.jl:79" is that vcat(unique(rand(1:1024, 10)), 1) sometimes has 1 twice, I think. Finally fixed here:

https://github.com/mcabbott/Tullio.jl/pull/57/files#diff-aa2ea92ca835ecf86ced3e9f51716c9957d30921f425b2c7e711e8b03e2c0c43R100

DilumAluthge · 2020-12-19T09:45:46Z

The bug in "/test/gradients.jl:79" is that vcat(unique(rand(1:1024, 10)), 1) sometimes has 1 twice, I think. Finally fixed here:

https://github.com/mcabbott/Tullio.jl/pull/57/files#diff-aa2ea92ca835ecf86ced3e9f51716c9957d30921f425b2c7e711e8b03e2c0c43R100

Thanks! I have applied that fix to this PR.

DilumAluthge · 2020-12-19T10:45:11Z

Alright, so the only remaining test failures are in group 5 on Julia 1.4 and 1.5 (but not Julia nightly).

Any idea what the cause is?

…htly (the tests that are causing CI to time out)

chriselrod · 2020-12-19T13:01:18Z

Yes, I saw that LoopVectorization tests in #71 timed out, on nightly, with lots of "WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead)." I presume this is some interaction with JuliaLang/julia#38136? Cc @chriselrod

Didn't see that PR. Thanks for the heads up.
My code made extensive use of Varargss and of .parameters.
I'll build the latest Julia master and try to fix all of it in a 1.5-compatible way.

mcabbott · 2020-12-19T17:32:26Z

The error with TensorOperations might be a real bug, brought to light by re-ordering the tests. (Zygote's gradient for sum makes a FillArray, which @tensor doesn't like, but ought to be caught by a spacial case here, but perhaps that fails...) Will have a look.

However I think I'd rather do something simpler to temporarily skip tests on nightly, rather than making more groups, like so:
ebf5c70

I see you're also adding a test on nightly to .buildkite/pipeline.yml. That's not a bad idea although perhaps it may as well wait until KernelAbstractions has some hope of running there?

chriselrod · 2020-12-19T17:34:38Z

Any idea how to find the source of those warnings?
When running LoopVectorization's tests:

     Testing Running tests...
WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).Julia Version 1.7.0-DEV.122
Commit 676ccf4eaa (2020-12-18 19:53 UTC)

Would be helpful to have line numbers.
Loading Julia with --depwarn=yes or --depwarn=error and using LoopVectorization from the REPL don't show anything.

mcabbott · 2020-12-19T17:57:45Z

For me --depwarn=error gives the following when I run tests:

     Testing Running tests...
ERROR: LoadError: LoadError: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).
Stacktrace:
  [1] UnionAll(v::TypeVar, t::Any)
    @ Core ./boot.jl:254
  [2] top-level scope
    @ ~/.julia/packages/LoopVectorization/N0Zry/src/map.jl:2
  [3] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
  [4] include(x::String)
    @ LoopVectorization ~/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:1
  [5] top-level scope
    @ ~/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:57
  [6] include
    @ ./Base.jl:386 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1208
  [8] top-level scope
    @ none:1
  [9] eval
    @ ./boot.jl:369 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [11] top-level scope
    @ none:1
in expression starting at /Users/me/.julia/packages/LoopVectorization/N0Zry/src/map.jl:2
in expression starting at /Users/me/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:1

chriselrod · 2020-12-19T17:58:29Z

It turns out the warnings/errors only come up during precompilation.
I should have them fixed soon.
EDIT: Should be fixed on master. I'll probably tag a release in the next few hours.

DilumAluthge mentioned this pull request Dec 19, 2020

TODO: reenable ALL tests on Julia 1.6 and Julia nightly #73

Closed

DilumAluthge marked this pull request as draft December 19, 2020 02:37

DilumAluthge changed the title ~~TEMPORARILY disable KernelAbstractions tests on Julia 1.6 and Julia nightly~~ TEMPORARILY disable KernelAbstractions CPU tests on Julia 1.6 and Julia nightly Dec 19, 2020

DilumAluthge changed the title ~~TEMPORARILY disable KernelAbstractions CPU tests on Julia 1.6 and Julia nightly~~ Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) Dec 19, 2020

Repository owner deleted a comment from codecov-io Dec 19, 2020

Add more test groups, and TEMPORARILY disable some tests on Julia nig…

7590c39

…htly (the tests that are causing CI to time out)

DilumAluthge added 2 commits December 20, 2020 15:21

Update compat entries for LoopVectorization and VectorizationBase

a39d0eb

Update Project.toml

f0f0be0

DilumAluthge mentioned this pull request Dec 20, 2020

Still seeing the "WARNING: Wrapping Vararg..." warnings with LoopVectorization v0.9.9 and VectorizationBase v0.14.3 JuliaSIMD/LoopVectorization.jl#175

Closed

Update Project.toml

294b215

mcabbott mentioned this pull request Jan 23, 2021

More LoopVectorization tests & checks #57

Merged

mcabbott closed this in #57 Jan 23, 2021

DilumAluthge deleted the dpa/skip-kernelabstractions-on-julia-nightly branch March 26, 2021 06:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) #72

Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) #72

DilumAluthge commented Dec 19, 2020 •

edited

Loading

DilumAluthge commented Dec 19, 2020

mcabbott commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

mcabbott commented Dec 19, 2020 •

edited

Loading

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

chriselrod commented Dec 19, 2020

mcabbott commented Dec 19, 2020

chriselrod commented Dec 19, 2020

mcabbott commented Dec 19, 2020

chriselrod commented Dec 19, 2020 •

edited

Loading

Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) #72

Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out) #72

Conversation

DilumAluthge commented Dec 19, 2020 • edited Loading

DilumAluthge commented Dec 19, 2020

mcabbott commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

mcabbott commented Dec 19, 2020 • edited Loading

DilumAluthge commented Dec 19, 2020

DilumAluthge commented Dec 19, 2020

chriselrod commented Dec 19, 2020

mcabbott commented Dec 19, 2020

chriselrod commented Dec 19, 2020

mcabbott commented Dec 19, 2020

chriselrod commented Dec 19, 2020 • edited Loading

DilumAluthge commented Dec 19, 2020 •

edited

Loading

mcabbott commented Dec 19, 2020 •

edited

Loading

chriselrod commented Dec 19, 2020 •

edited

Loading