Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle masked reverse mode atomic append #1691

Open
jeremiedb opened this issue Nov 27, 2023 · 8 comments
Open

Handle masked reverse mode atomic append #1691

jeremiedb opened this issue Nov 27, 2023 · 8 comments

Comments

@jeremiedb
Copy link

jeremiedb commented Nov 27, 2023

The following minimal function results in a segfault (run on Enzyme's main as of 2023-11-26, Julia v1.10.0-rc1). If removing the @turbo annotation, the autodiff works fine.

using LoopVectorization
using Enzyme

function my_conv_B(x, w)
    y = zero(x)
    @turbo for hi in axes(x, 2)
        for k in axes(x, 1)
            y[k] += w[k] * x[k, hi]
        end
    end
    return y
end
x = rand(Float32, 3, 5);
w = rand(Float32, 3);
y = my_conv_B(x, w);
loss_B(x, w) = sum(my_conv_B(x, w))
dw = zero(w);
loss_B(x, w)
grads = Enzyme.autodiff(Reverse, loss_B, Const(x), Duplicated(w, dw));

Below is the launch command along the resulting error info.
One thing I noticed is the reference to /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578. Is it expected to point to LLVM under what appears like a general system path raher than a Julia specific one? I'm on Ubuntu 23.10 and Julia was installed in a generic way (untar the official binaries under home directory and creation of sym link).

jeremiedb@client-X570S-AORUS-PRO-AX:~/github/ADTests.jl$ julia --project=@. --threads=4 experiments/conv-v3-lv.jl 
julia: /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From*) [with To = llvm::IntegerType; From = llvm::Type]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

[13573] signal (6.-6): Aborted
in expression starting at /home/jeremiedb/github/ADTests.jl/experiments/conv-v3-lv.jl:44
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fac7ce267da)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
cast<llvm::IntegerType, llvm::Type> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578 [inlined]
computeExitLimitFromICmp at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:432
computeExitLimitFromCondImpl at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:304
computeExitLimitFromCondCached at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:159
computeExitLimitFromCond at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:52
computeExitLimit at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:102
getContext at /workspace/srcdir/Enzyme/enzyme/Enzyme/CacheUtility.cpp:555
forceAugmentedReturns at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8198
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2350
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5176
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6183
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:301 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:582
EnzymeCreatePrimalAndGradient at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/api.jl:141
unknown function (ip: 0x7fabed3931db)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
enzyme! at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:3025
unknown function (ip: 0x7fabed38de79)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#codegen#447 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:4649
codegen at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:4232 [inlined]
_thunk at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5201
_thunk at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5201 [inlined]
cached_compilation at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5235 [inlined]
EnzymeAD/Enzyme.jl#484 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5301
JuliaContext at /home/jeremiedb/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
unknown function (ip: 0x7fabed33c7b5)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#s1040#483 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5253 [inlined]
#s1040#483 at ./none:0
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
GeneratedFunctionStub at ./boot.jl:600
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_call_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2883
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_35717.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
jl_type_infer at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2478 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2366
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2884 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
do_call at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:125
eval_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:222
eval_stmt_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:173 [inlined]
eval_body at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:616
jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
_include at ./loading.jl:2130
include at ./Base.jl:489
jfptr_include_46395.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82758.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
true_main at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
unknown function (ip: 0x7fac7ce280cf)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 29085878 (Pool: 29035582; Big: 50296); GC: 38
Aborted (core dumped)
@wsmoses
Copy link
Member

wsmoses commented Nov 28, 2023

Should be fixed by: #1563

Requires jll bump

@jeremiedb
Copy link
Author

Thanks for the quick consideration! Unfortunately, the same error is returned following upgrade to latest version:

[[deps.Enzyme]]
deps = ["CEnum", "EnzymeCore", "Enzyme_jll", "GPUCompiler", "LLVM", "Libdl", "LinearAlgebra", "ObjectFile", "Preferences", "Printf", "Random"]
git-tree-sha1 = "4e9d7571ccae47ee15f9ef40f09827323187c022"
repo-rev = "main"
repo-url = "https://github.com/EnzymeAD/Enzyme.jl.git"
uuid = "7da242da-08ed-463a-9acd-ee780be4f1d9"
version = "0.11.10"

[[deps.EnzymeCore]]
deps = ["Adapt"]
git-tree-sha1 = "2efe862de93cd87f620ad6ac9c9e3f83f1b2841b"
uuid = "f151be2c-9106-41f4-ab19-57ee4f262869"
version = "0.6.4"

[[deps.Enzyme_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "fc50e25508991df186a913d96970113fd04ce5f6"
uuid = "7cc45869-7501-5eee-bdea-0790c847d4ef"
version = "0.0.95+0"

@wsmoses
Copy link
Member

wsmoses commented Nov 28, 2023 via email

@jeremiedb
Copy link
Author

Are you referring to https://github.com/JuliaBinaryWrappers/Enzyme_jll.jl?
I noted v0.0.95+0 was released yesterday and I used it in the above test. Should I look forward v0.0.96+0?

@wsmoses
Copy link
Member

wsmoses commented Nov 29, 2023 via email

@wsmoses
Copy link
Member

wsmoses commented Nov 30, 2023

Jll bump is merged

@wsmoses wsmoses closed this as completed Nov 30, 2023
@jeremiedb
Copy link
Author

@wsmoses With Enzyme v0.11.11, the above my_conv_B still fails to differentiate, but only if executed in an environment where threads > 1. The following shows the stacktrace, where I think the key hint is the "Assertion `!mask' failed" mentionned at the beginning. Let me know if that looks of a different nature and should be opened in a seperate issue.

jeremiedb@client-X570S-AORUS-PRO-AX:~/github/ADTests.jl$ julia --project=@. --threads=2 experiments/conv-v3-lv.jl 
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:931: void DiffeGradientUtils::addToInvertedPtrDiffe(llvm::Instruction*, llvm::Value*, llvm::Type*, unsigned int, unsigned int, llvm::Value*, llvm::Value*, llvm::IRBuilder<>&, llvm::MaybeAlign, llvm::Value*): Assertion `!mask' failed.

[45830] signal (6.-6): Aborted
in expression starting at /home/jeremiedb/github/ADTests.jl/experiments/conv-v3-lv.jl:44
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f514e8267da)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
addToInvertedPtrDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:931
addToInvertedPtrDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:1231
visitLoadLike at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:687
handleAdjointForIntrinsic at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:3666
visitIntrinsicInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:3610
visitDbgInfoIntrinsic at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:208 [inlined]
visitDbgVariableIntrinsic at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:206 [inlined]
visitDbgDeclareInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:203 [inlined]
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:289 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5664
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6286
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:301 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:582
EnzymeCreatePrimalAndGradient at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/api.jl:141
unknown function (ip: 0x7f50ce39342b)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
enzyme! at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:3074
unknown function (ip: 0x7f50ce38e0c9)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#codegen#448 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:4698
codegen at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:4281 [inlined]
_thunk at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5272
_thunk at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5272 [inlined]
cached_compilation at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5306 [inlined]
EnzymeAD/Enzyme.jl#485 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5372
JuliaContext at /home/jeremiedb/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
unknown function (ip: 0x7f50ce33ca05)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#s1038#484 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5324 [inlined]
#s1038#484 at ./none:0
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
GeneratedFunctionStub at ./boot.jl:600
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_call_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2883
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_35717.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
jl_type_infer at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2478 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2366
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2884 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
do_call at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:125
eval_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:222
eval_stmt_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:173 [inlined]
eval_body at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:616
jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
_include at ./loading.jl:2130
include at ./Base.jl:489
jfptr_include_46395.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82758.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
true_main at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
unknown function (ip: 0x7f514e8280cf)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 30464515 (Pool: 30412228; Big: 52287); GC: 42
Aborted (core dumped)

@wsmoses wsmoses reopened this Jan 2, 2024
@wsmoses
Copy link
Member

wsmoses commented Feb 10, 2024

Upon the next jll bump this error message will now become:


Unimplemented masked atomic fadd for ptr:  %"ptr.1.i331'ipc_unwrap" = inttoptr i64 %81 to <8 x float>*, !dbg !392 dif:  %681 = load <8 x float>, <8 x float>* %"res.i333'de", align 32, !dbg !302 mask:   %m.i = icmp uge <8 x i32> %v.i, <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>, !dbg !166 orig:   %res.i333 = call <8 x float> @llvm.masked.load.v8f32.p0v8f32(<8 x float>* %ptr.1.i331, i32 noundef 4, <8 x i1> %m.i, <8 x float> noundef zeroinitializer) EnzymeAD/Enzyme.jl#21, !dbg !273, !alias.scope !205


Stacktrace:
 [1] macro expansion
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997
 [2] __vload
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997
 [3] _vload
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/strided_pointers/stridedpointers.jl:105
 [4] macro expansion
   @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107
 [5] _turbo_!
   @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107
 [6] my_conv_B
   @ ./REPL[3]:3


Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:1283
  [2] macro expansion
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997 [inlined]
  [3] __vload
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997 [inlined]
  [4] _vload
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/strided_pointers/stridedpointers.jl:105 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107 [inlined]
  [6] _turbo_!
    @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107 [inlined]
  [7] my_conv_B
    @ ./REPL[3]:3
  [8] loss_B
    @ ./REPL[7]:1 [inlined]
  [9] diffejulia_loss_B_3362wrap
    @ ./REPL[7]:0
 [10] macro expansion
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:5374 [inlined]
 [11] enzyme_call
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:5052 [inlined]
 [12] CombinedAdjointThunk
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:4994 [inlined]
 [13] autodiff
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:215 [inlined]
 [14] autodiff
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:238 [inlined]
 [15] autodiff(::ReverseMode{false, FFIABI}, ::typeof(loss_B), ::Const{Matrix{Float32}}, ::Duplicated{Vector{Float32}})
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:224
 [16] top-level scope
    @ REPL[10]:1
We should still add this, though now it is a purely Enzyme, not Enzyme.jl issue.

However FYI there are EnzymeRules for NNlib.conv if you'd prefer that to hand rolling your own (or using that custom rule for your conv)

@wsmoses wsmoses changed the title Crash on basic LoopVectorization operation Handle masked reverse mode atomic append Feb 10, 2024
@wsmoses wsmoses transferred this issue from EnzymeAD/Enzyme.jl Feb 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants