Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] GPU lower_thread_allreduce is_zero(index) check false #11898

Closed
ganler opened this issue Jun 25, 2022 · 0 comments
Closed

[Bug] GPU lower_thread_allreduce is_zero(index) check false #11898

ganler opened this issue Jun 25, 2022 · 0 comments
Labels
executor:graph runtime components of graph flow. src/runtime/graph_executor, python/tvm/contrib/graph_executor.py tir:transform TIR transforms: src/tir/transforms and python/tvm/tir/transforms type: bug

Comments

@ganler
Copy link
Contributor

ganler commented Jun 25, 2022 

import tvm
from tvm import relay
import numpy as np

"""
fn (%i0: Tensor[(1, 2, 1), float64], %i1: Tensor[(1, 1, 2), float64]) {
  %0 = broadcast_to(%i1, shape=[1i64, 1i64, 2i64]);
  %1 = maximum(%0, %i0);
  argmin(%1, axis=[2])
}
"""

i0 = relay.var("i0", shape=(1, 2, 1), dtype='float64')
i1 = relay.var("i1", shape=(1, 1, 2), dtype='float64')

v0 = relay.op.broadcast_to(i1, shape=relay.const([1, 1, 2], dtype="int64"))
v1 = relay.op.maximum(v0, i0)
v2 = relay.op.argmin(v1, axis=[2])

f = relay.Function([i0, i1], v2)

relay.create_executor('graph', device=tvm.cuda(), target='cuda').evaluate(f)

Expected behavior

Pass compile.

Actual behavior

Compiler failure at tir/transforms/lower_thread_allreduce.cc.

Core Logs [click to expand] 
"""
at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:741
  78: tvm::tir::StmtMutator::operator()(tvm::tir::Stmt)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:190
  77: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:240
  76: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  75: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  74: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#2}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:113
  73: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#2}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:113
  72: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AttrStmtNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:70
  71: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AttrStmtNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:257
  70: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  69: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  68: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  67: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  66: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  65: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:95
  64: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:311
  63: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  62: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  61: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  60: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  59: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  58: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:95
  57: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:311
  56: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  55: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  54: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  53: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  52: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  51: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:95
  50: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:311
  49: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  48: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  47: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  46: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  45: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:117
  44: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:95
  43: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AllocateNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:311
  42: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  41: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  40: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  39: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#2}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:113
  38: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#2}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:113
  37: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::AttrStmtNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:70
  36: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::AttrStmtNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:257
  35: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  34: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  33: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  32: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#13}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:124
  31: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#13}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:124
  30: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::SeqStmtNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:433
  29: tvm::tir::StmtMutator::Internal::Mutate(tvm::tir::StmtMutator*, tvm::runtime::Array<tvm::tir::Stmt, void> const&)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:213
  28: _ZN3tvm3tir11StmtMutator8Internal11MutateArrayINS0_4StmtEZNS2_6MutateEPS1_RKNS_7runtime5ArrayIS4_vEEEUlRKS4_E_EENS7_IT_NSt9enable_ifIXsr3std10is_base_ofINS6_9ObjectRefESE_EE5valueEvE4typeEEES5_RKSJ_T0_
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:186
  27: void tvm::runtime::Array<tvm::tir::Stmt, void>::MutateByApply<tvm::tir::StmtMutator::Internal::Mutate(tvm::tir::StmtMutator*, tvm::runtime::Array<tvm::tir::Stmt, void> const&)::{lambda(tvm::tir::Stmt const&)#1}>(tvm::tir::StmtMutator::Internal::Mutate(tvm::tir::StmtMutator*, tvm::runtime::Array<tvm::tir::Stmt, void> const&)::{lambda(tvm::tir::Stmt const&)#1})
        at /home/ganler/Documents/tvm-pr-2/include/tvm/runtime/container/array.h:623
  26: tvm::tir::StmtMutator::Internal::Mutate(tvm::tir::StmtMutator*, tvm::runtime::Array<tvm::tir::Stmt, void> const&)::{lambda(tvm::tir::Stmt const&)#1}::operator()(tvm::tir::Stmt const&) const
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:212
  25: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  24: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  23: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  22: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#3}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:114
  21: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#3}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:114
  20: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::IfThenElseNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:341
  19: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  18: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  17: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  16: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#3}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:114
  15: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#3}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:114
  14: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::IfThenElseNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:341
  13: tvm::tir::StmtMutator::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:244
  12: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:82
  11: tvm::NodeFunctor<tvm::tir::Stmt (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  10: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#15}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:126
  9: tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*)#15}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<tvm::tir::Stmt (tvm::tir::Stmt const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:126
  8: tvm::tir::ThreadAllreduceBuilder::VisitStmt_(tvm::tir::BufferStoreNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:158
  7: tvm::tir::StmtMutator::VisitStmt_(tvm::tir::BufferStoreNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/ir/stmt_functor.cc:364
  6: tvm::tir::StmtExprMutator::VisitExpr(tvm::PrimExpr const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/stmt_functor.h:320
  5: tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>::VisitExpr(tvm::PrimExpr const&)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/expr_functor.h:114
  4: tvm::NodeFunctor<tvm::PrimExpr (tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/node/functor.h:97
  3: tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*)#4}::__invoke(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*)
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/expr_functor.h:167
  2: tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*)#4}::operator()(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::PrimExpr (tvm::PrimExpr const&)>*) const
        at /home/ganler/Documents/tvm-pr-2/include/tvm/tir/expr_functor.h:167
  1: non-virtual thunk to tvm::tir::ThreadAllreduceBuilder::VisitExpr_(tvm::tir::BufferLoadNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc
  0: tvm::tir::ThreadAllreduceBuilder::VisitExpr_(tvm::tir::BufferLoadNode const*)
        at /home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc:126
  File "/home/ganler/Documents/tvm-pr-2/src/tir/transforms/lower_thread_allreduce.cc", line 126
TVMError: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (is_zero(index)) is false
"""

Environment

Ubuntu 20.04. TVM tag: ed638ef
@areusch areusch added the needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it label Oct 19, 2022
@driazati driazati added executor:graph runtime components of graph flow. src/runtime/graph_executor, python/tvm/contrib/graph_executor.py tir:transform TIR transforms: src/tir/transforms and python/tvm/tir/transforms and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Nov 16, 2022
@tqchen tqchen closed this as not planned Won't fix, can't repro, duplicate, stale Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
executor:graph runtime components of graph flow. src/runtime/graph_executor, python/tvm/contrib/graph_executor.py tir:transform TIR transforms: src/tir/transforms and python/tvm/tir/transforms type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@areusch @tqchen @driazati @ganler