Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash running inference on a specific ONNX model #22448

Closed
frenetj opened this issue Oct 15, 2024 · 1 comment
Closed

Crash running inference on a specific ONNX model #22448

frenetj opened this issue Oct 15, 2024 · 1 comment

Comments

@frenetj
Copy link

frenetj commented Oct 15, 2024

Describe the issue

When running CPU inference on the LaMa-ONNX model, ORT crashes with the following stack:

#0 0x00007fc882425841 in _mm_loadu_ps(float const*) (__P=0x20e50) at /opt/rh/gcc-toolset-11/root/usr/lib/gcc/x86_64-redhat-linux/11/include/xmmintrin.h:934
#1 0x00007fc882425841 in Eigen::internal::ploadu<float __vector(4)>(Eigen::internal::unpacket_traits<float __vector(4)>::type const*) (from=0x20e50)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/SSE/PacketMath.h:739
#2 0x00007fc8824443eb in Eigen::internal::ploadt<float __vector(4), 0>(Eigen::internal::unpacket_traits<float __vector(4)>::type const*) (from=0x20e50)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/GenericPacketMath.h:969
#3 0x00007fc8824443eb in Eigen::internal::mapbase_evaluator<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> >, Eigen::Matrix<float, -1, 1, 0, -1, 1> const>::packet<0, float __vector(4)>(long) const
(this=0x7fc2157e08f0, index=0) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/CoreEvaluators.h:944
#4 0x00007fc882520197 in Eigen::internal::evaluator_wrapper_base<Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > >::packet<0, float __vector(4)>(long) const
(this=0x7fc2157e08f0, index=0) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/CoreEvaluators.h:1430
#5 0x00007fc882598675 in Eigen::internal::binary_evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::IndexBased, Eigen::internal::IndexBased, float, float>::packet<0, float __vector(4)>(long) const
(this=0x7fc2157e08e0, index=0) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/CoreEvaluators.h:791
#6 0x00007fc88258f4d7 in Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >, Eigen::internal::assign_op<float, float>, 0>::assignPacket<16, 0, float __vector(4)>(long) (this=0x7fc2157e08a0, index=0)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:681
#7 0x00007fc882586a7e in Eigen::internal::dense_assignment_loop<Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >, Eigen::internal::assign_op<float, float>, 0>, 3, 0>::run(Eigen::internal::generic_dense_assignment_kernel<Eigen::internal::evaluator<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >, Eigen::internal::evaluator<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >, Eigen::internal::assign_op<float, float>, 0>&) (kernel=...)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:437
#8 0x00007fc8825769b8 in Eigen::internal::call_dense_assignment_loop<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::assign_op<float, float> >(Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> const&, Eigen::internal::assign_op<float, float> const&) (dst=..., src=..., func=...)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:785
#9 0x00007fc88256ba26 in Eigen::internal::Assignment<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::assign_op<float, float>, Eigen::internal::Dense2Dense, void>::run(Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> const&, Eigen::internal::assign_op<float, float> const&)
(dst=..., src=..., func=...) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:954
#10 0x00007fc882568e79 in Eigen::internal::call_assignment_no_alias<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::assign_op<float, float> >(Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> const&, Eigen::internal::assign_op<float, float> const&) (dst=..., src=..., func=...)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:890
#11 0x00007fc8825655bc in Eigen::internal::call_assignment<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::assign_op<float, float> >(Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> const&, Eigen::internal::assign_op<float, float> const&, Eigen::internal::enable_if<!Eigen::internal::evaluator_assume_aliasing<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const>, Eigen::internal::evaluator_traits<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >::Shape>::value, void*>::type) (dst=..., src=..., func=...)
at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:858
#12 0x00007fc88255d24d in Eigen::internal::call_assignment<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >(Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> >&, Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> const&) (dst=..., src=...) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/AssignEvaluator.h:836
#13 0x00007fc8825501bf in Eigen::MatrixBase<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >::operator=<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::interna--Type for more, q to quit, c to continue without paging--
l::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> >(Eigen::DenseBase<Eigen::CwiseBinaryOp<Eigen::internal::scalar_difference_op<float, float>, Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, Eigen::Array<float, -1, 1, 0, -1, 1> const> const, Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1, 0, -1, 1> const, 0, Eigen::Stride<0, 0> > > const> > const&) (this=0x7fc2157e0ac0, other=...) at /git/onnxruntime/build/Linux/Debug/_deps/eigen-src/Eigen/src/Core/Assign.h:66
#14 0x00007fc8825438c2 in onnxruntime::Sub::Compute(onnxruntime::OpKernelContext*) const::{lambda(onnxruntime::BroadcastHelper&)#1}::operator()(onnxruntime::BroadcastHelper&) const (__closure=0x0, per_iter_bh=...)
at /git/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:489
#15 0x00007fc8825438e2 in onnxruntime::Sub::Compute(onnxruntime::OpKernelContext*) const::{lambda(onnxruntime::BroadcastHelper&)#1}::_FUN(onnxruntime::BroadcastHelper&) ()
at /git/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:488
#16 0x00007fc8824ec769 in operator()(std::ptrdiff_t, std::ptrdiff_t) const (__closure=0x7ffe53b25540, first=33684, last=50526) at /git/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:960
#17 0x00007fc882511f36 in std::__invoke_impl<void, onnxruntime::ParallelizeSingleSpanonnxruntime::BroadcastHelper(onnxruntime::BroadcastHelper&, const onnxruntime::ProcessBroadcastSpanFuncs&)::<lambda(std::ptrdiff_t, std::ptrdiff_t)>&, long int, long int>(std::__invoke_other, struct {...} &) (__f=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:61
#18 0x00007fc88250872a in std::__invoke_r<void, onnxruntime::ParallelizeSingleSpanonnxruntime::BroadcastHelper(onnxruntime::BroadcastHelper&, const onnxruntime::ProcessBroadcastSpanFuncs&)::<lambda(std::ptrdiff_t, std::ptrdiff_t)>&, long int, long int>(struct {...} &) (__fn=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:111
#19 0x00007fc8824fd5c5 in std::_Function_handler<void(long int, long int), onnxruntime::ParallelizeSingleSpanonnxruntime::BroadcastHelper(onnxruntime::BroadcastHelper&, const onnxruntime::ProcessBroadcastSpanFuncs&)::<lambda(std::ptrdiff_t, std::ptrdiff_t)> >::_M_invoke(const std::_Any_data &, long &&, long &&) (__functor=..., __args#0=@0x7fc2157e0c70: 33684, __args#1=@0x7fc2157e0c68: 50526) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:290
#20 0x00007fc8832463ff in std::function<void (long, long)>::operator()(long, long) const (this=0x7ffe53b25540, __args#0=33684, __args#1=50526) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:590
#21 0x00007fc883244bfe in operator()(unsigned int) const (__closure=0x2312a3a0, idx=2) at /git/onnxruntime/onnxruntime/core/common/threadpool.cc:433
#22 0x00007fc88324ae3c in std::__invoke_impl<void, onnxruntime::concurrency::ThreadPool::ParallelForFixedBlockSizeScheduling(std::ptrdiff_t, std::ptrdiff_t, const std::function<void(long int, long int)>&)::<lambda(unsigned int)>&, unsigned int>(std::__invoke_other, struct {...} &) (__f=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:61
#23 0x00007fc8832499d4 in std::__invoke_r<void, onnxruntime::concurrency::ThreadPool::ParallelForFixedBlockSizeScheduling(std::ptrdiff_t, std::ptrdiff_t, const std::function<void(long int, long int)>&)::<lambda(unsigned int)>&, unsigned int>(struct {...} &) (__fn=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:111
#24 0x00007fc88324867e in std::_Function_handler<void(unsigned int), onnxruntime::concurrency::ThreadPool::ParallelForFixedBlockSizeScheduling(std::ptrdiff_t, std::ptrdiff_t, const std::function<void(long int, long int)>&)::<lambda(unsigned int)> >::_M_invoke(const std::_Any_data &, unsigned int &&) (__functor=..., __args#0=@0x7fc2157e0d74: 2) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:290
#25 0x00007fc88324740e in std::function<void (unsigned int)>::operator()(unsigned int) const (this=0x2fa113b0, __args#0=2) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:590
#26 0x00007fc88324ffef in onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}::operator()() const (__closure=0x2fa113b0)
at /git/onnxruntime/include/onnxruntime/core/platform/EigenNonBlockingThreadPool.h:1107
#27 0x00007fc883251596 in std::__invoke_impl<void, onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}&>(std::__invoke_other, onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}&) (__f=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:61
#28 0x00007fc883251116 in std::__invoke_r<void, onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}&>(void&&, (onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}&)...) (__fn=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:111
#29 0x00007fc883250c5c in std::_Function_handler<void (), onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::ScheduleOnPreferredWorkers(onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::PerThread&, onnxruntime::concurrency::ThreadPoolParallelSection&, absl::InlinedVector<int, 11ul, std::allocator >&, unsigned int, unsigned int, std::function<void (unsigned int)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...)
at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:290
#30 0x00007fc8821c1e42 in std::function<void ()>::operator()() const (this=0x7fc2157e0e50) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_function.h:590
#31 0x00007fc88324ac0f in onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::WorkerLoop(int) (this=0x32d488c0, thread_id=1) at /git/onnxruntime/include/onnxruntime/core/platform/EigenNonBlockingThreadPool.h:1628
#32 0x00007fc8832497f4 in onnxruntime::concurrency::ThreadPoolTemplonnxruntime::Env::WorkerLoop(int, Eigen::ThreadPoolInterface*) (id=1, param=0x32d488c0)
at /git/onnxruntime/include/onnxruntime/core/platform/EigenNonBlockingThreadPool.h:706
#33 0x00007fc88325420d in onnxruntime::(anonymous namespace)::PosixThread::ThreadMain(void*) (param=0x1693f0d0) at /git/onnxruntime/onnxruntime/core/platform/posix/env.cc:244
#34 0x00007fc8977df17a in start_thread () at /lib64/libpthread.so.0
#35 0x00007fc895b22dc3 in clone () at /lib64/libc.so.6

Note that running the same inference with the CUDA EP does NOT crash and returns the following error:

***ErrorMsg: CUDA failure 1: invalid argument ; GPU=0 ; hostname=xxx ; file=/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));

To reproduce

Download lama_fp32.onnx, image.jpg and mask.png, from the following web site:

https://huggingface.co/Carve/LaMa-ONNX/tree/main

Run inference using the default CPU handler, providing image.jpg and mask.png as model inputs.

Urgency

No response

Platform

Linux

OS Version

Rocky Linux 8.7

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.19.0

ONNX Runtime API

C

Architecture

X86

Execution Provider

Default CPU

Execution Provider Library Version

No response

@frenetj frenetj closed this as completed Nov 5, 2024
@frenetj
Copy link
Author

frenetj commented Nov 5, 2024

Crash was happening because the ORT client was not providing a valid buffer for the second image input.

We are using the following C API method to create tensor inputs:

CreateTensorWithDataAsOrtValue (const OrtMemoryInfo *info, void *p_data, size_t p_data_len, const int64_t *shape, size_t shape_len, ONNXTensorElementDataType type, OrtValue **out)

The second tensor input was created giving a NULL p_data pointer.

It seems the method didn’t return any errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant