error in ms_deformable_im2col_cuda: invalid configuration argument

[nainaigetuide]

Description

The following error occurred during the process of converting the onnx file to engine on jetson orin. For specific logs, see the attached file

The execution command is
/usr/src/tensorrt/bin/trtexec --onnx=./checkpoints/onnx/bevformer_small_epoch_24_cp2_add_cast.onnx --saveEngine=bevformer_small_epoch_24_cp2_add_cast.engine --fp16 --int8 --plugins=/home/orin/disk/BEVFormer_tensorrt/TensorRT/lib/libtensorrt_ops.so --dumpLayerInfo --profilingVerbosity=detailed --exportLayerInfo=layers.json --verbose

The error message is
‘error in ms_deformable_im2col_cuda: invalid configuration argument’

When I use tensorrt8.6, the above error does not occur. So I think it may be strongly related to tensorrt version.

Environment

TensorRT Version: 10.3

NVIDIA GPU: jetson orin

CUDA Version:12.2

Jetpack Version:12.2

Operating System: 22.04

Baremetal or Container (if so, version): nvcr.io/nvidia/l4t-jetpack:r36.4.0

Relevant Files

Model link: https://drive.google.com/file/d/1rAQWJJt5qaCkxevlAnzmCgnUYfXzCHoB/view?usp=sharing

Steps To Reproduce

1. Pull the image https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-jetpack/tags
2. The custom plugin is compiled to generate libtensorrt_ops.so. （Use the plugin in the following link
   https://github.com/DerryHub/BEVFormer_tensorrt/tree/main/TensorRT/plugin）
3. /usr/src/tensorrt/bin/trtexec --onnx=./checkpoints/onnx/bevformer_small_epoch_24_cp2_add_cast.onnx --saveEngine=bevformer_small_epoch_24_cp2_add_cast.engine --fp16 --int8 --plugins=/home/orin/disk/BEVFormer_tensorrt/TensorRT/lib/libtensorrt_ops.so --dumpLayerInfo --profilingVerbosity=detailed --exportLayerInfo=layers.json --verbose

[nainaigetuide]

https://drive.google.com/file/d/11ndILhN3KvQo2eu4-vYCi9OvJ0JGw5T2/view?usp=drive_link log is in the link.

[lix19937]

invalid configuration argument means nearly cuda/kernel has error, and you has better to recompile libtensorrt_ops in different trt versions.

[nainaigetuide]

My libtensorrt_ops was recompiled and generated in the environment of tensorrt 10.3. But this error is still reported. Why is it? Thank you. @lix19937

[lix19937]

You can do a unit test for your plugin.