Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build engine failure of TensorRT 10.0 when running trtexec with fp16 on GPU NVIDIA 3060 series #3800

Closed
roxanacincan opened this issue Apr 15, 2024 · 5 comments
Assignees
Labels
internal-bug-tracked Tracked internally, will be fixed in a future release. triaged Issue has been triaged by maintainers

Comments

@roxanacincan
Copy link

Description

I tried building a TRT engine using trtexec and fp16 precision, but the process fails without any errors. The TRT model is never generated. For fp32 the TRT model is correctly generated.

Environment

TensorRT Version: 10.0.0.6EA

NVIDIA GPU: NVIDIA GeForce RTX 3060

NVIDIA Driver Version: 551.86

CUDA Version: 11.8

CUDNN Version: 8.9.7

Operating System: Windows10

Relevant Files

Model link: https://drive.google.com/file/d/1JjA_9Ea4oTf-jnn41pYOMolpWOGYoWay/view?usp=drive_link
Build log: build.log

Steps To Reproduce

Commands or scripts: trtexec --onnx=best_model.onnx --saveEngine=best_model_fp16.plan --fp16 --verbose
I also tried using the build_engine.py script from the python samples directory of TensorRT10 but it returns exactly the same output as trtexec. No errors or warnings, just the process stopping abruptly.

Have you tried the latest release?: yes

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): yes it can run with Polygraphy

@RajUpadhyay
Copy link

Can you trying running the command with additional option of --useCudaGraph.
trtexec --onnx=model.onnx --saveEngine=engine_fp16.engine --fp16 --useCudaGraph --verbose

@roxanacincan
Copy link
Author

i tried using the option --useCudaGraph, but there were no changes.
I was still not able to build the model even though i don't get any errors.
Also, the build log file looks the same as the previous one i uploaded: build_with_useCudaGraph.log

@zerollzeng
Copy link
Collaborator

Just requested access to the model.

@zerollzeng zerollzeng self-assigned this Apr 18, 2024
@zerollzeng zerollzeng added the triaged Issue has been triaged by maintainers label Apr 18, 2024
@zerollzeng
Copy link
Collaborator

Thanks for reporting this, filed internal bug 4609232 to track.

@zerollzeng zerollzeng added the internal-bug-tracked Tracked internally, will be fixed in a future release. label Apr 18, 2024
@akhilg-nv
Copy link
Collaborator

The bug for this issue has been fixed and should be included in the latest minor release, closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internal-bug-tracked Tracked internally, will be fixed in a future release. triaged Issue has been triaged by maintainers
Projects
None yet
Development

No branches or pull requests

4 participants