Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue with fast_math in internal and upstream Onnxruntime MIGraphX EP #3021

Closed
5 tasks done
TedThemistokleous opened this issue Apr 30, 2024 · 2 comments · Fixed by ROCm/onnxruntime#32, ROCm/onnxruntime#33 or microsoft/onnxruntime#20520
Assignees

Comments

@TedThemistokleous
Copy link
Collaborator

TedThemistokleous commented Apr 30, 2024

It's come to my attention a patch in Onnxruntime MIGraphX EP wasn't upstreamed to our latest 6.1 and 6.2 branches.

Need to patch this + also ensure this is up streamed to mainline Microsoft/Onnxruntime

Patch in Question: ROCm/onnxruntime#25
Suspect this is the cause of this issue in JIRA: https://ontrack-internal.amd.com/browse/SWDEV-453466

Will need to also follow up with other perf issues where Driver vs EP is slightly faster: https://ontrack-internal.amd.com/browse/SWDEV-446822

Highly unlikely as I think the build used for this issue was after this batch was added and initial tests were done on 6.0.2 where this patch lives.

@TedThemistokleous
Copy link
Collaborator Author

https://ontrack-internal.amd.com/browse/SWDEV-446822 appears to have the fast_math batch applied, going by build numbers (115) on that ticket, and confirming with DevOps that the repo is cloned when using an internal testing branch to create the associated ROCm X.X files when we build the EP. It appears this is a different unrelated issue.

PR's to internal branches have been opened and awaiting review.

@TedThemistokleous
Copy link
Collaborator Author

Solves the SWB so closing this ticket out. Will hammer Onnxruntime upstream until we get this fix in as this is critical for GELU accuracy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment