Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add QNN EP HTP shared memory allocator #23136

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
110a3bc
save work
edgchen1 Nov 5, 2024
0ba3a2f
save work
edgchen1 Nov 9, 2024
8436b14
add logging for setting QNN tensor memory, update comment
edgchen1 Nov 11, 2024
c9826f4
add option to enable HTP shared memory allocator to onnxruntime_perf_…
edgchen1 Nov 11, 2024
c07c35e
hack - try to cache mem handles in QnnModel
edgchen1 Nov 12, 2024
60dc837
Remove duplicate include.
edgchen1 Nov 13, 2024
24e072f
hack, continued - move cache out to SharedContext
edgchen1 Nov 14, 2024
e66cbef
Merge remote-tracking branch 'origin/main' into edgchen1/qnn_ep_rpcmem
edgchen1 Nov 14, 2024
8c515da
move mem handle registration to allocator
edgchen1 Nov 15, 2024
18e2780
hook up some test code
edgchen1 Nov 15, 2024
09ddce5
Merge remote-tracking branch 'origin/main' into edgchen1/qnn_ep_rpcmem
edgchen1 Nov 19, 2024
a65bb71
rename to RpcMemAllocator to HtpSharedMemoryAllocator
edgchen1 Nov 27, 2024
bfb135e
Merge remote-tracking branch 'origin/main' into edgchen1/qnn_ep_rpcmem
edgchen1 Dec 2, 2024
f179a0d
remove onnx protobuf dependency from allocator.h, add shared provider…
edgchen1 Dec 3, 2024
7645ef4
remove unused CPUAllocator::TensorAlloc declaration
edgchen1 Dec 5, 2024
1043732
Check for nullptr when trying to free
baijumeswani Dec 5, 2024
022f4bc
move mem handle management to QNN backend manager
edgchen1 Dec 10, 2024
c527dee
remove IAllocator::TensorAlloc()
edgchen1 Dec 10, 2024
e4f72b3
document IAllocator::Free
edgchen1 Dec 10, 2024
39ff901
remove IAllocator__TensorAlloc
edgchen1 Dec 10, 2024
1bed5a4
Merge remote-tracking branch 'origin/main' into edgchen1/qnn_ep_rpcmem
edgchen1 Dec 10, 2024
d70db84
fix android build warning
edgchen1 Dec 10, 2024
45ef883
remove shared mem handles from shared context
edgchen1 Dec 11, 2024
d2e7b3c
remove allocation clean up callback removal, use weak_ptrs in allocat…
edgchen1 Dec 16, 2024
c892c18
some clean up
edgchen1 Dec 17, 2024
b295eef
more clean up
edgchen1 Dec 17, 2024
13f5e30
add helper to get qnn error message
edgchen1 Dec 17, 2024
d5eace1
use make_shared for QnnBackendManager
edgchen1 Dec 17, 2024
bacbcdc
add test to qnn_basic_test.cc, document allocator parameter.
edgchen1 Dec 17, 2024
30cd9ed
Merge remote-tracking branch 'origin/main' into edgchen1/qnn_ep_rpcmem
edgchen1 Dec 17, 2024
b29ab61
rename variables
edgchen1 Dec 18, 2024
67a54b8
revert changes to onnxruntime/test/providers/qnn/max_min_op_test.cc
edgchen1 Dec 18, 2024
c0569e2
fix formatting
edgchen1 Dec 19, 2024
dd45c84
skip test if not android and not windows
edgchen1 Dec 19, 2024
959d8df
update comment
edgchen1 Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
more clean up
  • Loading branch information
edgchen1 committed Dec 17, 2024
commit b295eef03a612ea370cd244043810b463c8ae3d7
4 changes: 4 additions & 0 deletions onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc
Original file line number Diff line number Diff line change
@@ -1682,6 +1682,10 @@ Status QnnBackendManager::GetOrRegisterContextMemHandle(Qnn_ContextHandle_t cont
return;
}

// TODO should also ensure that the QNN context handle is still valid.
// This *should* be true as long as the QNN contexts are not freed from anywhere other than
// ~QnnBackendManager(). If we are able to lock weak_backend_manager, we haven't gotten to the dtor yet.

auto unregister_status = context_mem_handle_manager->Unregister(shared_memory_address);
if (!unregister_status.IsOK()) {
LOGS(logger, ERROR) << "Failed to unregister shared memory mem handle for address: "
Original file line number Diff line number Diff line change
@@ -91,7 +91,6 @@ Status QnnContextMemHandleManager::GetOrRegister(void* shared_memory_address, co
const auto unregister_result = qnn_interface_.memDeRegister(&raw_mem_handle, 1);
if (unregister_result != QNN_SUCCESS) {
LOGS(logger_, ERROR) << "qnn_interface.memDeRegister() failed: " << unregister_result;
return;
}
};