Npu allocator #437
Npu allocator #437
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Crashing on tensor destruction. Might have UMD exceptions. Needs further debug. Unknown if values are correct.
saurabhkale17
force-pushed
the
npu_allocator
branch
from
df617dd to
966c48a
Compare
| @@ -50,6 +50,15 @@ constexpr const char* HIP = "Hip"; | |||
| constexpr const char* HIP_PINNED = "HipPinned"; | |||
| constexpr const char* OpenVINO_CPU = "OpenVINO_CPU"; | |||
| constexpr const char* OpenVINO_GPU = "OpenVINO_GPU"; | |||
| constexpr const char* OpenVINO_NPU = "OpenVINO_RT_NPU"; | |||
There was a problem hiding this comment.
OpenVINO_NPU is a redefinition of OpenVINO_RT_NPU.
Remove if its not referenced in the code.
There was a problem hiding this comment.
not referenced so removed it in the new pr
| size_t batch_size = 1; | ||
| Ort::UnownedValue output_tensor = | ||
| GetOutputTensor(context, batch_size, infer_request, std::move(output_name), subgraph_context_.output_names); | ||
| auto mem_info = output_tensor.GetTensorMemoryInfo(); | ||
| if (mem_info.GetAllocatorName() == OpenVINO_GPU) { |
There was a problem hiding this comment.
Check if this has effect on OpenVINO_GPU IOBuffer
| if (mem_info.GetAllocatorName() == OpenVINO_GPU) { | ||
| return; | ||
| auto allocator_name = output_tensor.GetTensorMemoryInfo().GetAllocatorName(); | ||
| ov_tensor_data_t ov_tensor_data; |
There was a problem hiding this comment.
check if the declaration in startasyncinference is redundant
There was a problem hiding this comment.
we will require the ov_tensor_data for creating the input/output tensor before the inference in startasyncinference
| @@ -854,6 +862,25 @@ select from 'TF8', 'TF16', 'UINT8', 'FLOAT', 'ITENSOR'. \n)"); | |||
| input_names_str_[i] = m.GetInputName(i); | |||
| input_names_[i] = input_names_str_[i].c_str(); | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Encapsulate the output tensor creation only if use_device_mem is set
There was a problem hiding this comment.
made the relevant changes only if the use_device_mem is true in the new PR
Description
draft PR for remote tensor feature implementation in OVEP