Allow users to bind arbitrary memory using raw pointers #10428
Conversation
| /// <param name="elementType">element type</param> | ||
| /// <param name="pointer">the actual pointer to memory</param> | ||
| /// <param name="sizeInBytes">size of the allocation in bytes</param> | ||
| public OrtExternalAllocation(OrtMemoryInfo memInfo, long[] shape, Tensors.TensorElementType elementType, IntPtr pointer, long sizeInBytes) |
There was a problem hiding this comment.
Do we need this to be provided separately to the size that can be inferred from the shape? #Closed
There was a problem hiding this comment.
I was going back and forth on this. Finally, I decided to supply this argument/property to validate the user knows what they are doing. It is a common mistake to make a wrong choice betwen then number of elements and the buffer size in bytes.
| if (width < 1) | ||
| { | ||
| throw new OnnxRuntimeException(ErrorCode.InvalidArgument, "Unsupported data type (such as string)"); | ||
| } |
There was a problem hiding this comment.
nit: include elementType in the error message #Closed
| int width; | ||
| TensorElementTypeConverter.GetTypeAndWidth(elementType, out type, out width); | ||
| if (width < 1) | ||
| { |
There was a problem hiding this comment.
nit: can we update GetTypeAndWidth to return true/false if successful? if not, would checking type==null be a little more obvious than width < 1? #Closed
| } | ||
|
|
||
| /// <summary> | ||
| /// Bind externally allocated memory as input |
There was a problem hiding this comment.
nit: Should this description also include a line similar to the other BindInput() overload (OrtMemoryAllocation continues to own the chunk of native memory and should be alive until the end of execution.) for completeness ? #Resolved
| @@ -102,6 +109,23 @@ public void TestIOBindingWithOrtAllocation() | |||
| Assert.Equal(outputData, tensor.ToArray<float>(), new FloatComparer()); | |||
| } | |||
| } | |||
| // 3. Pretend we are using external allocation which is currently on CPU | |||
| throw new NotSupportedException(nameof(NativeOnnxTensorMemory<T>) + " does not support T = " + nameof(T)); | ||
| { | ||
| var message = String.Format("The NativeOnnxTensorMemory<T> type being instantiated for T = : {0} while supplied OrtValue contains T = {1}", | ||
| nameof(T), nameof(type)); |
| /// The model will read the specified input from that memory | ||
| /// possibly avoiding the need to copy between devices. The user code continues to own | ||
| /// the chunk of externally allocated memory, and the allocation should be alive until the end of execution. | ||
| /// by the Tensor of the given size. |
| if (elementType == TensorElementType.String) | ||
| { | ||
| throw new OnnxRuntimeException(ErrorCode.InvalidArgument, | ||
| "Can not use map managed strings buffer to native OrtValue"); |
yuslepukhin
force-pushed
the
yuslepukhin/bind_external
branch
from
af53a53 to
dd43824
Compare
Description:
Introduce OrtExternalAllocation class that allows to supply type, shape and raw memory pointer for binding and other purposes. Introduce reasonable shape checks against the supplied size.
Could not find a suitable framework class that would give its memory back as a raw Ptr.
Motivation and Context
OrtIoBinding does not currently allow binding of raw native and/or device-based buffers that were user-allocated or potentially by another framework. Those are usually expressed by
IntPtr.Re: #10180