WIP [Documentation]: C# Workflow for consuming "Augmented" Onnx model with Custom Operators

[GeorgeS2019]

3 ways to add a new operator

There is an issue to missing link

Use Case: PyTorch exports a custom_op_test.onnx with TWO custom operators

custom_op_test.onnx

• CustomOpOne
• CustomOpTwo

To consume in CSharp exported PyTorch Onnx with custom operators, it is necessary to register these Custom Operators (written in C++) and provided in a Shared Library

CSharp API for loading Shared Library with Custom Operators

Create Custom_op_library.dll

• Custom_op_library.dll : source files

CSharp api for loading/registrating custom op shared library

RegisterCustomOps

custom_op_library.h

#pragma once
#include "onnxruntime_c_api.h"

#ifdef __cplusplus
extern "C" {
#endif

ORT_EXPORT OrtStatus* ORT_API_CALL RegisterCustomOps(OrtSessionOptions* options, const OrtApiBase* api);

#ifdef __cplusplus
}
#endif

custom_op_library.cc

• With 2 Custom Operators
  • CustomOpOne
  • CustomOpTwo

#include "custom_op_library.h"

#define ORT_API_MANUAL_INIT
#include "onnxruntime_cxx_api.h"
#undef ORT_API_MANUAL_INIT


OrtStatus* ORT_API_CALL RegisterCustomOps(OrtSessionOptions* options, const OrtApiBase* api) {
  OrtCustomOpDomain* domain = nullptr;
  const OrtApi* ortApi = api->GetApi(ORT_API_VERSION);

  if (auto status = ortApi->CreateCustomOpDomain(c_OpDomain, &domain)) {
    return status;
  }

  AddOrtCustomOpDomainToContainer(domain, ortApi);

  if (auto status = ortApi->CustomOpDomain_Add(domain, &c_CustomOpOne)) {
    return status;
  }

  if (auto status = ortApi->CustomOpDomain_Add(domain, &c_CustomOpTwo)) {
    return status;
  }

  return ortApi->AddCustomOpDomain(options, domain);
}

In CSharp Register Custom Operators within the Shared Library: Custom_op_library.dll

CSharp UnitTest: NetCoreApp/InferenceTest.netcore.c

private void TestRegisterCustomOpLibrary()
{
    using (var option = new SessionOptions())
    {
        string libName = "custom_op_library.dll";
        string modelPath = "custom_op_test.onnx";

The c++ unit tests discussion

@natke

WIP for CSharp
[Documentation] How to register custom operator in onnxruntime

Other references:

• NER transformers model packed in one onnx file with Sentencepiece Tokenizer

[GeorgeS2019]

related issues:

• [C#] ContribOperators
• [Documentation, Windows, c#] Add an entry on Build for Custom Op development
• [C#] Documentation on how to coordinate the external Onnxruntime-Extension
• [C#] Basic build instruction to generate custom_op_library.dll
• [C#] Provide custom_op_library.dll

[GeorgeS2019]

The c# example for testing custom operator is here.

We need c# example on how to use contributed operator

[GeorgeS2019]

Is your feature request related to a problem? Please describe.

The main build instruction page on Build ONNX Runtime from source does not provide the simplest step for building for custom op development

==> In addition to what is listed below, perhaps we need a section on Build for Custom Op development

Build for inferencing
Build for training
Build with different EPs
Build for Custom Op development <======

[GeorgeS2019]

FYI: Classes of torchvision\ops
@xhuan8 @kaiidams @NiklasGustafsson @natke @luisquintanilla

Reusability of PyTorch/OnnxExtension C++ Operators

in ML.NET, TorchSharp, Onnxruntime and PyTorch

flowchart TB
    Operators\nstaticLibrary1-->Operators\nstaticLibrary2
Operators\nDynamicLibrary1-->Operators\nDynamicLibrary2
Operators\nDynamicLibrary1-->Operators\nDynamicLibrary3
    subgraph ML.NET-Or-ORT
    Onnx-->C#-register
C#-register-->Operators\nDynamicLibrary3
  Operators\nDynamicLibrary3-->Augmented\nORT\nInference
   end
    subgraph ORT-Extension
    Custom\nOperators\nC++-->Operators\nDynamicLibrary1
    end
    subgraph TorchSharp-TorchVision
    C#Interop-->Operators\nstaticLibrary2
    C#Interop\nRegister-->Operators\nDynamicLibrary2
Operators\nstaticLibrary2-->Augmented\nTorchSharp\nModel
Operators\nDynamicLibrary2-->Augmented\nTorchSharp\nModel
    end
    subgraph PyTorch-TorchVision
    Operators\nC++-->Operators\nstaticLibrary1
    Operators\nstaticLibrary1-->PyBind
    PyBind-->Augmented\nPyTorch\nModel
    end

Loading

[GeorgeS2019]

@natke

If I understand correctly, most of the contributed Ops in ORT, (some of them) come from experimental Custom Ops development in ORT Extension.

If these contributed Ops could be "Reversed" back to Custom Ops and made available through ORT Extension, then other projects e.g. ML.NET and TorchSharp could benefit from the Ops reusability!