Classes of torchvision\ops #790

xhuan8 · 2022-10-09T05:35:52Z

Is there any plan to add classes of torchvision/ops?

roi_align and boxes are required, they are not implemented in TorchSharp

return torch.ops.torchvision.roi_align(
        input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned
    )

The text was updated successfully, but these errors were encountered:

GeorgeS2019 · 2022-10-09T09:20:14Z

TORCHVISION.OPS

suggestion: Need to approach this ALSO from the perspective of Onnx contributed Ops
e.g. com.microsoft.vision

flowchart TD
    image-->|com.microsoft.vision\nops| TorchSharp\nmodel-->|com.microsoft.vision\nops| output\nimage

GeorgeS2019 · 2022-10-09T09:27:53Z

@kaiidams

Why there are no Torchaudio.ops?

Are they in here?
https://github.com/pytorch/audio/tree/main/torchaudio/csrc

e.g. com.microsoft.audio

flowchart TD
    audio-->|com.microsoft.audio\nops| TorchSharp\nmodel-->|com.microsoft.audio\nops| output\naudio

GeorgeS2019 · 2022-10-09T09:29:58Z

Most of the Torchtext.ops are here
https://github.com/pytorch/text/tree/main/torchtext/csrc

com.microsoft.nlp

flowchart TD
    text-->|com.microsoft.nlp\nops| TorchSharp\nmodel-->|com.microsoft.nlp\nops| output\ntext

kaiidams · 2022-10-10T04:17:31Z

@GeorgeS2019

Probably ONNX runtime approach is not related to this.

Why there are no Torchaudio.ops?

torchaudio has C code that uses Kaldi and Sox, FFMpeg, which is not implemented in TorchSharp. IMHO, Modern models don't depend on these except I/O, but still useful in some cases. torchaudio doesn't call them ops.

@NiklasGustafsson I don't know about what happened to torchvision.ops, if they are missing features, do you have plan to add them?

NiklasGustafsson · 2022-10-10T14:09:42Z

There's a small number of ops in torchvision.ops. Feel free to contribute more, if you have time. It would be good to have an issue to track them. I'll open something up.

NiklasGustafsson · 2022-10-12T16:30:10Z

BTW, it seems like FFMPEG is disabled by default in PyTorch, now:

https://pytorch.org/vision/stable/#torchvision.set_video_backend

You have to build from source to enable it, apparently.

xhuan8 · 2022-10-17T10:01:41Z

seems there is no prebuild package for torchvision, what is the best way to add it after build from source?

NiklasGustafsson · 2022-10-17T13:58:27Z

seems there is no prebuild package for torchvision, what is the best way to add it after build from source?

TorchVision is now available on NuGet: https://www.nuget.org/packages/TorchVision, starting with version 0.98.1, when it was split out from TorchSharp.

xhuan8 · 2022-10-18T02:00:08Z

I try to call the native method from torchvision, torch.ops.torchvision.nms
https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/nms.cpp
https://github.com/pytorch/vision/blob/main/torchvision/ops/_register_onnx_ops.py

how to do it in TorchSharp?

NiklasGustafsson · 2022-10-18T03:39:31Z

I believe someone already implemented nms in C#.

TorchSharp/src/TorchVision/Ops.cs

Line 48 in 2f89e58

    
           public static Tensor nms(Tensor boxes, Tensor scores, double iou_threshold = 0.5)

NiklasGustafsson · 2022-10-18T14:05:23Z

To answer your question, though -- I haven't understood where the native code binaries and header files for TV are to be found. If you figure that out, we should bundle it in with the other native code we already have.

xhuan8 · 2022-10-18T14:32:30Z

They are under https://github.com/pytorch/vision/tree/main/torchvision/csrc/ops, contains cpu and cuda implementations.
We don't need the IO operations, so FFMPEG and JPEG are not needed.
Currently I'll try build from souce and use it on Windows.

NiklasGustafsson · 2022-10-18T14:34:59Z

Have you found where pre-built binaries are available for download?

xhuan8 · 2022-10-18T15:19:01Z

no, only build from source.

NiklasGustafsson · 2022-10-18T16:59:52Z

Note that the current libtorch binaries are 1.11, with CUDA 11.3 -- that will matter for building compatible TV binaries, I believe.

GeorgeS2019 · 2022-10-18T19:19:49Z

I try to call the native method from torchvision, torch.ops.torchvision.nms
https://github.com/pytorch/vision/blob/main/torchvision/csrc/ops/nms.cpp
https://github.com/pytorch/vision/blob/main/torchvision/ops/_register_onnx_ops.py

For Onnxruntime, here is some preliminary discussion how to register Onnx Ops
WIP [Documentation]: C# Workflow for consuming "Augmented" Onnx model with Custom Operators

xhuan8 · 2022-10-19T13:41:24Z

Fortunately I have done the build, there are 3 files, torchvision.dll, torchvision.exp, torchvision.lib,
and quite small, only 2 MB

NiklasGustafsson · 2022-10-19T17:51:13Z

@xhuan8 -- this is really cool! There are a couple of things to think about next:

A different output names (more on that later), since 'torchvision.dll' will conflict with the DLL name for the .NET project.
How we distribute the native library. Currently, we have a package (libtorch-*) containing the CPU backends for all three platforms, and one for CUDA on Windows, and one for CUDA on Linux. We needed this because the CUDA backends are gigantic.

So, we have to decide where these native libraries go, whether in the same package as the managed TorchVision binaries, or in a separate one. It will depend on the overall size (separate Windows + MacOS + Linux binaries), as well as whether a CUDA backend can be loaded on a machine without a CUDA-capable GPU. This will impact the library names we pick, too.

NiklasGustafsson · 2022-10-19T18:07:47Z

Also, how to automate the build -- do we do this manually and add the binaries to the build, or do we somehow integrate the pytorch/vision repo into the TorchSharp build process? We should consider pytorch/text and pytorch/audio at the same time.

Thanks for doing this work, it's going to make a huge difference!

NiklasGustafsson · 2022-10-19T23:41:09Z

TORCHVISION.OPS

suggestion: Need to approach this ALSO from the perspective of Onnx contributed Ops e.g. com.microsoft.vision

@GeorgeS2019 -- I love your enthusiasm for ONNX runtime ops. However, I want to say that TorchSharp is about providing a .NET layer on top of the native library (libraries) underlying PyTorch. Nothing less, nothing more. It's simplistic, perhaps, but the main point is that it simple. Other projects can and should go beyond TorchSharp and provide all kinds of additional features, including interactions with ONNX.

Thus, ONNX runtime ops (contributed or not) lie outside the scope of what TorchSharp will be designed for.

xhuan8 · 2022-10-24T07:56:22Z

try to load torchvision native dll with LoadLibrary, it fails with error 1114, @NiklasGustafsson @kaiidams do you have any idea?
the dll upload to:
https://drive.google.com/file/d/1LHd1jwuFlFT87-vT09hzsOWpUUeD-TKF/view?usp=sharing

GeorgeS2019 · 2022-10-27T02:37:56Z

Instead of loading a native dll, the latest way is to embedded a static library *.lib through *.csproj and compile that into a normal c# dll.
More and more of Microsoft products which are previously c++ are now in c#

Godot4 .NET6 NativeAOT is an excellent recent example.

GeorgeS2019 · 2022-10-27T02:46:54Z

The Godot c# library attaches a c++ Shared Library _internal ( or _internal.lib )

GeorgeS2019 · 2022-10-27T23:03:54Z

@xhuan8
An interesting MIT License Visual Programming approach with pre/post vision processing node(Ops) around ONNX

FYI: could be interesting and the example involves TorchScript, however not related to TorchSharp, more investigation needed.

kaiidams · 2022-10-29T02:42:57Z

@xhuan8 If this is the build from torchvision, it is a C++ torchvision library. You'll need to make a C wrapper so that C# can use it with P/Invoke.

(py310) C:\local>dumpbin /dependents "libtorchvision.dll"
Microsoft (R) COFF/PE Dumper Version 14.33.31630.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file libtorchvision.dll

File Type: DLL

  Image has the following dependencies:

    c10.dll
    c10_cuda.dll
    torch_cuda_cu.dll
    torch_cuda_cpp.dll
    torch_cpu.dll
    KERNEL32.dll
    MSVCP140.dll
    VCRUNTIME140.dll
    VCRUNTIME140_1.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll

  Summary

        D000 .data
        1000 .nvFatBi
      149000 .nv_fatb
        7000 .pdata
       4E000 .rdata
        2000 .reloc
        1000 .rsrc
       F0000 .text

If Torch C++ provides torch::fft::hfft2() then we have THSTensor_hfft2 for C# to call.

TorchSharp/src/Native/LibTorchSharp/THSFFT.cpp

Line 64 in 0704771

    
           Tensor THSTensor_hfft2(const Tensor tensor, const int64_t* s, const int64_t* dim, int8_t norm)

kaiidams · 2022-10-29T02:51:01Z

@NiklasGustafsson To build torchvision.dll you'll need Python (and zlib, libpng, CUDA, etc) I think it should be built outside TorchSharp.

NiklasGustafsson · 2022-10-31T20:42:32Z

@NiklasGustafsson To build torchvision.dll you'll need Python (and zlib, libpng, CUDA, etc) I think it should be built outside TorchSharp.

Yeah, you're right... Ideally, we would re-implement the C++ bodies of TorchVision in C# -- that would solve a lot of problems.

NiklasGustafsson · 2022-11-01T21:25:18Z

@xhuan8 -- would that be feasible, do you think? To just re-implement the ops in C# using Torch operators, like how nms() was implemented?

xhuan8 · 2022-11-02T01:22:48Z

@NiklasGustafsson It takes time to implement with C# and verify the correctness, and also not sure how is the performace without cuda.
Currently I'll use the c++ wrapper, the method name from torchvision is little bit strange, like ?nms@ops@vision@@YA?AVTensor@at@@AEBV34@0N@Z

NiklasGustafsson · 2022-11-02T01:58:36Z

If the ops are implemented in terms of TorchSharp ops, CUDA should come for free.

That name is a C++ mangled name, which means that it's missing the 'extern "C"' declaration.

NiklasGustafsson · 2023-02-22T21:14:09Z

Same as I asked about Faster RCNN -- you've been working on this, it appears. Anything ready for a PR?

GeorgeS2019 · 2023-10-20T19:27:58Z

@NiklasGustafsson
Now we have a user routinely compiling NLP ops in TorchSharp. Perhaps a wiki on this with links to user projects?

GeorgeS2019 mentioned this issue Oct 18, 2022

WIP [Documentation]: C# Workflow for consuming "Augmented" Onnx model with Custom Operators microsoft/onnxruntime#11657

Open

1 task

xhuan8 added a commit to xhuan8/TorchSharp that referenced this issue Dec 7, 2022

dotnet#790 Add Ops\Boxes

ce72285

xhuan8 added a commit to xhuan8/TorchSharp that referenced this issue Jan 19, 2023

dotnet#790 Add BackboneUtils

7fc03d7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Classes of torchvision\ops #790

Classes of torchvision\ops #790

xhuan8 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

kaiidams commented Oct 10, 2022 •

edited

Loading

NiklasGustafsson commented Oct 10, 2022 •

edited

Loading

NiklasGustafsson commented Oct 12, 2022

xhuan8 commented Oct 17, 2022

NiklasGustafsson commented Oct 17, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

GeorgeS2019 commented Oct 18, 2022

xhuan8 commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

TORCHVISION.OPS

xhuan8 commented Oct 24, 2022

GeorgeS2019 commented Oct 27, 2022 •

edited

Loading

GeorgeS2019 commented Oct 27, 2022

GeorgeS2019 commented Oct 27, 2022 •

edited

Loading

kaiidams commented Oct 29, 2022 •

edited

Loading

kaiidams commented Oct 29, 2022

NiklasGustafsson commented Oct 31, 2022

NiklasGustafsson commented Nov 1, 2022

xhuan8 commented Nov 2, 2022

NiklasGustafsson commented Nov 2, 2022

NiklasGustafsson commented Feb 22, 2023

GeorgeS2019 commented Oct 20, 2023

Classes of torchvision\ops #790

Classes of torchvision\ops #790

Comments

xhuan8 commented Oct 9, 2022 • edited Loading

GeorgeS2019 commented Oct 9, 2022 • edited Loading

TORCHVISION.OPS

GeorgeS2019 commented Oct 9, 2022 • edited Loading

GeorgeS2019 commented Oct 9, 2022 • edited Loading

kaiidams commented Oct 10, 2022 • edited Loading

NiklasGustafsson commented Oct 10, 2022 • edited Loading

NiklasGustafsson commented Oct 12, 2022

xhuan8 commented Oct 17, 2022

NiklasGustafsson commented Oct 17, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

xhuan8 commented Oct 18, 2022

NiklasGustafsson commented Oct 18, 2022

GeorgeS2019 commented Oct 18, 2022

xhuan8 commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

NiklasGustafsson commented Oct 19, 2022

TORCHVISION.OPS

xhuan8 commented Oct 24, 2022

GeorgeS2019 commented Oct 27, 2022 • edited Loading

GeorgeS2019 commented Oct 27, 2022

GeorgeS2019 commented Oct 27, 2022 • edited Loading

kaiidams commented Oct 29, 2022 • edited Loading

kaiidams commented Oct 29, 2022

NiklasGustafsson commented Oct 31, 2022

NiklasGustafsson commented Nov 1, 2022

xhuan8 commented Nov 2, 2022

NiklasGustafsson commented Nov 2, 2022

NiklasGustafsson commented Feb 22, 2023

GeorgeS2019 commented Oct 20, 2023

xhuan8 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

GeorgeS2019 commented Oct 9, 2022 •

edited

Loading

kaiidams commented Oct 10, 2022 •

edited

Loading

NiklasGustafsson commented Oct 10, 2022 •

edited

Loading

GeorgeS2019 commented Oct 27, 2022 •

edited

Loading

GeorgeS2019 commented Oct 27, 2022 •

edited

Loading

kaiidams commented Oct 29, 2022 •

edited

Loading