Tensor data preparation

[guillaume-be]

Hello,

I am trying to load a model and am unsure how to prepare the input in order to pass it to a model. I tried referring to the batched example provided in https://github.com/sonos/tract/blob/main/examples/onnx-mobilenet-v2-batch/src/main.rs but was not able to have a solution running. Below is a minimal example causing issues:

use std::path::Path;
use tract_onnx::prelude::*;

fn main() -> anyhow::Result<()> {
    let model_path = Path::new("path/to/model.onnx");

    let batch_symbol = Symbol::new('N');
    let sequence_symbol = Symbol::new('S');

    let model = tract_onnx::onnx()
        .model_for_path(model_path)?
        .with_input_fact(
            0,
            InferenceFact::dt_shape(
                i64::datum_type(),
                tvec!(batch_symbol.to_dim(), sequence_symbol.to_dim()),
            ),
        )?
        .into_optimized()?
        .into_runnable()?;

    let input_ids = tract_ndarray::Array2::from_shape_vec(
        (2, 8),
        vec![
            [
                0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64,
            ],
            [
                0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64,
            ],
        ],
    )?;

    let result = model.run(tvec!(input_ids.into()))?;

    println!("{:?}", result);

    Ok(())
}

I am then facing the error:

60 |     let result = model.run(tvec!(input_ids.into()))?;
   |                                            ^^^^ the trait `tract_onnx::prelude::Datum` is not implemented for `[i64; 8]`

Is there an additional conversion step required when using tract_ndarray::Array2::from_shape_vec to prepare the input?

[kali]

The problem is on the input_ids. From the look of it it is an Array of [i64;8], and you need an Array of i64. I thing from_shape_vec does not do what you think it does :) look at arr2 maybe (or even tract tensor2).

[guillaume-be]

Thank you for the quick feedback! I was originally trying to use arr2/tensor2 but was facing issues because of the trait not being implemented:

    |
30  |       let input_ids = arr2(&[
    |  _____________________----_^
    | |                     |
    | |                     required by a bound introduced by this call
31  | |         [
32  | |             0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64, 8i64, 655i64, 187i64,
33  | |             172i64, 127i64, 1028i64, 34i64, 57, 27169i64, 1295i64, 8378i64, 38i64, 304i64, 110i64,
...   |
50  | |         ],
51  | |     ])?;
    | |_____^ the trait `FixedInitializer` is not implemented for `[i64; 34]`
    |
    = help: the following other types implement trait `FixedInitializer`:
              [T; 0]
              [T; 10]
              [T; 11]
              [T; 12]
              [T; 13]
              [T; 14]
              [T; 15]
              [T; 16]
            and 9 others
note: required by a bound in `arr2`
   --> C:\Users\guill\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\ndarray-0.15.3\src\free_functions.rs:212:26
    |
212 | pub fn arr2<A: Clone, V: FixedInitializer<Elem = A>>(xs: &[V]) -> Array2<A>
    |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `arr2`

[kali]

This is because 34 is tool long for the trait initializer. OK. from_shape_vec assumes that all your items are in a single vector, so you would need to flatten the nested structure you have been using (giving the function a single flat Vec<i64>, instead of a Vec<[i64; 8]>. So into_iter().flat_map(|v| v.into_iter()).collect() should work.

Alternatively, you can also try from_shape_fn()

[guillaume-be]

Thank you @kali , flattening the vector helps getting past the compilation error. I am now unfortunately faced with a runtime error. The updated code I am attempting to run is:

Rust version

use std::path::Path;
use tract_onnx::prelude::*;
fn main() -> anyhow::Result<()> {
    let model_path = Path::new("path/to/model_optimized.onnx");

    let batch_symbol = Symbol::new('N');
    let sequence_symbol = Symbol::new('S');

    let model = onnx()
        .model_for_path(model_path)?
        .with_input_fact(
            0,
            InferenceFact::dt_shape(
                i64::datum_type(),
                tvec!(batch_symbol.to_dim(), sequence_symbol.to_dim()),
            ),
        )?
        .with_input_fact(
            1,
            InferenceFact::dt_shape(
                i64::datum_type(),
                tvec!(batch_symbol.to_dim(), sequence_symbol.to_dim()),
            ),
        )?
        .into_optimized()?
        .into_runnable()?;

    let input_ids = tract_ndarray::Array2::from_shape_vec(
        (4, 34),
        vec![
            0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64, 8i64, 655i64, 187i64,
            172i64, 127i64, 1028i64, 34i64, 57, 27169i64, 1295i64, 8378i64, 38i64, 304i64, 110i64,
            1553i64, 4i64, 2i64, 2i64, 713i64, 2788i64, 16i64, 59i64, 1830i64, 4i64, 2i64, 1i64,
            0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64, 8i64, 655i64, 187i64,
            172i64, 127i64, 1028i64, 34i64, 57, 27169i64, 1295i64, 8378i64, 38i64, 304i64, 110i64,
            1553i64, 4i64, 2i64, 2i64, 713i64, 2788i64, 16i64, 59i64, 998i64, 4i64, 2i64, 1i64,
            0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64, 8i64, 655i64, 187i64,
            172i64, 127i64, 1028i64, 34i64, 57, 27169i64, 1295i64, 8378i64, 38i64, 304i64, 110i64,
            1553i64, 4i64, 2i64, 2i64, 713i64, 2788i64, 16i64, 59i64, 19892i64, 4i64, 2i64, 1i64,
            0i64, 10285i64, 186i64, 38i64, 7084i64, 127i64, 6894i64, 1732i64, 8i64, 655i64, 187i64,
            172i64, 127i64, 1028i64, 34i64, 57, 27169i64, 1295i64, 8378i64, 38i64, 304i64, 110i64,
            1553i64, 4i64, 2i64, 2i64, 713i64, 2788i64, 16i64, 59i64, 1316i64, 899i64, 4i64, 2i64,
        ],
    )?
    .into();

    let attention_mask = tract_ndarray::Array2::from_shape_vec(
        (4, 34),
        vec![
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1, 1i64,
            1i64, 1i64, 1i64, 1i64, 0i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1, 1i64, 1i64, 1i64, 1i64, 1i64, 0i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1, 1i64, 1i64, 1i64, 1i64, 1i64,
            0i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1i64, 1,
            1i64, 1i64, 1i64, 1i64, 1i64, 1i64,
        ],
    )?
    .into();
    
    let result = model.run(tvec!(input_ids, attention_mask))?;

    println!("{:?}", result);

    Ok(())
}

with error:

Error: Failed analyse for node #217 "Where_20" Iff

Caused by:
    0: Infering facts
    1: Applying rule inputs[2].datum_type == outputs[0].datum_type
    2: Impossible to unify TDim with I64.
error: process didn't exit successfully: `target\debug\examples\onnx-distilbert.exe` (exit code: 1)

Process finished with exit code 1

The model works fine in Python/ONNX. I am uploading it for reference at https://drive.google.com/file/d/19Rnzp6hT477T307tEX6TPdJliJi1ZJhT/view?usp=sharing (shared under the same Apache 2.0 license as the original: https://huggingface.co/cross-encoder/nli-distilroberta-base)

Python version

from onnxruntime import GraphOptimizationLevel, InferenceSession, SessionOptions, get_all_providers
from onnxruntime.transformers import optimizer
from pathlib import Path
from transformers import RobertaTokenizerFast

model_dir = Path("path/to/")
optimized_model_name = model_dir / "model_optimized.onnx"

tokenizer = RobertaTokenizerFast.from_pretrained(""cross-encoder/nli-distilroberta-base")
options = SessionOptions()
options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL
session = InferenceSession(str(optimized_model_name), options, providers=["CPUExecutionProvider"])

input_texts = [
    "This is a pair of sentences",
    "This is sentence 2"
]

encoded_input = tokenizer(input_texts , padding=True, truncation=True, return_tensors="pt")
encoded_input = {key: np.array(value, dtype=np.int64) for key, value in encoded_input.items()}
output = session.run(None, encoded_input)
output

Note that I am directly including the tokens in the Rust version to avoid importing tokenization dependencies.

[kali]

Thanks for the detailed report, I will have a look.

[kali]

I think #757 should fix the last issue. I've just cut a 0.17.1 with the fix.

[guillaume-be]

Thank you @kali ,

Is there a change with compiler dependencies requirements?? I am faced with
error: failed to run custom build command for prost-build v0.10.4 at compile time:

cargo:rerun-if-changed=C:\.cargo\registry\src\github.aaakk.us.kg-1ecc6299db9ec823\prost-build-0.10.4\third-party\protobuf\cmake
  CMAKE_TOOLCHAIN_FILE_x86_64-pc-windows-msvc = None
  CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc = None
  HOST_CMAKE_TOOLCHAIN_FILE = None
  CMAKE_TOOLCHAIN_FILE = None
  CMAKE_GENERATOR_x86_64-pc-windows-msvc = None
  CMAKE_GENERATOR_x86_64_pc_windows_msvc = None
  HOST_CMAKE_GENERATOR = None
  CMAKE_GENERATOR = None
  CMAKE_PREFIX_PATH_x86_64-pc-windows-msvc = None
  CMAKE_PREFIX_PATH_x86_64_pc_windows_msvc = None
  HOST_CMAKE_PREFIX_PATH = None
  CMAKE_PREFIX_PATH = None
  CMAKE_x86_64-pc-windows-msvc = None
  CMAKE_x86_64_pc_windows_msvc = None
  HOST_CMAKE = None
  CMAKE = None
  running: "cmake" "C:\\\\.cargo\\registry\\src\\github.aaakk.us.kg-1ecc6299db9ec823\\prost-build-0.10.4\\third-party\\protobuf\\cmake" "-G" "Visual Studio 16 2019" "-Thost=x64" "-Ax64" "-Dprotobuf_BUILD_TESTS=OFF" "-DCMAKE_INSTALL_PREFIX=E:\\Coding\\rust-bert\\target\\debug\\build\\prost-build-6f904f4a73ff74d8\\out" "-DCMAKE_C_FLAGS= -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_DEBUG= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Debug"
  --- stderr
  thread 'main' panicked at '
  failed to execute command: program not found
  is `cmake` not installed?

[kali]

I don't know :) This is in prost-build build, maybe they change something ? tract windows CI is still in the green. I guess you need either protoc pre-installed or cmake so that the crate can be compiled.

[tgolsson]

This is because of my update in #748. They (prost) added (unfortunately) a dependency on source-protobuf, and thus require cmake to build it. If you install cmake that should fix it. See tokio-rs/prost#620 for more context.

[kali]

@guillaume-be are you ok with closing this ?

[guillaume-be]

Thank you @kali - yes this can be closed. I don't want to impose cmake to consumer of my projects, so I will probably wait for the changes to prost to make their way to this project. Thank you for the update!