Implement Tokenizer op

[yuslepukhin]

Inputs:
X: tensor, string
Outputs:
Y: tensor, string
Attribute:
mark: bool. Whether to mark the beginning/end character with start of text character (0x02)/end of text character (0x03).

padvalue: string. The string used to pad output tensors when the tokens extracted doesn't match the maximum number of tokens found.

separators: list of strings (type: AttributeProto::STRINGS). The list of separators, two consecutive segments in X connected by a separator would be divided into two tokens. For example, if the input is "Hello World!" and this attribute contains only one space character, the corresponding output would be ["Hello", "World!"]. To achieve character-level tokenization, the user should set the separators to [""], which contains only one empty string. This attribute is a 1-D string tensor and each single string in this attribute is a separator. If 'separators' is a L-element array, there will be L rounds of tokenization using one stop word. More specifically, in the first round, the first element in 'separators' is used to tokenize each string in the input. Then, the second element in 'separators' will be used to tokenize the resulted strings produced at the first round.

mincharnum: int. Minimum number of characters allowed in the output. For example, if mincharnum is 2, tokens such as “A” and “B” would be ignored.
Description:
Tokenizer divides each string in X into a vector of strings along the last axis. Allowed input shapes are [C] and [N, C]. If the maximum number of tokens found per input string is D, the output shape would be [N, C, D] when input shape is [N, C]. Similarly, if input shape is [C] then the output should be [C, D]. For example, if input is ["Hello World", "I love computer science !"] whose shape is [2], then the output would be [["Hello", "World", padvalue, padvalue, padvalue], ["I", "love", "computer", "science", "!"]] whose shape is [2, 5] because you can find at most 5 tokens per input string. Note that the input at most can have two axes, so 3-D and higher dimension are not supported.
Let's consider another example where mark is set to true. If input is ["Hello", "World"], then the corresponding output would be [0x02, "Hello", "World", 0x03]. This implies that if mark is true, [C]/[N, C]-input's output shape becomes [C, D+2]/[N, C, D+2]. All input strings including attributes are UTF-8 encoded.

[yuslepukhin]

Need to clarify the behavior if the output is empty.
Also current implementation searches all patterns at once which may not the same when patterns applied sequentially. #Closed

[pranavsharma]

Can we add unit tests for the functions in this file? #WontFix

[yuslepukhin]

It is added though tested indirectly but sufficiently

---

In reply to: 238894458 [](ancestors = 238894458)

[pranavsharma]

Should we mention about UTF-8 in the spec just to be super clear? #Closed

[yuslepukhin]

Yes, will do. The author of the spec claimed that all strings in ONNX are UTF-8. Well, they use it for Bing. So there you go.

---

In reply to: 238894817 [](ancestors = 238894817)

[pranavsharma]

std::make_unique #Closed

[yuslepukhin]

make_unique is not always present in GCC versions.

---

In reply to: 238895544 [](ancestors = 238895544)

[pranavsharma]

Our code makes use of make_unique extensively and relies on it's existence. It's safe to assume it'll be there. I think we even have a function where we define our own make_unique when not present. #Closed

[pranavsharma]

can probably take it outside the while loop. #Closed

[pranavsharma]

May be later we can consider parallelizing the operation for N batches since they're all independent.; might be early optimization at this point since we don't know the nature of the input. Worth adding a TODO to identify opportunities for optimization upfront.

[xadupre]

Why one test and not multiple ones for each case? #Closed

[xadupre]

I added the following unit test, it happens when a separator is a prefix of another one. I don't know if this case is expected but if not we should raised an exception or warning in the constructor.

TEST(ContribOpTest, TokenizerWithSeparators__) {
// Separators and strings with a mix of latin, Spanish, Cyrillic and Chinese
// characters and with start/end text markers
// [C] dimensions
// Output [C][D]
{
std::vectorstd::string separators = {
u8";",
u8";;;"};

OpTester test("Tokenizer", opset_ver, domain);
InitTestAttr(test, true, separators, 1);

std::vector<int64_t> dims{4};
std::vector<std::string> input{u8"a;b", u8"a;;;b", u8"b;c;;;d;e", u8"a;;b;;;c"};
test.AddInput<std::string>("T", dims, input);

std::vector<int64_t> output_dims(dims);
// Must split both in 2
output_dims.push_back(int64_t(6));
std::vector<std::string> output {
    start_mark, u8"a", u8"b", end_mark, padval, padval,
    start_mark, u8"a", u8"b", end_mark, padval, padval, 
    start_mark, u8"b", u8"c", u8"d", u8"e", end_mark,
    start_mark, u8"a", /* u8"", if we allow null string*/ u8"b", u8"c", end_mark, padval,
};

test.AddOutput<std::string>("Y", output_dims, output);

std::cout << "*****\n";
test.Run(OpTester::ExpectResult::kExpectSuccess);

} // namespace test
}

And it returns the following:

a;b

a
b

b;c
d;e

a;;b
c #Closed

[yuslepukhin]

Thank you for the test. I will investigate. The spec allows separators that are prefixes of each other. This is what it says: "If 'separators' is a L-element array, there will be L rounds of tokenization using one stop word. More specifically, in the first round, the first element in 'separators' is used to tokenize each string in the input. Then, the second element in 'separators' will be used to tokenize the resulted strings produced at the first round."

---

In reply to: 239077639 [](ancestors = 239077639)

[yuslepukhin]

I have added this test and fixed the issue according to the spec. We do allow separators with common prefix. The one specified earlier in the sequence will win.

---

In reply to: 239194043 [](ancestors = 239194043,239077639)