-
Documentation and the whole project tree updated. No functional improvements.
-
Corrected typos.
-
Notion of binary tokenizer and a library for embedded tokenization.
-
Separation of punctuaction marks.
-
Simple tokenization is desired.
Documentation and the whole project tree updated. No functional improvements.
Corrected typos.
Notion of binary tokenizer and a library for embedded tokenization.
Separation of punctuaction marks.
Simple tokenization is desired.