Improve tokenization API

[mariosaenger]

Currently, flair supports tokenization only in a rather simple, non-standardised form, using different functions / callables. In course of #1513 the need for a more sophisticated implementation emerged

The goal of this issue is to provide a general tokenization programming interface based on a abstract superclass. The new implementation should enable users to define and use custom tokenizers easily.

[mauryaland]

Isn't it already the case with #1022 ?

[mariosaenger]

Right #1022 introduces a consistent function interface for tokenizers. However, modelling tokenizers just as functions is insufficient / disadvantageous in some situations. Take the integration of NER data sets in Flair as an example. In general, these data sets are integrated by downloading and preparing them upon first instantiation, leading to (fixed) CONLL files. Any further instantiation will just load the prepared CONLL files. If the data set isn't pre-tokenized, tokenization has to be done during the preparation steps. However, persisting this tokenization doesn't enable the user to change the desired tokenization after the first instantiation.

But good point anyway. Maybe we have a too restricted view to solve this issue.

@alanakbik: Do you have any further ideas / arguments / plans why changing tokenizers to classes?

[alanakbik]

One advantage is that classes can be passed variables when initializing, which is more convenient for instance if a tokenizer requires parameterization (otherwise, you would always have to pass the same parameters with the tokenize method call).