Support for custom architectures for the TextClassifier class

[tkon3]

Hello,
I didn't find a way to use a custom architecture to classify texts so I guess it is not implemented yet.
Currently the TextClassifier class only uses a simple linear layer (nn.Linear) in order to do the task (+sigmoid).

Is allowing custom pytorch layers planned in the futur ?
Something allowing us to specify a custom architecture to the TextClassifier class with torch Sequential or a list :

custom_model = [nn.Linear(embedding_length,50), 
                nn.functional.tanh(nn.Linear(50,25)), 
                nn.Linear(50,len(label_dictionary))
               ]

[NielsRogge]

Sorry I can't answer your question, but where did you find that the TextClassifier only uses a simple linear layer?

[alanakbik]

Hello @tkon3 this is a really good question. We are currently looking into refactoring some parts of Flair such as the flair.nn.model interface and ModelTrainer classes to make it easier for people to define their own architectures and tasks (see #563).

However at this point we don't yet know how exactly this will look like. And since many of us are on vacation until beginning of April, real development of these ideas will begin only then. But yes, generally we want to make it possible for users to do this and perhaps your idea with passing a list of layers could work. Hope this answers the question somewhat - as the ideas develop, we'll keep you posted!

[alanakbik]

Hi @NielsRogge the TextClassifier itself is indeed simply a linear layer on top of whatever DocumentEmbeddings you use. See the forward method of the TextClassifier, which basically takes the embeddings and puts them through the 'decoder':

https://github.com/zalandoresearch/flair/blob/ccb2ffb4080550d25e1e067e2e82e661c8b81411/flair/models/text_classification_model.py#L59

The self.decoder is simply a linear layer, see:

https://github.com/zalandoresearch/flair/blob/ccb2ffb4080550d25e1e067e2e82e661c8b81411/flair/models/text_classification_model.py#L38

However, this does not mean that the classifier is only a linear layer, since the choice of DocumentEmbeddings impacts how the architecture looks like in the end. For instance, most configurations use DocumentRNNEmbeddings which use an RNN over the words in the text to produce an embedding that is then used in the linear layer to make a prediction. Since this RNN is trained on the task, this means that your final archtecture will be word embeddings -> RNN -> linear layer.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.