Adding Module Models

[tomassosorio]

🚀 Feature

Similar to torchvision would be interesting to have the architecture of the most important models available, such as Wav2Letter, DeepSpeech, DeepSpeech2 between others...

Motivation

When trying to train a model it is already available most features needed such as Datasets, Data Transformations, however it is not available any architecture of a known model.

Pitch

Adding a module with the architecture of the most known models.

[vincentqb]

This is a good idea. Would you like to contribute some?

EDIT: We need a way of validating a model implementation before being able to add it in torchaudio:

• For wav2letter and wavernn, we have a training pipeline showing reasonable convergence.
• In Add MobileNetV3 Architecture in TorchVision vision#3182, we (also) provided pre-trained weights that worked well with the model.
• If this is taken from an existing pytorch implementation, the reference can be used to compare numerically against.

[tomassosorio]

Sure!

I will inspire into torchvision structure and add at first Wav2Letter.

At the time I cannot offer the model with the trained weights, but I can talk with my company to see if we could offer this to torchaudio in the future.

[vincentqb]

Let's start with just the model class, untrained. Hosting trained models (and datasets) is a separate discussion :)

[vincentqb]

@tomassosorio -- thanks again for merging wav2letter! Which other ones do you believe would be a good idea to request help to implement?

• Wav2Letter
• DeepSpeech
• DeepSpeech2

[tomassosorio]

Yes, I fell both of them would be good options as future implementations, such as the Wav2Letter++, in this link there is also some other models that could be interesting to implement :)

[koukyo1994]

How about PANNs? It has a rich collection of pre-trained audio tagging / sed models.

[limazix]

Hi guys, I'd like to help with this task. Where can I start?

[Edresson]

@vincentqb I think it would be interesting to implement Jasper too, as it reported state of the art on LibreSpeech :).

It is available on OpenSeq2Seq

What do you think about converting the checkpoints of these models to pytorch and using them in the initialization of model weights?

[vincentqb]

@vincentqb I think it would be interesting to implement Jasper too, as it reported state of the art on LibreSpeech :).

It is available on OpenSeq2Seq

Thanks for suggesting jasper here, and there :) Do ping me when you open a pull request for it :)

What do you think about converting the checkpoints of these models to pytorch and using them in the initialization of model weights?

We do plan on offering some pre-trained models for those we offer. Is that what you mean?

[vincentqb]

@Edresson -- thanks also for suggestion wav2vec -- it is on our radar, and we would love to have example pipeline such as wav2letter for it :) we can talk more about this if this is of interest to you

[Edresson]

@vincentqb I think it would be interesting to implement Jasper too, as it reported state of the art on LibreSpeech :).
It is available on OpenSeq2Seq

Thanks for suggesting jasper here, and there :) Do ping me when you open a pull request for it :)

What do you think about converting the checkpoints of these models to pytorch and using them in the initialization of model weights?

We do plan on offering some pre-trained models for those we offer. Is that what you mean?

Yes, I thought about maybe converting the openseq2seq checkpoints. But training and providing a checkpoint may be a better option.

[discort]

@vincentqb

I'd like to add vanilla deepspeech model

[mthrok]

Closing the issue as torchaudio now has models submodule.

For adding a new model, please propose it by opening a new issue.