Self-supervised tutorial & update #3344

sam1373 · 2021-12-15T21:33:40Z

Basic tutorial for self-supervised pre-training and subsequent supervised fine-tuning
Config for conformer SSL (may be changed later)
Update to convert_to_tarred script to enable not having "text" field in manifest, and allow using "offset" for manifest without labels
Add optional non-stride layers to reconstruction decoder
Add option to load part of model from second nemo checkpoint from config (for example if we want to load encoder from one pre-trained model, and decoder from another)

Changelog

New way to initialize parts of models into current model for ssl

  init_from_nemo_model: Str path to a .nemo model in order to load state_dict from single nemo file;
  if loading from multiple files, pass in a dict where the values have the following fields:
      - path: Str path to .nemo model
      - include: Optional list of strings, at least one of which needs to be contained in parameter name
      to be loaded from this .nemo file. Default: everything is included.
      - exclude: Optional list of strings, which can be used to exclude any parameter containing one of
      these strings from being loaded from this .nemo file. Default: nothing is excluded.
      hydra usage example:

...

init_from_nemo_model:
    model0:
        path:<path/to/model1>
        include:["encoder"]
    model1:
        path:<path/to/model2>
        include:["decoder"]
        exclude:["embed"]

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2021-12-15T21:46:45Z

This pull request introduces 1 alert and fixes 1 when merging 7bc8f88 into 8ce6e7a - view on LGTM.com

new alerts:

1 for Unnecessary delete statement in function

fixed alerts:

1 for Unnecessary delete statement in function

lgtm-com · 2021-12-15T22:09:46Z

This pull request introduces 1 alert and fixes 1 when merging 690b8ec into 8ce6e7a - view on LGTM.com

new alerts:

1 for Unnecessary delete statement in function

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2021-12-16T01:06:48Z

This pull request fixes 1 alert when merging 1b81e44 into 89910ae - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

okuchaiev · 2021-12-16T06:58:18Z

/blossom-ci

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2021-12-20T04:57:39Z

This pull request fixes 1 alert when merging b8e5bf0 into a3312f3 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2021-12-20T16:10:56Z

This pull request fixes 1 alert when merging d5ded21 into a3312f3 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

lgtm-com · 2021-12-20T16:32:17Z

This pull request fixes 1 alert when merging b42c747 into a3312f3 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

lgtm-com · 2021-12-20T19:17:40Z

This pull request fixes 1 alert when merging 62f4bc9 into eb33ddd - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2021-12-21T02:01:15Z

This pull request fixes 1 alert when merging c899811 into f7e4ed7 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2022-01-24T18:35:41Z

This pull request fixes 1 alert when merging 6d38032 into 7c97e33 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

examples/asr/conf/ssl/citrinet/citrinet_ssl_1024.yaml

examples/asr/conf/citrinet/citrinet_1024.yaml

examples/asr/conf/ssl/conformer/conformer_ssl.yaml

nemo/core/classes/modelPT.py

VahidooX · 2022-01-24T19:54:44Z

nemo/core/classes/modelPT.py


            init_from_pretrained_model: Str name of a pretrained model checkpoint (obtained via cloud).
                The model will be downloaded (or a cached copy will be used), instantiated and then
-                its state dict will be extracted.
+                its state dict will be extracted. If loading from multiple files, you can pass in a dict


How can user pass such a dictionary with hydra?

Added example

tutorials/asr/README.md

VahidooX · 2022-01-24T20:00:13Z

tutorials/asr/README.md

@@ -29,6 +29,8 @@ In this repository, you will find several tutorials discussing what is Automatic

 10) `ASR_with_Transducers`: In this tutorial, we take a deep dive into Transducer based ASR models, discussing the similarity of setup and config to CTC models and then train a small ContextNet model on the AN4 dataset. We then discuss how to change the decoding strategy of a trained Transducer from greedy search to beam search. Finally, we wrap up this tutorial by extraining the alignment matrix from a trained Transducer model. 

+11) `Self_Supervised_Pre_Training`: It can often be difficult to obtain labeled data for ASR training. In this tutorial, we demonstrate how to pre-train a small Citrinet model in an unsupervised manner, and then fine-tune with CTC loss.


How about adding this new feature of self-supervised learning to the readme of nemo?

lgtm-com · 2022-01-24T20:05:54Z

This pull request fixes 1 alert when merging b3564c1 into 7c97e33 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2022-01-24T20:45:59Z

This pull request fixes 1 alert when merging 223e048 into 7c97e33 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

Signed-off-by: sam1373 <[email protected]>

lgtm-com · 2022-01-24T21:06:13Z

This pull request fixes 1 alert when merging 3767be4 into 7c97e33 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

lgtm-com · 2022-01-24T21:50:43Z

This pull request fixes 1 alert when merging 78869b9 into 7c97e33 - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

VahidooX

LGTM!

lgtm-com · 2022-01-26T14:15:23Z

This pull request fixes 1 alert when merging 40823c1 into 360fa7c - view on LGTM.com

fixed alerts:

1 for Unnecessary delete statement in function

* update Signed-off-by: sam1373 <[email protected]> * update Signed-off-by: sam1373 <[email protected]> * version test Signed-off-by: sam1373 <[email protected]> * version test Signed-off-by: sam1373 <[email protected]> * image for tutorial Signed-off-by: sam1373 <[email protected]> * enc_final in model_defaults Signed-off-by: sam1373 <[email protected]> * enc_final in model_defaults Signed-off-by: sam1373 <[email protected]> * self-supervised tutorial Signed-off-by: sam1373 <[email protected]> * fix Signed-off-by: sam1373 <[email protected]> * fix Signed-off-by: sam1373 <[email protected]> * contextnet ssl config Signed-off-by: sam1373 <[email protected]> * remove test_ds from config Signed-off-by: sam1373 <[email protected]> * update recon decoder Signed-off-by: sam1373 <[email protected]> * don't save -last if val_loss is nan Signed-off-by: sam1373 <[email protected]> * check if val_loss is there Signed-off-by: sam1373 <[email protected]> * keep entries from same file together when tarring Signed-off-by: sam1373 <[email protected]> * keep entries from same file together when tarring Signed-off-by: sam1373 <[email protected]> * print num of files in shard Signed-off-by: sam1373 <[email protected]> * update Signed-off-by: sam1373 <[email protected]> * style Signed-off-by: sam1373 <[email protected]> * moving configs, add docstrings Signed-off-by: sam1373 <[email protected]> * tutorial updates Signed-off-by: sam1373 <[email protected]> * update test Signed-off-by: sam1373 <[email protected]> * update loading Signed-off-by: sam1373 <[email protected]> * update loading Signed-off-by: sam1373 <[email protected]> * update loading Signed-off-by: sam1373 <[email protected]> * update loading Signed-off-by: sam1373 <[email protected]> * fix Signed-off-by: sam1373 <[email protected]> * fix Signed-off-by: sam1373 <[email protected]> * citrinet configs Signed-off-by: sam1373 <[email protected]> * citrinet configs update Signed-off-by: sam1373 <[email protected]> * update Signed-off-by: sam1373 <[email protected]> * default include all Signed-off-by: sam1373 <[email protected]> * comments Signed-off-by: sam1373 <[email protected]> * docstring hydra example Signed-off-by: sam1373 <[email protected]>

sam1373 added 8 commits December 15, 2021 02:29

update

60ab6d4

Signed-off-by: sam1373 <[email protected]>

update

93c0e28

Signed-off-by: sam1373 <[email protected]>

version test

fe3926f

Signed-off-by: sam1373 <[email protected]>

version test

ddab8dd

Signed-off-by: sam1373 <[email protected]>

image for tutorial

3f66999

Signed-off-by: sam1373 <[email protected]>

enc_final in model_defaults

9355f94

Signed-off-by: sam1373 <[email protected]>

enc_final in model_defaults

0698b4e

Signed-off-by: sam1373 <[email protected]>

self-supervised tutorial

7bc8f88

Signed-off-by: sam1373 <[email protected]>

Merge branch 'main' into pre_training_5

690b8ec

sam1373 marked this pull request as ready for review December 15, 2021 21:59

sam1373 requested a review from titu1994 December 15, 2021 22:01

sam1373 added 3 commits December 15, 2021 16:48

fix

3de5fbe

Signed-off-by: sam1373 <[email protected]>

Merge remote-tracking branch 'origin/pre_training_5' into pre_training_5

93c7b11

fix

1b81e44

Signed-off-by: sam1373 <[email protected]>

contextnet ssl config

b8e5bf0

Signed-off-by: sam1373 <[email protected]>

remove test_ds from config

d5ded21

Signed-off-by: sam1373 <[email protected]>

sam1373 removed the request for review from titu1994 December 20, 2021 15:59

Merge branch 'main' into pre_training_5

b42c747

Merge branch 'main' into pre_training_5

62f4bc9

sam1373 added 2 commits December 20, 2021 17:49

update recon decoder

ca0ca7f

Signed-off-by: sam1373 <[email protected]>

Merge remote-tracking branch 'origin/pre_training_5' into pre_training_5

c899811

sam1373 added 3 commits January 19, 2022 14:20

update loading

1ff9a88

Signed-off-by: sam1373 <[email protected]>

update loading

b5f2c6d

Signed-off-by: sam1373 <[email protected]>

update loading

7702b34

Signed-off-by: sam1373 <[email protected]>

sam1373 requested a review from titu1994 January 19, 2022 22:45

titu1994 mentioned this pull request Jan 20, 2022

K2 losses #3351

Merged

sam1373 added 7 commits January 20, 2022 03:35

fix

14be631

Signed-off-by: sam1373 <[email protected]>

fix

2cd0d7f

Signed-off-by: sam1373 <[email protected]>

citrinet configs

6106d6c

Signed-off-by: sam1373 <[email protected]>

Merge branch 'main' into pre_training_5

74c9352

citrinet configs update

a00c399

Signed-off-by: sam1373 <[email protected]>

Merge remote-tracking branch 'origin/pre_training_5' into pre_training_5

da1b142

update

6d38032

Signed-off-by: sam1373 <[email protected]>

default include all

b3564c1

Signed-off-by: sam1373 <[email protected]>

VahidooX requested changes Jan 24, 2022

View reviewed changes

comments

223e048

Signed-off-by: sam1373 <[email protected]>

docstring hydra example

3767be4

Signed-off-by: sam1373 <[email protected]>

Merge branch 'main' into pre_training_5

78869b9

VahidooX approved these changes Jan 26, 2022

View reviewed changes

Merge branch 'main' into pre_training_5

40823c1

Merge branch 'main' into pre_training_5

e18caf6

sam1373 merged commit 9dc612e into NVIDIA:main Jan 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Self-supervised tutorial & update #3344

Self-supervised tutorial & update #3344

sam1373 commented Dec 15, 2021 •

edited by titu1994

Loading

lgtm-com bot commented Dec 15, 2021

lgtm-com bot commented Dec 15, 2021

lgtm-com bot commented Dec 16, 2021

okuchaiev commented Dec 16, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 21, 2021

lgtm-com bot commented Jan 24, 2022

VahidooX Jan 24, 2022

sam1373 Jan 24, 2022

VahidooX Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

VahidooX left a comment

lgtm-com bot commented Jan 26, 2022

		@@ -29,6 +29,8 @@ In this repository, you will find several tutorials discussing what is Automatic

		10) `ASR_with_Transducers`: In this tutorial, we take a deep dive into Transducer based ASR models, discussing the similarity of setup and config to CTC models and then train a small ContextNet model on the AN4 dataset. We then discuss how to change the decoding strategy of a trained Transducer from greedy search to beam search. Finally, we wrap up this tutorial by extraining the alignment matrix from a trained Transducer model.

		11) `Self_Supervised_Pre_Training`: It can often be difficult to obtain labeled data for ASR training. In this tutorial, we demonstrate how to pre-train a small Citrinet model in an unsupervised manner, and then fine-tune with CTC loss.

Self-supervised tutorial & update #3344

Self-supervised tutorial & update #3344

Conversation

sam1373 commented Dec 15, 2021 • edited by titu1994 Loading

Changelog

lgtm-com bot commented Dec 15, 2021

lgtm-com bot commented Dec 15, 2021

lgtm-com bot commented Dec 16, 2021

okuchaiev commented Dec 16, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 20, 2021

lgtm-com bot commented Dec 21, 2021

lgtm-com bot commented Jan 24, 2022

VahidooX Jan 24, 2022

Choose a reason for hiding this comment

sam1373 Jan 24, 2022

Choose a reason for hiding this comment

VahidooX Jan 24, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

lgtm-com bot commented Jan 24, 2022

VahidooX left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jan 26, 2022

sam1373 commented Dec 15, 2021 •

edited by titu1994

Loading