Merge r1.16.0 release main (#6160)

* update branch Signed-off-by: ericharper <[email protected]> * Tn doc 16 (#5954) * fix new repo links Signed-off-by: Yang Zhang <[email protected]> * fix new repo links Signed-off-by: Yang Zhang <[email protected]> * fix links Signed-off-by: Yang Zhang <[email protected]> * fix spelling Signed-off-by: Yang Zhang <[email protected]> * add warning Signed-off-by: Yang Zhang <[email protected]> * add comment Signed-off-by: Yang Zhang <[email protected]> --------- Signed-off-by: Yang Zhang <[email protected]> * fix links, add missing file (#6044) * fix link, add missing file Signed-off-by: ekmb <[email protected]> * fix primer notebook links Signed-off-by: ekmb <[email protected]> --------- Signed-off-by: ekmb <[email protected]> * Davidm/cherrypick r1.16.0 (#6082) * gpt fix Signed-off-by: David Mosallanezhad <[email protected]> * per-micro-batch input loader (#5635) * per-micro-batch input loader * per-micro-batch input loader set arg default val * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fix * apply per-microbatch-loader to only GPT * update docstring on micro-batch input loader * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed the default arg val * fix batch size to 1 at log stat registration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update container for CI Signed-off-by: ericharper <[email protected]> * update container in jenkinsfile Signed-off-by: ericharper <[email protected]> * update container for CI Signed-off-by: ericharper <[email protected]> fix merge conflict * revert Jenkinsfile * Revert "revert Jenkinsfile" This reverts commit d23b775. * Update nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Signed-off-by: Tim Moon <[email protected]> * add GradScaler * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ericharper <[email protected]> Co-authored-by: Tim Moon <[email protected]> * added PR#5995 Signed-off-by: David Mosallanezhad <[email protected]> * Distributed Adam optimizer overlaps param all-gather with forward compute (#5684) * Add distopt support for overlapping param all-gather with forward compute Signed-off-by: Tim Moon <[email protected]> * Update Apex commit Signed-off-by: Tim Moon <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]> Co-authored-by: Eric Harper <[email protected]> * per-micro-batch input loader (#5635) * per-micro-batch input loader * per-micro-batch input loader set arg default val * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * minor fix * apply per-microbatch-loader to only GPT * update docstring on micro-batch input loader * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed the default arg val * fix batch size to 1 at log stat registration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update container for CI Signed-off-by: ericharper <[email protected]> * update container in jenkinsfile Signed-off-by: ericharper <[email protected]> * update container for CI Signed-off-by: ericharper <[email protected]> fix merge conflict * revert Jenkinsfile * Revert "revert Jenkinsfile" This reverts commit d23b775. * Update nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Signed-off-by: Tim Moon <[email protected]> * add GradScaler * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Tim Moon <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ericharper <[email protected]> Co-authored-by: Tim Moon <[email protected]> * adding early stop callback to ptuning (#6028) * patch to allow using tokenizers without additional_special_tokens_ids attribute Signed-off-by: arendu <[email protected]> * early stop callback for prompt/p tuning Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update Signed-off-by: arendu <[email protected]> * added exp manager config for early stop Signed-off-by: arendu <[email protected]> * pushed logic for creating early stopping inside exp manager Signed-off-by: arendu <[email protected]> * pushed logic for creating early stopping inside exp manager Signed-off-by: arendu <[email protected]> * minor updates and added dataclass check Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * more args Signed-off-by: arendu <[email protected]> * more args Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: ericharper <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: arendu <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Sangkug Lym <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ericharper <[email protected]> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Adi Renduchintala <[email protected]> * update branch for pynini install script (#6150) Signed-off-by: ericharper <[email protected]> * update branch Signed-off-by: ericharper <[email protected]> * revert Signed-off-by: ericharper <[email protected]> * resolve conflict Signed-off-by: ericharper <[email protected]> * revert file add Signed-off-by: ericharper <[email protected]> * revert import Signed-off-by: ericharper <[email protected]> * revert Signed-off-by: ericharper <[email protected]> --------- Signed-off-by: ericharper <[email protected]> Signed-off-by: Yang Zhang <[email protected]> Signed-off-by: ekmb <[email protected]> Signed-off-by: David Mosallanezhad <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: Tim Moon <[email protected]> Signed-off-by: arendu <[email protected]> Co-authored-by: Yang Zhang <[email protected]> Co-authored-by: Evelina <[email protected]> Co-authored-by: David <[email protected]> Co-authored-by: David Mosallanezhad <[email protected]> Co-authored-by: Sangkug Lym <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Tim Moon <[email protected]> Co-authored-by: Adi Renduchintala <[email protected]>
NVIDIA · Mar 9, 2023 · 02ae827 · 02ae827
1 parent 2327fec
commit 02ae827
Show file tree

Hide file tree

Showing 9 changed files with 27 additions and 24 deletions.
diff --git a/examples/nlp/language_modeling/megatron_gpt_pretraining.py b/examples/nlp/language_modeling/megatron_gpt_pretraining.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 
+import torch.multiprocessing as mp
 from omegaconf.omegaconf import OmegaConf, open_dict
 from pytorch_lightning import Trainer
 from pytorch_lightning.plugins.environments import TorchElasticEnvironment
@@ -29,6 +30,8 @@
 from nemo.utils import logging
 from nemo.utils.exp_manager import exp_manager
 
+mp.set_start_method("spawn", force=True)
+
 
 @hydra_runner(config_path="conf", config_name="megatron_gpt_config")
 def main(cfg) -> None:

diff --git a/nemo/collections/nlp/data/language_modeling/megatron/indexed_dataset.py b/nemo/collections/nlp/data/language_modeling/megatron/indexed_dataset.py
@@ -500,10 +500,10 @@ def __init__(self, path, skip_warmup=False):
     def __getstate__(self):
         return self._path
 
-    # def __setstate__(self, state):
-    #     self._do_init(state)
+    def __setstate__(self, state):
+        self._do_init(state)
 
-    def _do_init(self, path, skip_warmup):
+    def _do_init(self, path, skip_warmup=True):
         self._path = path
         self._index = self.Index(index_file_path(self._path), skip_warmup)
 

diff --git a/nemo/package_info.py b/nemo/package_info.py
@@ -14,7 +14,7 @@
 
 
 MAJOR = 1
-MINOR = 16
+MINOR = 17
 PATCH = 0
 PRE_RELEASE = 'rc0'
 

diff --git a/tutorials/AudioTranslationSample.ipynb b/tutorials/AudioTranslationSample.ipynb
@@ -280,4 +280,4 @@
     },
     "nbformat": 4,
     "nbformat_minor": 4
-}
+}
diff --git a/tutorials/VoiceSwapSample.ipynb b/tutorials/VoiceSwapSample.ipynb
@@ -325,4 +325,4 @@
     },
     "nbformat": 4,
     "nbformat_minor": 4
-}
+}
diff --git a/tutorials/tts/FastPitch_ChineseTTS_Training.ipynb b/tutorials/tts/FastPitch_ChineseTTS_Training.ipynb
@@ -61,7 +61,7 @@
     "# !pip install wget text-unidecode matplotlib>=3.3.2\n",
     "\n",
     "## Install NeMo\n",
-    "BRANCH = 'r1.16.0'\n",
+    "BRANCH = 'main'\n",
     "# !python -m pip install \"git+https://github.com/NVIDIA/NeMo.git@${BRANCH}#egg=nemo_toolkit[all]\"\n",
     "\n",
     "## Install pynini\n",

diff --git a/tutorials/tts/FastPitch_Finetuning.ipynb b/tutorials/tts/FastPitch_Finetuning.ipynb
@@ -110,9 +110,9 @@
                 "\n",
                 "```\n",
                 "{\"audio_filepath\": \"audio/dartagnan03part1_027_dumas_0047.wav\", \"text\": \"yes monsieur\", \"duration\": 1.04, \"text_no_preprocessing\": \"Yes, monsieur.\", \"text_normalized\": \"Yes, monsieur.\"}\n",
-                  "```"
-              ]
-          },
+                "```"
+            ]
+        },
         {
             "cell_type": "code",
             "execution_count": null,

diff --git a/tutorials/tts/FastPitch_GermanTTS_Training.ipynb b/tutorials/tts/FastPitch_GermanTTS_Training.ipynb
@@ -61,7 +61,7 @@
     "# !pip install wget text-unidecode matplotlib>=3.3.2\n",
     "\n",
     "## Install NeMo\n",
-    "BRANCH = 'r1.16.0'\n",
+    "BRANCH = 'main'\n",
     "# !python -m pip install \"git+https://github.com/NVIDIA/NeMo.git@${BRANCH}#egg=nemo_toolkit[all]\"\n",
     "\n",
     "## Install pynini\n",

diff --git a/tutorials/tts/NeMo_TTS_Primer.ipynb b/tutorials/tts/NeMo_TTS_Primer.ipynb
@@ -129,7 +129,7 @@
     "While this is the most common structure, there may be fewer or additional steps depending on the use case. For example, some languages do not require G2P and can instead rely on the model to convert raw text/graphemes to spectrogram.\n",
     "\n",
     "<div>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/tts_pipeline.png\" width=\"500\", height=\"500\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/tts_pipeline.png\" width=\"500\", height=\"500\"/>\n",
     "</div>"
    ]
   },
@@ -199,9 +199,9 @@
     "  </tr>\n",
     "</table>\n",
     "\n",
-    "The above examples may be slightly different than the output of the NeMo text normalization code. More details on NeMo text normalization can be found in the [TN documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/text_normalization/intro.html).\n",
+    "The above examples may be slightly different than the output of the NeMo text normalization code. More details on NeMo text normalization can be found in the [TN documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/text_normalization/intro.html).\n",
     "\n",
-    "A more comprehensive list of text normalization rules, examples, and languages are available in the [code](https://github.com/NVIDIA/NeMo/tree/main/nemo_text_processing/text_normalization).\n",
+    "A more comprehensive list of text normalization rules, examples, and languages are available in the [code](https://github.com/NVIDIA/NeMo-text-processing/tree/main/nemo_text_processing/text_normalization).\n",
     "\n"
    ]
   },
@@ -343,7 +343,7 @@
     "\n",
     "For non-phonetic languages like English it is still possible to train a TTS model directly on the graphemes. But doing so will make the pronunciation of some words less accurate.\n",
     "\n",
-    "Details on how NeMo G2P works can be found in our [G2P documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/g2p.html)."
+    "Details on how NeMo G2P works can be found in our [G2P documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tts/g2p.html)."
    ]
   },
   {
@@ -756,7 +756,7 @@
     "\n",
     "<br>\n",
     "<center>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/sampling_rate.png\" width=\"800\", height=\"300\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/sampling_rate.png\" width=\"800\", height=\"300\"/>\n",
     "<figcaption>https://wiki.hydrogenaud.io/index.php?title=File:Digital_wave.png\n",
     "</figcaption>\n",
     "</center>"
@@ -1050,7 +1050,7 @@
     "\n",
     "<br>\n",
     "<div>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/tacotron2_diagram.png\" width=\"600\", height=\"500\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/tacotron2_diagram.png\" width=\"600\", height=\"500\"/>\n",
     "</div>\n",
     "\n",
     "The model is fairly complex. At a high level, it contains:\n",
@@ -1467,7 +1467,7 @@
     "In NeMo we support [FastPitch](https://fastpitch.github.io/), a parallel transformer-based model with pitch and duration control and prediction.\n",
     "\n",
     "<div>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/fastpitch_diagram.png\" width=\"600\", height=\"500\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/fastpitch_diagram.png\" width=\"600\", height=\"500\"/>\n",
     "</div>\n",
     "\n",
     "At a high level it contains:\n",
@@ -1845,7 +1845,7 @@
     "\n",
     "<br>\n",
     "<center>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/wavenet_diagram.png\" width=\"600\", height=\"300\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/wavenet_diagram.png\" width=\"600\", height=\"300\"/>\n",
     "<figcaption>Diagram of a dilated causal CNN</figcaption>\n",
     "<br>\n",
     "</center>\n",
@@ -1885,7 +1885,7 @@
     "In addition to penalizing the model if the discriminator can classify the synthesized audio as fake, it also uses **feature matching loss** to penalize the model if the distribution of intermediate layer outputs in the discriminator networks differ between the real and synthesized audio.\n",
     "\n",
     "<center>\n",
-    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/tts/images/hifi_gan_discriminator_diagram.png\" width=\"600\", height=\"300\"/>\n",
+    "<img src=\"https://raw.githubusercontent.com/NVIDIA/NeMo/stable/tutorials/tts/images/hifi_gan_discriminator_diagram.png\" width=\"600\", height=\"300\"/>\n",
     "<figcaption>HiFi-Gan scale and period discriminators</figcaption>\n",
     "</center>"
    ]
@@ -1977,12 +1977,12 @@
     }
    },
    "source": [
-    "To learn more about what TTS technology and models are available in NeMo, please look through our [documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/intro.html#).\n",
+    "To learn more about what TTS technology and models are available in NeMo, please look through our [documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tts/intro.html#).\n",
     "\n",
-    "To get more hands on experience with NeMo TTS, look through some of our other [tutorials](https://github.com/NVIDIA/NeMo/tree/main/tutorials/tts).\n",
+    "To get more hands on experience with NeMo TTS, look through some of our other [tutorials](https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts).\n",
     "\n",
-    "*   Running pretrained models: [Inference_ModelSelect](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Inference_ModelSelect.ipynb)\n",
-    "*   FastPitch [training](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_MixerTTS_Training.ipynb) and [fine-tuning](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb)\n",
+    "*   Running pretrained models: [Inference_ModelSelect](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/Inference_ModelSelect.ipynb)\n",
+    "*   FastPitch [training](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/FastPitch_MixerTTS_Training.ipynb) and [fine-tuning](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/FastPitch_Finetuning.ipynb)\n",
     "\n",
     "To learn how to deploy and serve your TTS models, visit [Riva](https://docs.nvidia.com/deeplearning/riva/index.html)."
    ]
-Original file line number
+Diff line change
@@ Expand Up / @@ -280,4 +280,4 @@ @@
         },
         "nbformat": 4,
         "nbformat_minor": 4
-    }
+    }