Merge remote-tracking branch 'nvidia/master' into u_switch_apex_ddp_t…

…o_torch

Signed-off-by: Jason <[email protected]>

docs/sources/source/nlp/bert_pretraining.rst

@@ -6,6 +6,12 @@ Make sure you have ``nemo`` and ``nemo_nlp`` installed before starting this tuto
 
 The code used in this tutorial can be found at ``examples/nlp/language_modeling/bert_pretraining.py``.
 
+.. tip::
+    Pretrained BERT models can be found at 
+    `https://ngc.nvidia.com/catalog/models/nvidia:bertlargeuncasedfornemo <https://ngc.nvidia.com/catalog/models/nvidia:bertlargeuncasedfornemo>`__
+    `https://ngc.nvidia.com/catalog/models/nvidia:bertbaseuncasedfornemo <https://ngc.nvidia.com/catalog/models/nvidia:bertbaseuncasedfornemo>`__
+    `https://ngc.nvidia.com/catalog/models/nvidia:bertbasecasedfornemo <https://ngc.nvidia.com/catalog/models/nvidia:bertbasecasedfornemo>`__
+
 Introduction
 ------------
 

docs/sources/source/nlp/joint_intent_slot_filling.rst

@@ -9,6 +9,10 @@ There are four pre-trained BERT models that we can select from using the argumen
 using the script for loading pre-trained models from `pytorch_transformers`. See the list of available pre-trained models
 `here <https://huggingface.co/pytorch-transformers/pretrained_models.html>`__. 
 
+.. tip::
+
+    For pretraining BERT in NeMo and pretrained model checkpoints go to `BERT pretraining <https://nvidia.github.io/NeMo/nlp/bert_pretraining.html>`__.
+
 
 Preliminaries
 -------------

docs/sources/source/nlp/ner.rst

@@ -4,6 +4,12 @@ Tutorial
 Make sure you have ``nemo`` and ``nemo_nlp`` installed before starting this
 tutorial. See the :ref:`installation` section for more details.
 
+.. tip::
+
+    For pretraining BERT in NeMo and pretrained model checkpoints go to `BERT pretraining <https://nvidia.github.io/NeMo/nlp/bert_pretraining.html>`__.
+
+
+
 Introduction
 ------------
 

docs/sources/source/nlp/punctuation.rst

@@ -7,6 +7,8 @@ An ASR system typically generates text with no punctuation and capitalization of
 .. tip::
 
     We recommend you to try this example in Jupyter notebook examples/nlp/token_classification/PunctuationWithBERT.ipynb.
+    For pretraining BERT in NeMo and pretrained model checkpoints go to `BERT pretraining <https://nvidia.github.io/NeMo/nlp/bert_pretraining.html>`__.
+
 
 Task Description
 ----------------

docs/sources/source/nlp/question_answering.rst

@@ -13,6 +13,12 @@ The pretrained back-bone models can be specified by `--model_type` and the speci
 See the list of available pre-trained models
 `here <https://huggingface.co/transformers/pretrained_models.html>`__. 
 
+.. tip::
+
+    For pretraining BERT in NeMo and pretrained model checkpoints go to `BERT pretraining <https://nvidia.github.io/NeMo/nlp/bert_pretraining.html>`__.
+
+
+
 Preliminaries
 -------------
 

examples/nlp/language_modeling/bert_pretraining.py

@@ -74,6 +74,15 @@
 
 350000 iterations on a DGX1 with 8 V100 32GB GPUs with AMP O1 optimization
 should finish under 5 days and yield an MRPC score of ACC/F1 85.05/89.35.
+
+More information about BERT pretraining can be found at 
+https://nvidia.github.io/NeMo/nlp/bert_pretraining.html
+
+Pretrained BERT models can be found at 
+https://ngc.nvidia.com/catalog/models/nvidia:bertlargeuncasedfornemo
+https://ngc.nvidia.com/catalog/models/nvidia:bertbaseuncasedfornemo
+https://ngc.nvidia.com/catalog/models/nvidia:bertbasecasedfornemo
+
 """
 import argparse
 import math