---

RNN Architectures

Structure

• Bi-directional RNN [Paper]
• Multi-dimensional RNN [Paper]
• GFRNN [Paper-arXiv] [Paper-ICML] [Supplementary]
• Tree-Structured RNNs [Paper] [Paper]
• Grid LSTM [Paper] [Code]
• Segmental RNN [Paper]
• Seq2seq for Sets [Paper]
• Hierarchical Recurrent Neural Networks [Paper]

Memory

• LSTM [Paper] Paper
• GRU (Gated Recurrent Unit) [Paper]
• NTM [Paper] [Paper]
• Neural GPU [Paper]
• Memory Network [Paper]
• Pointer Network [Paper]
• Deep Attention Recurrent Q-Network [Paper]
• Dynamic Memory Networks [Paper]

---

Timeline of RNNs

---

Comparision of LSTM networks

---

Comparision of RNN networks

---

NLP

Basic Embedding Model

• NNLM(Neural Network Language Model) - Predict Next Word [Paper(2003)] [NNLM_Tensor.ipynb] [NNLM_Torch.ipynb]
• Word2Vec(Skip-gram) - Embedding Words and Show Graph [Distributed Representations of Words and Phrases and their Compositionality(2013)] [Word2Vec_Tensor(NCE_loss).ipynb] [Word2Vec_Tensor(Softmax).ipynb] [Word2Vec_Torch(Softmax).ipynb]
• FastText(Application Level) - Sentence Classification [Bag of Tricks for Efficient Text Classification(2016)] [FastText.ipynb]

CNN(Convolutional Neural Network)

• TextCNN - Binary Sentiment Classification [Paper(2014)] [Colab] [TextCNN_Torch.ipynb
• DCNN(Dynamic Convolutional Neural Network)

RNN(Recurrent Neural Network)

• TextRNN - Predict Next Step [Finding Structure in Time(1990)] [TextRNN_Tensor.ipynb] [TextRNN_Torch.ipynb]
• TextLSTM - Autocomplete [LONG SHORT-TERM MEMORY(1997)] [TextLSTM_Tensor.ipynb] [TextLSTM_Torch.ipynb]
• Bi-LSTM - Predict Next Word in Long Sentence [Bi_LSTM_Tensor.ipynb] [Bi_LSTM_Torch.ipynb]

Attention Mechanism

• Seq2Seq - Change Word [Paper(2014)] [Colab] [Seq2Seq_Torch.ipynb]
• Seq2Seq with Attention - Translate [Paper(2014)] [Colab] [Seq2Seq(Attention)_Torch.ipynb]
• Bi-LSTM with Attention - Binary Sentiment Classification [Bi_LSTM(Attention)_Tensor.ipynb] [Bi_LSTM(Attention)_Torch.ipynb]

Model based on Transformer

• The Transformer - Translate [Paper(2017)] [Colab], [Transformer(Greedy_decoder)_Torch.ipynb]
• BERT - Classification Next Sentence & Predict Masked Tokens [Paper(2018)] [Colab]

---

---

---

---

NLP Pre-Trained Models

---

Taxonomy of Pre-Trained Models (NLP)

---

---

Benchmarks

---

---

---

source : graykode/nlp-roadmap

---

source : graykode/nlp-roadmap

---

References

• NLP SOTA
• rnn-surveys
• ref-implementations
• language-modeling
• conversation-modeling
• machine-translation
• qa
• speech-processing
• vision-nlp
• rnn-vision
• rnn-robot
• turing-machines
• rnn-other
• BERT-related-papers
• NiuTrans/ABigSurvey

---

• A Survey: Time Travel in Deep Learning Space: An Introduction to Deep Learning Models and How Deep Learning Models Evolved from the Initial Ideas
• Survey on the attention based RNN model and its applications in computer vision
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Best Practices for Applying Deep Learning to Novel Applications
• ParlAI: A Dialog Research Software Platform
• Statistical Machine Translation
• Adversarial Examples: Attacks and Defenses for Deep Learning
• Deep Learning: A Critical Appraisal
• From Word to Sense Embeddings: A Survey on Vector Representations of Meaning
• Natural Language Processing for Information Extraction
• A Review of the Neural History of Natural Language Processing
• EMNLP 2018 Highlights: Inductive bias, cross-lingual learning, and more
• A Survey of the Usages of Deep Learning in Natural Language Processing
• Adversarial Attacks and Defences: A Survey
• Secure Deep Learning Engineering: A Software Quality Assurance Perspective
• Tackling Sequence to Sequence Mapping Problems with Neural Networks
• Security for Machine Learning-based Systems: Attacks and Challenges during Training and Inference
• Speech processing: recognition, synthesis + Survey on chatbot platforms and API's
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Deep RNN Framework for Visual Sequential Applications
• EcoRNN: Efficient Computing of LSTM RNN Training on GPUs
• Training for 'Unstable' CNN Accelerator:A Case Study on FPGA
• Modular Mechanistic Networks: On Bridging Mechanistic and Phenomenological Models with Deep Neural Networks in Natural Language Processing
• Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing
• Attention, please! A Critical Review of Neural Attention Models in Natural Language Processing
• Recent Trends in Deep Learning Based Natural Language Processing
• Quantifying Uncertainties in Natural Language Processing Tasks
• A Survey on Natural Language Processing for Fake News Detection
• Visualizing memorization in RNNs
• Language Models are Few-Shot Learners | Paper Explained | openai/gpt-3 | OpenAI GPT-3
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• A Survey of Evaluation Metrics Used for NLG Systems
• NLPStatTest: A Toolkit for Comparing NLP System Performance
• Robustness Gym: Unifying the NLP Evaluation Landscape
• Exploring and Predicting Transferability across NLP Tasks
• Neuron-level Interpretation of Deep NLP Models: A Survey
• A Survey of Data Augmentation Approaches for NLP
• A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLP
• Language (Technology) is Power: A Critical Survey of "Bias" in NLP
• Taxonomic survey of Hindi Language NLP systems
• An Introductory Survey on Attention Mechanisms in NLP Problems
• Indian Legal NLP Benchmarks : A Survey
• Post-hoc Interpretability for Neural NLP: A Survey
• Explanation-Based Human Debugging of NLP Models: A Survey
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

---

• linguistics
• Machine translation
• awesome-nlp
• awesome-bert
• PLMpapers
• nlp-tutorial
• nlp_tasks
• DeepNLP-models-Pytorch
• oxford.nlp.lectures
• stanford.nlp.lectures
• nltk.org/book
• DL4NLP
• cs388.utexas.nlp
• nlp-datasets
• DL-NLP-Readings
• gt-nlp-class
• embedding-models
• Facebook: Advancing understanding at ACL 2017
• Facebook: Visual reasoning and dialog
• ilya_sutskever_phd_thesis
• Notes on state of the art techniques for language modeling
• ASR 2017-18: lectures
• Sebastian Ruder
• wer_are_we
• NLP-progress
• NLP-Models-Tensorflow
• graykode/nlp-roadmap
• huggingface/nlp | huggingface channel
• bert-nlp
• The Future of Natural Language Processing
• paperswithcode/natural-language-processing
• paperswithcode/speech
• nlp-methods
• practical-nlp/practical-nlp
• ICLR 2020: NLP Highlights

---

Maintainer

Gopala KR / @gopala-kr

---