Transfer learning enables using pretrained deep neural networks trained on various large datasets and adapt them for various tasks. Fine-tuning such pre-trained models in computer vision has been a far more common practice than training from scratch. In NLP, however, due to the lack of models pretrained on large corpus, the most common transfer learning technique had been fine-tuning pretrained word embeddings. These embeddings are used as the first layer of the model on the new dataset, and still require training from scratch with large amounts of labeled data to obtain good performance. 

 

Finally, over the past year, several pretrained language models (ULMFiT, OpenAI GPT, BERT, etc) emerged.These models are trained on very large corpus, and enable robust transfer learning for fine-tuning many NLP tasks with little labeled data. 

 

In this talk we'll learn the architecture of these pretrained language models. In particular, we'll share how different transfer learning techniques have been used with BERT to solve various downstream tasks in the NLP community.