Generative Pre-Training: GPT-2 vs. GPT-3
OpenAI’s language models GPT-2 and GPT-3 have revolutionized natural language processing (NLP) technology through their Transformer-based language models. When GPT-2 was first released in 2019 in OpenAI’s paper Language Models are Unsupervised Multitask Learners [1] it was groundbreaking, leading to extensions by Nvidia (Megatron-LM, 2020) and by Microsoft (Turing-NLG, 2020). In this review, I will summarize the two key papers by OpenAI that have motivated much of the recent R&D around notional.ai by East Agile.
