AI Research - A Comprehensive Survey of Large Language Models
A timeline of existing large language models (having a size larger than 10B) in recent years. Open-source LLMs are marked in yellow color.
Image Credit : Article
Overview
Large language models (LLMs) have become one of the most significant breakthroughs in natural language processing (NLP) in recent years. With the development of deep learning and the availability of massive amounts of text data, LLMs have achieved remarkable performance in various NLP tasks, such as language generation, translation, summarization, and question-answering. In this comprehensive survey, the authors provide a detailed review of the recent progress in LLMs and their key concepts, findings, and techniques.
The survey focuses on large-sized models with a size larger than 10 billion parameters, excluding the early pre-trained language models like BERT and GPT-2 that have been extensively covered in the existing literature. The survey discusses four important aspects of LLMs, i.e., pre-training, adaptation tuning, utilization, and evaluation, and highlights the techniques and findings that are key to the success of LLMs in each aspect.
The authors also summarize the available resources for developing LLMs and discuss important implementation guidelines for reproducing LLMs. Furthermore, they introduce the challenges and future directions for LLMs in theory and principle, model architecture, model training, model utilization, safety and alignment, and application and ecosystem. The authors call for more formal theories and principles to understand and explain the behaviors of LLMs, as well as more efficient Transformer variants in building LLMs.
Model training
The authors suggest developing more systemic, economical pre-training approaches for optimizing LLMs, considering the factors of model effectiveness, efficiency optimization, and training stability. They also call for more open-source models with complete preprocessing and training logs for reproducing LLMs. Furthermore, the authors emphasize the importance of including safety-relevant prompts during reinforcement learning from human feedback (RLHF) and improving the RLHF framework for reducing the efforts of human labelers and seeking a more efficient annotation approach with guaranteed data quality.
Model utilization
The authors suggest developing more informative, flexible task formatting methods to describe complex tasks that require specific knowledge or logic rules. They also suggest automating the generation of effective prompts for solving various tasks to reduce human efforts in prompt design.
Safety and alignment
The authors suggest establishing the learning mechanism for LLMs by communicating with humans, where the feedback given by humans via chatting can be directly utilized by LLMs for self-improvement. They also emphasize the importance of AI safety in the development of LLMs, making AI lead to good for humanity but not bad.
Application and ecosystem
The authors suggest that LLMs have a significant impact on information-seeking techniques, including both search engines and recommender systems. They also suggest that the development and use of intelligent information assistants would be highly promoted with the technology upgrade from LLMs. Lastly, the authors suggest that the rising of LLMs sheds light on the exploration of artificial general intelligence (AGI) and the development of more smart intelligent systems with multi-modality signals.
Conclusion
This survey provides a comprehensive review of the recent progress in LLMs and their key concepts, findings, and techniques. The survey also introduces the challenges and future directions for LLMs in theory and principle, model architecture, model training, model utilization, safety and alignment, and application and ecosystem. The authors call for more formal theories and principles to understand and explain the behaviors of LLMs, as well as more efficient Transformer variants in building LLMs. Furthermore, the authors emphasize the importance of AI safety in the development of LLMs, making AI lead to good for humanity but not bad, and suggest establishing the learning mechanism for LLMs by communicating with humans.