A Survey on Language Models for Code

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   A Survey on Large Language Models with some Insights on their Capabilities and Limitations

   Andrea Matarazzo, Riccardo Torlone · 2025 · Computation and Language · AI summary available

   Abstract

   The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and archite…

2. Very close Not relevant

   A Survey of Large Language Models

   Wayne Xin Zhao, Kun Zhou, Junyi Li et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study t…

3. Close Not relevant

   Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

   Weigao Sun, Jiaxi Hu, Yucheng Zhou et al. · 2025 · Computation and Language · AI summary available

   Abstract

   Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties. However, the traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment. In this survey, we offer a systematic examination of innovative LLM architectures that address the inherent limitations of transformers and boost the efficiency. Starting from language modeling, this survey covers the background and technical details of…

4. Close Not relevant

   A Survey on Evaluation of Large Language Models

   Yupeng Chang, Xu Wang, Jindong Wang et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the …

5. Close Not relevant

   AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

   Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, Sivanesan Sangeetha · 2021 · Computation and Language · AI summary available

   Abstract

   Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core conc…

6. Close Not relevant

   Augmented Language Models: a Survey

   Grégoire Mialon, Roberto Dessì, Maria Lomeli et al. · 2023 · Computation and Language · AI summary available

   Abstract

   This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. The former is defined as decomposing a potentially complex task into simpler subtasks while the latter consists in calling external modules such as a code interpreter. LMs can leverage these augmentations separately or in combination via heuristics, or learn to do so from demonstrations. While adhering to a standard missing tokens prediction objective, such augmented LMs can use various, possibly non-parametric external modules to expand their context processing ability, thus departing from the pure language modeling paradigm. We therefore refer to them as Augmented Lang…

7. Close Not relevant

   Large Language Models on Graphs: A Comprehensive Survey

   Bowen Jin, Gang Liu, Chi Han et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs), such as ChatGPT and LLaMA, are creating significant advancements in natural language processing, due to their strong text encoding/decoding ability and newly found emergent capability (e.g., reasoning). While LLMs are mainly designed to process pure texts, there are many real-world scenarios where text data are associated with rich structure information in the form of graphs (e.g., academic networks, and e-commerce networks) or scenarios where graph data are paired with rich textual information (e.g., molecules with descriptions). Besides, although LLMs have shown their pure text-based reasoning ability, it is underexplored whether such ability can be generalize…

8. Close Not relevant

   LLM Post-Training: A Deep Dive into Reasoning Large Language Models

   Komal Kumar, Tajamul Ashraf, Omkar Thawakar et al. · 2025 · Computation and Language · AI summary available

   Abstract

   Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Pretraining on vast web-scale data has laid the foundation for these models, yet the research community is now increasingly shifting focus toward post-training techniques to achieve further breakthroughs. While pretraining provides a broad linguistic foundation, post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations. Fine-tuning, reinforcement learning, and test-time scaling have emerged as critical strategies for optimizing LLMs performance, …