# LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

**Authors:** Aoxiao Zhong, Dengyao Mo, Guiyang Liu, Jinbu Liu, Qingda Lu, Qi Zhou, Jiesheng Wu, Quanzheng Li, Qingsong Wen  
**arXiv:** [2408.13727](https://arxiv.org/abs/2408.13727)  
**Date:** 2024-08-25  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2408.13727v1/

## Key points

- Logs are essential digital footprints in system diagnostics, security analysis, and performance optimization.
- Log parsing is crucial for extracting valuable insights from logs by transforming raw data into structured formats for analysis.
- Large Language Models (LLMs) have revolutionized log parsing by providing extensive knowledge and contextual understanding.
- LogParser-LLM is a novel log parser integrated with LLM capabilities that combines semantic insights with statistical nuances for efficient parsing.
- LogParser-LLM addresses the challenge of parsing granularity by introducing a new metric and incorporating human interactions for fine-tuning according to specific requirements.
- Empirical evidence demonstrates LogParser-LLM's efficiency, achieving high grouping accuracy (90.6% F1 score) and parsing accuracy (81.1%) on datasets, surpassing current state-of-the-art log parsers.
- The research authored by Aoxiao Zhong et al. has been accepted by ACM KDD 2024 and falls under primary categories of computer science software engineering (cs.SE) and artificial intelligence (cs.AI).

## Summary

, , , , 

In the realm of system diagnostics, security analysis, and performance optimization, logs serve as essential digital footprints. The process of extracting valuable insights from these logs heavily relies on log parsing, which transforms raw data into structured formats for further analysis. However, the intricate nature of contemporary systems and the dynamic characteristics of logs present significant challenges to existing automatic parsing techniques. The advent of Large Language Models (LLMs) has opened up new possibilities in this domain. Leveraging their extensive knowledge and contextual understanding, LLMs have proven to be transformative across various applications. Building upon this foundation, LogParser-LLM emerges as a novel log parser integrated with LLM capabilities. This integration seamlessly merges semantic insights with statistical nuances, eliminating the need for hyper-parameter tuning and labeled training data while ensuring swift adaptability through online parsing. Delving deeper into the exploration, LogParser-LLM tackles the complex issue of parsing granularity by introducing a new metric and incorporating human interactions to enable users to fine-tune granularity according to their specific requirements. Empirical evidence showcasing the efficacy of this method is demonstrated through evaluations conducted on both the Loghub-2k dataset and the extensive LogPub benchmark. During evaluations on the LogPub benchmark encompassing an average of 3.6 million logs per dataset across 14 datasets, LogParser-LLM showcased remarkable efficiency by requiring only 272.5 LLM invocations on average. It achieved an impressive 90.6% F1 score for grouping accuracy and an 81.1% score for parsing accuracy, surpassing current state-of-the-art log parsers including pattern-based approaches, neural network-based methods, and existing LLM-enhanced techniques. Authored by Aoxiao Zhong, Dengyao Mo, Guiyang Liu, Jinbu Liu, Qingda Lu, Qi Zhou, Jiesheng Wu, Quanzheng Li, and Qingsong Wen; this research has been accepted by ACM KDD 2024 and falls under primary categories of computer science software engineering (cs.SE) and artificial intelligence (cs.AI). This comprehensive study not only highlights the advancements in efficient log parsing facilitated by Large Language Models but also underscores its superiority over existing methodologies in terms of accuracy and effectiveness in log analysis tasks.