# LogShrink: Effective Log Compression by Leveraging Commonality and Variability of Log Data

**Authors:** Xiaoyun Li, Hongyu Zhang, Van-Hoang Le, Pengfei Chen  
**arXiv:** [2309.09479](https://arxiv.org/abs/2309.09479)  
**Date:** 2023-09-18  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2309.09479v1/

## Key points

- Log data plays a vital role in system execution by capturing events and states.
- The increasing scale of systems has led to a surge in log data generation, reaching several petabytes per day in production environments.
- Log compression is crucial to reduce storage burden while enabling comprehensive log analysis.
- Existing compression methods struggle to effectively leverage the unique characteristics of log data.
- A comprehensive empirical study identified key characteristics of log data that can enhance the compression process.
- LogShrink was introduced as an innovative and efficient log compression method that utilizes commonality and variability in log data.
- LogShrink employs an analyzer based on longest common subsequence and entropy techniques to identify shared patterns and differences within log messages for more compact representations.
- The clustering-based sequence sampler in LogShrink expedites the identification of commonality and variability in log data.
- Extensive experiments using 16 public datasets showcased LogShrink's superior performance over specific and general-purpose compression methods, achieving compression ratios ranging from 16% to 356% on average while maintaining reasonable speeds.

## Summary

In the realm of system execution, log data plays a vital role in capturing events and states. However, with the increasing scale of systems, the generation of log data has surged dramatically. This results in a substantial burden on storage resources, reaching several petabytes per day in production environments. To mitigate this challenge, log compression has emerged as a critical task to reduce disk storage while enabling comprehensive log analysis. Unfortunately, existing compression methods have struggled to effectively leverage the unique characteristics of log data. To address these limitations, a comprehensive empirical study was conducted to delve into the intricacies of log data. Three key observations were made regarding its characteristics that could significantly aid in enhancing the compression process. Building upon these insights, LogShrink was introduced as an innovative and efficient log compression method that capitalizes on the commonality and variability present in log data. By employing an analyzer based on longest common subsequence and entropy techniques, LogShrink identifies shared patterns and differences within log messages to achieve more compact representations. A pivotal component of LogShrink is its clustering-based sequence sampler which expedites the identification of commonality and variability in log data. Extensive experiments were carried out using 16 public datasets from diverse software systems to evaluate LogShrink's performance. The results showcased its superiority over two other specific compression methods for logs and three general-purpose compression methods by achieving compression ratios ranging from 16% to 356% on average while maintaining reasonable speeds. The ablation study conducted on LogShrink further validated the efficacy of its commonality and variability analyzer along with the clustering-based sequence sampling technique in enhancing both compression ratio and speed. In summary, this paper contributes significantly by conducting empirical studies on real-world log datasets, proposing a novel approach for log compression based on commonality and variability analysis, showcasing superior performance through extensive experiments, and making the tool's source code and experimental data publicly available for further research endeavors.