# StreamingRAG: Real-time Contextual Retrieval and Generation Framework

**Authors:** Murugan Sankaradas, Ravi K. Rajendran, Srimat T. Chakradhar  
**arXiv:** [2501.14101](https://arxiv.org/abs/2501.14101)  
**Date:** 2025-01-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.14101v1/

## Key points

- The paper addresses the challenge of extracting real-time insights from multi-modal data streams in domains such as healthcare, intelligent transportation, and satellite remote sensing.
- Limitations faced when applying Multi-Modal Large Language Models (MM-LLMs) to these data streams include high computational demands and restricted knowledge scope.
- Traditional Retrieval-Augmented Generation (RAG) systems have been used to address knowledge limitations but suffer from slow preprocessing, making them unsuitable for real-time analysis.
- To overcome these challenges, the authors propose StreamingRAG, a novel RAG framework designed for streaming data that leverages evolving knowledge graphs capturing scene-object-entity relationships in real-time.
- StreamingRAG utilizes MM-LLMs to achieve temporal-aware scene representations and enable timely responses for specific events or user queries, leading to 5-6x faster throughput compared to existing methods.
- The framework enhances contextual accuracy through the implementation of a temporal knowledge graph while reducing resource consumption by using lightweight models by 2-3x.
- The innovative framework showcased in the paper can revolutionize real-time analysis across various domains by providing more efficient and accurate insights from multi-modal data streams.

## Summary

The paper addresses the challenge of extracting real-time insights from multi-modal data streams in domains such as healthcare, intelligent transportation, and satellite remote sensing. The authors highlight the limitations faced due to high computational demands and restricted knowledge scope when applying Multi-Modal Large Language Models (MM-LLMs) to these data streams. Traditional Retrieval-Augmented Generation (RAG) systems have been used to address the knowledge limitations of MM-LLMs; however, they suffer from slow preprocessing, making them unsuitable for real-time analysis. To overcome these challenges, the authors propose StreamingRAG, a novel RAG framework specifically designed for streaming data. StreamingRAG leverages evolving knowledge graphs that capture scene-object-entity relationships in real-time. By utilizing MM-LLMs, the framework achieves temporal-aware scene representations and enables timely responses for specific events or user queries. This approach not only addresses the limitations of existing methods but also leads to significant improvements in real-time analysis with a 5-6x faster throughput. Moreover, StreamingRAG enhances contextual accuracy through the implementation of a temporal knowledge graph while simultaneously reducing resource consumption by using lightweight models by 2-3x. The authors' work showcases how this innovative framework can revolutionize real-time analysis across various domains by providing more efficient and accurate insights from multi-modal data streams. The paper was accepted and presented at AI4Sys during HPDC 2024, underscoring its significance in advancing research in artificial intelligence systems.