In their work titled "Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search," authors Howard Yen, Ashwin Paranjape, Mengzhou Xia, Thejas Venkatesh, Jack Hessel, Danqi Chen, and Yuhao Zhang address the challenges of long-horizon agentic search. This type of search involves iteratively exploring the web over extended trajectories and synthesizing information from multiple sources to enable advanced applications like deep research systems. The authors highlight that existing agentic search frameworks struggle to scale to long trajectories due to context limitations. These frameworks often accumulate lengthy and noisy content, exceed context window and tool budgets, or terminate prematurely. To address these issues, the authors introduce SLIM (Simple Lightweight Information Management), a straightforward framework that separates retrieval into distinct search and browse tools. SLIM periodically summarizes the trajectory to maintain concise context while facilitating longer and more focused searches. Through experimental evaluation on long-horizon tasks, SLIM demonstrates comparable performance at significantly lower cost and with fewer tool calls compared to strong open-source baselines across various base models. Notably, when using o3 as the base model, SLIM achieves 56% on BrowseComp and 31% on HLE tasks, surpassing all open-source frameworks by 8 and 4 absolute points respectively while requiring only 4-6 times fewer tool calls. Furthermore,the authors release an automated fine-grained trajectory analysis pipeline along with an error taxonomy for characterizing long-horizon agentic search frameworks. They find that SLIM exhibits fewer hallucinations compared to previous systems. The hope is that their analysis framework and simple tool design will guide future developments in long-horizon agents. For those interested in exploring further,the code and data related to this study are available at https://github.com/howard-yen/SLIM.
- Long-horizon agentic search involves exploring the web over extended trajectories and synthesizing information from multiple sources for advanced applications.
- Existing agentic search frameworks struggle with scaling to long trajectories due to context limitations, noisy content accumulation, exceeding context window and tool budgets, or premature termination.
- Authors introduce SLIM (Simple Lightweight Information Management) as a framework that separates retrieval into distinct search and browse tools, maintaining concise context for longer and more focused searches.
- Experimental evaluation shows that SLIM performs comparably at lower cost and with fewer tool calls compared to strong open-source baselines across various base models.
- When using o3 as the base model, SLIM achieves 56% on BrowseComp and 31% on HLE tasks, surpassing all open-source frameworks by 8 and 4 absolute points respectively while requiring only 4-6 times fewer tool calls.
- Authors release an automated fine-grained trajectory analysis pipeline along with an error taxonomy for characterizing long-horizon agentic search frameworks, showing that SLIM exhibits fewer hallucinations compared to previous systems.
- The hope is that their analysis framework and simple tool design will guide future developments in long-horizon agents.
Summary- Long-horizon agentic search means looking for information on the internet over a long period and combining details from different sources for advanced uses.
- Some current search methods have trouble handling long searches because of limits in understanding context, collecting noisy content, going over budget, or stopping early.
- SLIM (Simple Lightweight Information Management) is a new way to search that splits retrieval into separate tools for searching and browsing, keeping focus during longer searches.
- Tests show that SLIM works well at a lower cost and with fewer tool uses compared to other popular methods across different models.
- When using o3 as the base model, SLIM outperforms other frameworks by achieving higher scores on tasks while needing much fewer tool calls.
Definitions- Agentic: Having the ability to act independently or make decisions.
- Trajectories: The paths or routes followed during a journey or process.
- Synthesizing: Combining different elements to form a coherent whole.
- Framework: A basic structure used as a guide for building something more complex.
- Baselines: Standard or reference points used for comparison in experiments.
Introduction
In today's digital age, the ability to search and retrieve information from vast amounts of data is crucial for various applications such as deep research systems. However, traditional search frameworks often struggle to handle long-horizon agentic search tasks, which involve iteratively exploring the web over extended trajectories and synthesizing information from multiple sources. In their paper titled "Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search," authors Howard Yen, Ashwin Paranjape, Mengzhou Xia, Thejas Venkatesh, Jack Hessel, Danqi Chen, and Yuhao Zhang address these challenges and propose a new framework called SLIM (Simple Lightweight Information Management).
Context Limitations in Long-Horizon Agentic Search
The authors highlight that existing agentic search frameworks face several limitations when it comes to handling long-horizon tasks. These limitations include accumulating lengthy and noisy content, exceeding context window and tool budgets, or terminating prematurely. This can result in inefficient use of resources and suboptimal performance.
Introducing SLIM
To overcome these limitations, the authors introduce SLIM - a straightforward framework that separates retrieval into distinct search and browse tools. This separation allows for periodic summarization of the trajectory to maintain concise context while facilitating longer and more focused searches.
Experimental Evaluation
To evaluate the effectiveness of SLIM on long-horizon tasks, the authors conducted experiments using various base models such as o3 on BrowseComp (a task involving browsing through webpages) and HLE (a task involving finding relevant information within a webpage). The results showed that SLIM achieved comparable performance at significantly lower cost with fewer tool calls compared to strong open-source baselines.
Automated Fine-Grained Trajectory Analysis Pipeline
In addition to proposing a new framework for long-horizon agentic search tasks, the authors also released an automated fine-grained trajectory analysis pipeline. This pipeline, along with an error taxonomy for characterizing long-horizon agentic search frameworks, can help guide future developments in this area.
Reducing Hallucinations
One of the key findings from the analysis was that SLIM exhibited fewer hallucinations compared to previous systems. This is a significant improvement as hallucinations can lead to incorrect information being retrieved and affect the overall performance of the system.
Conclusion
In conclusion, "Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search" presents a new framework called SLIM that addresses the challenges faced by existing agentic search frameworks when it comes to handling long-horizon tasks. Through experimental evaluation and automated fine-grained trajectory analysis, the authors demonstrate the effectiveness of SLIM in terms of performance and reducing hallucinations. The hope is that their work will guide future developments in long-horizon agents and facilitate more efficient and accurate retrieval of information from vast amounts of data.
For those interested in exploring further, all code and data related to this study are available at https://github.com/howard-yen/SLIM. With its simple tool design and promising results, SLIM has opened up new possibilities for long-horizon agentic search tasks and could potentially be applied to other areas such as natural language processing or artificial intelligence research.
