# ControlLLM: Augment Language Models with Tools by Searching on Graphs

**Authors:** Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, Wenhai Wang  
**arXiv:** [2310.17796](https://arxiv.org/abs/2310.17796)  
**Date:** 2023-10-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.17796v1/

## Key points

- ControlLLM is a framework designed to enhance the capabilities of large language models (LLMs) in solving complex real-world tasks using multi-modal tools.
- ControlLLM addresses challenges faced by LLMs, such as ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling.
- ControlLLM consists of three key components: Task Decomposer, Thoughts-on-Graph (ToG) Paradigm, and an Execution Engine with Rich Toolbox.
- The authors evaluate ControlLLM on diverse tasks involving image, audio, and video processing.
- Results demonstrate superior accuracy, efficiency, and versatility compared to existing methods.
- ControlLLM is compared with other methods in terms of features that facilitate multi-modal interaction and scalability.
- Different language models (M), such as LLaMA trained through self-instruct method or finetuning an off-the-shelf LLM like GPT4Tools, are considered.
- A benchmark consisting of over 100 instructions classified into three levels of difficulty is built to further evaluate the proposed framework.

## Summary

In this paper, the authors present ControlLLM, a novel framework designed to enhance the capabilities of large language models (LLMs) in solving complex real-world tasks using multi-modal tools. To address the challenges faced by LLMs such as ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling, ControlLLM consists of three key components: Task Decomposer; Thoughts-on-Graph (ToG) Paradigm; and an Execution Engine with Rich Toolbox. The authors evaluate ControlLLM on diverse tasks involving image, audio, and video processing. The results demonstrate its superior accuracy, efficiency, and versatility compared to existing methods. Additionally, they compare ControlLLM with other methods in terms of features that facilitate multi-modal interaction and highlight its high scalability. They also consider different language models (M), such as LLaMA trained through self-instruct method or finetuning an off-the-shelf LLM like GPT4Tools. To evaluate the proposed framework further, the authors build a benchmark consisting of tasks that require various tools to solve complex problems. The benchmark includes over 100 instructions classified into three levels of difficulty: easy, medium, and hard.