# CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation

**Authors:** Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, Heng Ji  
**arXiv:** [2305.14318](https://arxiv.org/abs/2305.14318)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14318v1/

## Key points

- Large Language Models (LLMs) have made advancements in utilizing external APIs for various tasks
- Limitations of LLMs include availability of suitable APIs and instability of implicit reasoning
- CREATOR framework empowers LLMs to create their own tools through documentation and code realization
- CREATOR separates the LLM's ability into abstract tool creation and concrete decision execution phases
- CREATOR improves the performance of LLMs by separating these phases
- Experiments on MATH and TabMWP benchmarks show that CREATOR outperforms existing baselines
- A new dataset called Creation Challenge highlights the necessity and benefits of LLMs' tool creation ability
- Leveraging LLMs as tool creators facilitates knowledge transfer between domains
- LLMs exhibit varying levels of tool creation abilities, enabling them to tackle diverse situations
- The study represents a promising avenue for maximizing the potential of LLMs towards intelligent AI systems.

## Summary

In recent years, Large Language Models (LLMs) have made significant advancements in utilizing external APIs as tools for various tasks. However, their ability to use these tools is limited by the availability of suitable APIs and the instability of implicit reasoning when engaging in reasoning about plans and calculations simultaneously. To overcome these limitations, a team of researchers proposes a novel framework called CREATOR. CREATOR empowers LLMs to create their own tools through documentation and code realization. It disentangles the LLM's ability into two distinct phases: abstract tool creation and concrete decision execution. By separating these phases, CREATOR improves the performance of LLMs. To evaluate the effectiveness of CREATOR, the researchers conducted experiments on two established benchmarks: MATH and TabMWP. The MATH benchmark consists of challenging math competition problems, while TabMWP includes diverse tabular contents for problem-solving. Remarkably, CREATOR outperformed existing chain-of-thought (CoT), program-of-thought (PoT), and tool-using baselines on both benchmarks. Additionally, the researchers introduced a new dataset called Creation Challenge which comprises 2K diverse questions. This dataset highlights the necessity and benefits of LLMs' tool creation ability in effectively addressing complex problems. Furthermore, this research reveals that leveraging LLMs as tool creators facilitates knowledge transfer between different domains and demonstrates that they exhibit varying levels of tool creation abilities enabling them to flexibly tackle diverse situations. Overall, this study represents a promising avenue for maximizing the potential of LLMs and advancing towards truly intelligent and adaptable AI systems. The proposed CREATOR framework shows great promise in enhancing LLM performance by enabling them to create their own tools for problem-solving tasks.