# Octopus: On-device language model for function calling of software APIs

**Authors:** Wei Chen, Zhiyuan Li, Mingyuan Ma  
**arXiv:** [2404.01549](https://arxiv.org/abs/2404.01549)  
**Date:** 2024-04-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.01549v1/

## Key points

- Large Language Models (LLMs) are essential in artificial intelligence for their advanced text processing and generation capabilities.
- A novel strategy is introduced to leverage on-device LLMs for invoking software APIs, addressing challenges in real-time updates and task execution.
- Fine-tuning LLMs with 2B, 3B, and 7B parameters enhances their proficiency in software API interactions by refining understanding of API structures and syntax.
- Conditional masking techniques ensure outputs in desired formats, reduce error rates, and maintain inference speeds during API function calls.
- Octopus, a fine-tuned model, outperforms GPT-4 for software APIs calling, demonstrating superior performance.
- The integration of LLMs with external APIs advances automated software development and API integration by balancing large-scale model dependency against efficiency and cost considerations.
- Task-oriented LLMs are developed to preserve essential functionality while minimizing operational costs through curriculum learning strategies on an expansive dataset of over 30,000 widely-utilized APIs from Rapid API Hub.

## Summary

, , , , 

In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) have become essential due to their advanced text processing and generation capabilities. This study introduces a novel strategy aimed at leveraging on-device LLMs for invoking software APIs, addressing the challenges faced by LLMs in assimilating real-time updates and executing specific tasks. By compiling a dataset derived from software API documentation and applying fine-tuning to LLMs with capacities of 2B, 3B, and 7B parameters, the models' proficiency in software API interactions is significantly enhanced. The approach focuses on refining the models' understanding of API structures and syntax to improve the accuracy of API function calls. Additionally, conditional masking techniques are proposed to ensure outputs in desired formats and reduce error rates while maintaining inference speeds. A new benchmark is introduced to evaluate the effectiveness of LLMs in API interactions, with Octopus, the fine-tuned model, demonstrating superior performance compared to GPT-4 for software APIs calling. The integration of LLMs with external APIs represents a pivotal advancement in automated software development and API integration. By balancing large-scale model dependency against efficiency and cost considerations, smaller task-oriented LLMs are developed to preserve essential functionality while minimizing operational costs. The framework proposed in this study leverages curriculum learning strategies on an expansive dataset of over 30,000 widely-utilized APIs from Rapid API Hub. By selecting appropriate base models such as Codellama7b, Google's Gemma 7B & 2B, and Stable Code 3B, the effectiveness of the approach is underscored through outperforming GPT-4 benchmarks. Conditional masking techniques during inference ensure consistent output formatting, improving accuracy without sacrificing speed. This research contributes significantly towards enhancing automated software development processes and advancing the integration of LLM capabilities with practical software engineering applications.