# Trace is the New AutoDiff -- Unlocking Efficient Optimization of Computational Workflows

**Authors:** Ching-An Cheng, Allen Nie, Adith Swaminathan  
**arXiv:** [2406.16218](https://arxiv.org/abs/2406.16218)  
**Date:** 2024-06-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.16218v1/

## Key points

- Authors: Ching-An Cheng, Allen Nie, and Adith Swaminathan
- Introduction of an innovative end-to-end optimization framework called Trace for automating the design and enhancement of AI systems
- Computational workflows viewed as graphs similar to neural networks and built upon a generalized form of back-propagation
- Proposal of Optimization with Trace Oracle (OPTO) mathematical framework to address challenges in optimizing computational workflows
- Implementation through Trace tool with a Python interface that converts workflows into OPTO instances using a PyTorch-like interface
- Development of OptoPrime—a versatile LLM-based optimizer capable of efficiently solving OPTO problems
- Empirical studies demonstrating OptoPrime's excellence in various tasks including first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, among others
- Integration of Trace, OptoPrime, and the OPTO framework expected to enable next-generation interactive agents adept at automatic adaptation through diverse forms of feedback

## Summary

In their paper titled "Trace is the New AutoDiff -- Unlocking Efficient Optimization of Computational Workflows," authors Ching-An Cheng, Allen Nie, and Adith Swaminathan introduce an innovative end-to-end optimization framework called Trace for automating the design and enhancement of AI systems. This approach views computational workflows as graphs similar to neural networks and is built upon a generalized form of back-propagation. The authors propose a novel mathematical framework known as Optimization with Trace Oracle (OPTO) to address the unique challenges posed by optimizing computational workflows. OPTO captures key properties essential for designing effective optimizers across multiple domains and is implemented through Trace—a tool equipped with a Python interface that seamlessly converts workflows into OPTO instances using a PyTorch-like interface. Leveraging Trace's capabilities, the authors develop OptoPrime—a versatile LLM-based optimizer capable of efficiently solving OPTO problems. Through empirical studies, it was demonstrated that OptoPrime excels in various tasks including first-order numerical optimization, prompt optimization, hyper-parameter tuning, robot controller design, code debugging, among others. Remarkably, OptoPrime often rivals specialized optimizers tailored for specific domains. The integration of Trace, OptoPrime, and the OPTO framework is expected to pave the way for next-generation interactive agents adept at automatic adaptation through diverse forms of feedback.