# MetaTune: Meta-Learning Based Cost Model for Fast and Efficient Auto-tuning Frameworks

**Authors:** Jaehun Ryu, Hyojin Sung  
**arXiv:** [2102.04199](https://arxiv.org/abs/2102.04199)  
**Date:** 2021-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2102.04199v2/

## Key points

- Deep learning compiler frameworks are becoming more popular for deep learning applications on diverse hardware
- They can match the performance of target-specific libraries while reducing overhead costs
- MetaTune is a new paper proposing a meta-learning based cost model that predicts performance of optimized codes with pre-trained model parameters
- MetaTune encodes convolution kernel codes as structurally similar graphs to facilitate meta-learning and then meta-trains a GNN model with a small input data set
- The resulting framework provides 8 to 13% better inference time on average for four CNN models with comparable or lower optimization time while outperforming transfer learning by 10% in cross-platform cases
- Accuracy of the cost model is essential in locating ideal optimization parameters; many auto-tuning frameworks adopt machine-learning based cost models that can dynamically adapt to search results
- MetaTune offers an efficient solution to optimize deep learning applications across different hardware platforms without sacrificing performance or increasing overhead costs associated with auto-tuning frameworks.

## Summary

Deep learning compiler frameworks are becoming increasingly popular as a more portable back-end for deep learning applications on diverse hardware. They offer the potential to match the performance of hand-tuned target-specific libraries while reducing overhead costs associated with auto-tuning frameworks. To address this issue, a new paper proposes MetaTune: a meta-learning based cost model that quickly and accurately predicts the performance of optimized codes with pre-trained model parameters. MetaTune encodes convolution kernel codes as structurally similar graphs to facilitate meta-learning and then meta-trains a GNN model with a very small input data set. This enables it to predict optimization parameters for unseen convolution operations with varying sizes and structures during compilation. The resulting framework provides 8 to 13% better inference time on average for four CNN models with comparable or lower optimization time while outperforming transfer learning by 10% in cross-platform cases. The accuracy of the cost model is essential in locating ideal optimization parameters; many auto-tuning frameworks including TVM adopt machine-learning based cost models that can dynamically adapt to search results. In TVM, the combination of parameters called knobs determines optimal performance predicted by f which is updated based on hardware measurements. The key insight behind MetaTune is that f can be pre-conditioned with randomly sampled points in a small number of similar optimization spaces so that it can quickly adapt to f∗, target cost model function. Overall, MetaTune offers an efficient solution to optimize deep learning applications across different hardware platforms without sacrificing performance or increasing overhead costs associated with auto-tuning frameworks.