# GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms

**Authors:** Jiwei Liu, Bojan Tunguz, Gilberto Titericz  
**arXiv:** [2012.04201](https://arxiv.org/abs/2012.04201)  
**Date:** 2020-12-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2012.04201v3/

## Key points

- Black-box optimization algorithms can be used to tune complex machine learning algorithms
- A simple ensemble of black-box optimization algorithms can outperform any single algorithm
- Finding the optimal ensemble requires a large number of experiments
- The authors propose a Multi-GPU-optimized framework to accelerate the search for the optimal ensemble
- The framework leverages GPUs for expensive model training and evaluations, while CPUs handle lightweight optimizations
- By distributing the workload across GPUs, the search time is reduced from over 10 days on two 20-core CPUs to less than 24 hours on 8-GPUs
- The authors evaluate 15 different optimizers by training 2.7 million models and conducting 541,440 optimizations
- The experiments were conducted on a DGX-1 system and achieved second place in the NeurIPS 2020 black-box optimization challenge
- The paper presents a novel solution for accelerating the search for an optimal ensemble through GPU acceleration
- The findings contribute to advancing efficient algorithm tuning techniques in machine learning domains.

## Summary

The paper titled "GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms" by Jiwei Liu, Bojan Tunguz, and Gilberto Titericz explores the use of black-box optimization algorithms for tuning complex machine learning algorithms. The authors demonstrate that a simple ensemble of these black-box optimization algorithms can outperform any single algorithm. However, finding the optimal ensemble requires a large number of experiments. To address this challenge, the authors propose a Multi-GPU-optimized framework that accelerates the search for the optimal ensemble by running multiple experiments in parallel. The framework leverages GPUs for expensive model training and evaluations while lightweight optimizations are performed by CPUs. By distributing the workload across GPUs, the authors significantly reduce the search time from over 10 days on two 20-core CPUs to less than 24 hours on 8-GPUs. In their evaluation, the authors assess 15 different optimizers by training an impressive 2.7 million models and conducting 541,440 optimizations. They conduct their experiments on a DGX-1 system and demonstrate the effectiveness of their GPU-accelerated exhaustive search approach. The results obtained using this optimized framework allowed the authors to achieve second place in the NeurIPS 2020 black-box optimization challenge; highlighting its practical significance and competitiveness in real-world scenarios. Overall, this paper presents a novel solution to accelerate the search for an optimal ensemble of black-box optimization algorithms through GPU acceleration. The findings contribute to advancing our understanding of efficient algorithm tuning techniques and have potential applications in various machine learning domains.