# Better than classical? The subtle art of benchmarking quantum machine learning models

**Authors:** Joseph Bowles, Shahnawaz Ahmed, Maria Schuld  
**arXiv:** [2403.07059](https://arxiv.org/abs/2403.07059)  
**Date:** 2024-03-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2403.07059v2/

## Key points

- Benchmarking models through classical simulations is crucial in quantum machine learning until noise-free hardware becomes available
- Challenges include the impact of experimental design, limited scales achievable today, and narratives influenced by commercialization of quantum technologies
- An open-source package based on the PennyLane software framework was developed to address these challenges
- Study revealed that classical machine learning models generally outperformed quantum classifiers out-of-the-box
- Removing entanglement from a quantum model often resulted in equal or superior performance, suggesting "quantumness" may not be essential for small learning tasks
- Insights beyond simplistic leaderboard comparisons were provided, identifying five important questions for future quantum model design
- Systematic approach taken to select models for implementation from 3500 papers accessed via arXiv API within "quant-ph" category with specific keywords related to classification and supervised learning
- Final selection of 12 quantum machine learning models implemented across three families: quantum neural networks, quantum kernel methods, and quantum convolutional neural networks
- Various issues highlighted in existing approaches such as oversimplification of classification tasks leading to inflated performance metrics
- Overall results ranged from accuracies of 70% to 99.6%, indicating mixed outcomes compared to known baselines like the MNIST dataset
- Careful consideration of experimental design and model complexity is essential in drawing meaningful conclusions about the efficacy of quantum machine learning compared to classical counterparts

## Summary

, , , , 

In the realm of quantum machine learning, benchmarking models through classical simulations is crucial for evaluating ideas until noise-free hardware becomes available. However, the impact of experimental design, limited scales achievable today, and narratives influenced by the commercialization of quantum technologies pose challenges in gaining robust insights. To address this issue and facilitate better decision-making, an open-source package based on the PennyLane software framework was developed. This package was used to conduct a large-scale study testing 12 popular quantum machine learning models on 6 binary classification tasks across 160 individual datasets. The study revealed that classical machine learning models generally outperformed quantum classifiers out-of-the-box. Surprisingly, removing entanglement from a quantum model often resulted in equal or superior performance, suggesting that "quantumness" may not be essential for small learning tasks considered in the study. The benchmarks provided insights beyond simplistic leaderboard comparisons and identified five important questions for future quantum model design. To select models for implementation, a systematic approach was taken. A total of 3500 papers were accessed via the arXiv API within the "quant-ph" category with specific keywords related to classification and supervised learning. From this pool, papers with significant citations were sub-selected to focus on widely-used ideas in quantum machine learning research. Subsequently, papers proposing new quantum models suitable for numerical testing were chosen for implementation. From a final selection of 11 papers, 12 quantum machine learning models were implemented across three families: quantum neural networks, quantum kernel methods, and quantum convolutional neural networks. The study highlighted various issues in existing approaches, such as oversimplification of classification tasks leading to inflated performance metrics. Overall results ranged from accuracies of 70% to 99.6%, indicating mixed outcomes compared to known baselines like the MNIST dataset. In conclusion, while quantum machine learning holds promise for future advancements, careful consideration of experimental design and model complexity is essential to draw meaningful conclusions about their efficacy compared to classical counterparts.