# Compact Multi-Class Boosted Trees

**Authors:** Natalia Ponomareva, Thomas Colthurst, Gilbert Hendry, Salem Haykal, Soroush Radpour  
**arXiv:** [1710.11547](https://arxiv.org/abs/1710.11547)  
**Date:** 2017-10-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1710.11547v1/

## Key points

- Authors Natalia Ponomareva, Thomas Colthurst, Gilbert Hendry, Salem Haykal, and Soroush Radpour introduce two extensions to standard tree boosting algorithm for gradient boosted decision trees.
- First enhancement: Extending boosting formalism from scalar-valued trees to vector-valued trees allows individual trees to function as multiclass classifiers, reducing model size for multiclass problems.
- Various popular modifications of vector-valued gradient boosted trees can be seamlessly integrated into this framework.
- Second extension: Layer-by-layer boosting involves taking smaller steps in function space, leading to faster convergence and a more compact ensemble overall.
- Both improvements implemented in open-source TensorFlow Boosted Trees (TFBT) package with successful demonstrations on multiclass datasets.
- Extensions beneficial for applications requiring small model sizes (e.g., embedded devices), rapid inference, and interpretable models.

## Summary

In their paper titled "Compact Multi-Class Boosted Trees," authors Natalia Ponomareva, Thomas Colthurst, Gilbert Hendry, Salem Haykal, and Soroush Radpour introduce two extensions to the standard tree boosting algorithm aimed at enhancing the efficiency of gradient boosted decision trees. These extensions are designed to improve the accuracy of small models commonly used in machine learning applications. The first enhancement involves extending the boosting formalism from scalar-valued trees to vector-valued trees. This innovation allows individual trees to function as multiclass classifiers, eliminating the need for a separate tree per class and significantly reducing the model size required for multiclass problems. The authors also demonstrate that various popular modifications of vector-valued gradient boosted trees can be seamlessly integrated into this framework. The second extension introduced in the paper is layer-by-layer boosting, which involves taking smaller steps in function space. Empirical evidence presented by the authors suggests that this approach leads to faster convergence and results in a more compact ensemble overall. Both improvements have been implemented in the open-source TensorFlow Boosted trees (TFBT) package, with successful demonstrations of their efficacy on a range of multiclass datasets. The authors anticipate that these extensions will be particularly beneficial for applications of boosted tree models that necessitate small model sizes, such as those deployed on embedded devices or requiring rapid inference. Additionally, these enhancements may appeal to applications seeking more interpretable models due to their ability to streamline multiclass classification tasks and optimize model efficiency.