# DETRs Beat YOLOs on Real-time Object Detection

**Authors:** Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, Yi Liu  
**arXiv:** [2304.08069](https://arxiv.org/abs/2304.08069)  
**Date:** 2023-04-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08069v1/

## Key points

- End-to-end transformer-based detectors (DETRs) have shown impressive performance in object detection tasks.
- DETRs have a significant drawback of high computational cost, limiting their practical application and ability to leverage techniques like non-maximum suppression (NMS).
- Real-Time DEtection TRansformer (RT-DETR) was developed by a team led by Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang and Yi Liu to address the issue of computational cost and eliminate inference delay caused by NMS.
- RT-DETR introduces innovations such as an efficient hybrid encoder for multi-scale feature processing and IoU-aware query selection for enhanced object query initialization.
- RT-DETR offers flexibility in adjusting inference speed without requiring retraining, enhancing the practical utility of real-time object detectors.
- RT-DETR-L achieved 53.0% Average Precision (AP) at 114 Frames Per Second (FPS), while RT-DETR-X outperformed YOLO detectors with 54.8% AP at 74 FPS.
- RT-DETR-R50 achieved 53.1% AP at 108 FPS surpassing other models in both accuracy and speed.
- The research outcomes are promising for advancing real-time object detection capabilities, with plans to make source code and pretrained models accessible through PaddleDetection platform.

## Summary

In recent years, end-to-end transformer-based detectors (DETRs) have emerged as a powerful tool for object detection tasks, showcasing impressive performance. However, a significant drawback of DETRs has been their high computational cost, hindering their practical application and limiting their ability to fully leverage the advantages of no post-processing techniques like non-maximum suppression (NMS). To address this issue, a team of researchers led by Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang and Yi Liu have developed a groundbreaking solution in the form of Real-Time DEtection TRansformer (RT-DETR). The primary focus of their work was to analyze the impact of NMS on the speed of modern real-time object detectors and establish a benchmark for end-to-end speed. By introducing RT-DETR as the first real-time end-to-end object detector known to date, the researchers aimed to eliminate the inference delay caused by NMS. The key innovations in RT-DETR include an efficient hybrid encoder that processes multi-scale features by separating intra-scale interaction and cross-scale fusion. Additionally, they introduced IoU-aware query selection to enhance the initialization of object queries. One notable feature of RT-DETR is its flexibility in adjusting inference speed by utilizing different decoder layers without requiring retraining. This adaptability enhances the practical utility of real-time object detectors. The team's efforts culminated in impressive results: RT-DETR-L achieved 53.0% Average Precision (AP) on COCO val2017 dataset at 114 Frames Per Second (FPS) on T4 GPU. Similarly, RT-DETR-X outperformed all YOLO detectors of comparable scale in both speed and accuracy with 54.8% AP at 74 FPS. Furthermore, RT-DETR-R50 achieved 53.1% AP and 108 FPS surpassing DINO-Deformable-DETR-R50 by 2.2% AP in accuracy and approximately 21 times in FPS. The research outcomes are not only promising but also impactful for advancing real-time object detection capabilities. The team plans to make their source code and pretrained models accessible through PaddleDetection platform for further exploration and application in diverse domains requiring efficient object detection solutions.