# Transformer-based stereo-aware 3D object detection from binocular images

**Authors:** Hanqing Sun, Yanwei Pang, Jiale Cao, Jin Xie, Xuelong Li  
**arXiv:** [2304.11906](https://arxiv.org/abs/2304.11906)  
**Date:** 2023-04-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.11906v3/

## Key points

- Transformers have shown progress in visual object detection tasks such as monocular 2D/3D and surround-view 3D detection
- Existing Transformer-based detectors face challenges in binocular stereo 3D object detection due to oversight of stereo-specific image correspondence information
- TS3D (Transformer-based Stereo-aware 3D object detector) addresses this issue by incorporating a Disparity-Aware Positional Encoding (DAPE) module to embed image correspondence information into stereo features
- TS3D effectively provides accurate 3D location information for objects by encoding normalized sub-pixel-level disparity alongside sinusoidal 2D positional encoding
- A Stereo Preserving Feature Pyramid Network (SPFPN) is introduced in TS3D to extract enriched multi-scale stereo features while preserving correspondence information and aggregating cross-scale features

## Summary

, , , , 

In the field of visual object detection, Transformers have shown significant progress in tasks such as monocular 2D/3D detection and surround-view 3D detection. The attention mechanism in Transformer models and the image correspondence in binocular stereo are both similarity-based, highlighting the potential for utilizing Transformers in stereo-aware 3D object detection. However, existing Transformer-based detectors face challenges when applied directly to binocular stereo 3D object detection, leading to slow convergence and decreased precision. This is primarily due to the oversight of stereo-specific image correspondence information by current Transformer models. To address this issue, a new model design called TS3D (Transformer-based Stereo-aware 3D object detector) is proposed. The key innovation lies in the incorporation of a Disparity-Aware Positional Encoding (DAPE) module within TS3D, which embeds image correspondence information into stereo features. By encoding normalized sub-pixel-level disparity alongside sinusoidal 2D positional encoding, TS3D effectively provides accurate 3D location information for objects in the scene. Additionally, a Stereo Preserving Feature Pyramid Network (SPFPN) is introduced to extract enriched multi-scale stereo features while preserving correspondence information and aggregating cross-scale features. Experimental results on the KITTI test set demonstrate the effectiveness of TS3D, achieving a Moderate Car detection average precision of 41.29% with an inference speed of 88 ms per binocular image pair. Comparative analysis with existing feature pyramid networks further validates the superiority of SPFPN in extracting multi-scale stereo features for improved object detection performance. Overall, TS3D presents a competitive solution for stereo-aware 3D object detection by leveraging Transformer models and emphasizing the importance of incorporating stereo-specific image correspondence information into the model design.