PyramidTNT: Improved Transformer-in-Transformer Baselines with Pyramid Architecture

In the realm of computer vision tasks, Transformer networks have made significant strides. The Transformer-in-Transformer (TNT) architecture stands out for its ability to extract both local and global representations using inner and outer transformers. Building upon this foundation, a team of researchers including Kai Han, Jianyuan Guo, Yehui Tang, and Yunhe Wang have introduced new TNT baselines featuring two innovative designs: a pyramid architecture and a convolutional stem. The resulting "PyramidTNT" model represents a substantial improvement over the original TNT by establishing hierarchical representations. Notably, PyramidTNT has demonstrated superior performance compared to leading vision transformers like Swin Transformer. The researchers anticipate that this enhanced baseline will prove invaluable for advancing research and practical applications in the field of vision transformers. For those interested in exploring further, the code for PyramidTNT is set to be made available at https://github.com/huawei-noah/CV-Backbones/tree/master/tnt_pytorch. This work serves as an extension of the "Transformer in Transformer" concept and underscores the ongoing evolution of transformer-based approaches in computer vision.

In recent years, transformer networks have emerged as a powerful tool in the field of computer vision. These networks, originally designed for natural language processing tasks, have shown great potential in handling visual data and have achieved state-of-the-art results in various computer vision tasks. Among these transformer-based architectures, Transformer-in-Transformer (TNT) has stood out for its ability to extract both local and global representations using inner and outer transformers.

A team of researchers including Kai Han, Jianyuan Guo, Yehui Tang, and Yunhe Wang has recently introduced new TNT baselines featuring two innovative designs: a pyramid architecture and a convolutional stem. This work serves as an extension of the "Transformer in Transformer" concept and underscores the ongoing evolution of transformer-based approaches in computer vision.

The original TNT model was proposed by Han et al. in their paper "Transformers with convolutional context for efficient image recognition". It introduced the concept of using inner transformers to capture local features within an image while outer transformers were used to capture global information. This approach proved effective but had limitations when it came to handling hierarchical structures within images.

To address this issue, the team proposed a new architecture called PyramidTNT which incorporates both inner and outer transformers into a pyramid structure. The resulting model is able to establish hierarchical representations by capturing features at different scales within an image. This allows PyramidTNT to better handle complex images with multiple levels of detail.

Furthermore, the researchers also introduced a convolutional stem design which replaces the initial linear projection layer with a series of convolutional layers. This modification not only reduces computational costs but also improves feature extraction capabilities compared to the original TNT model.

To evaluate their proposed architecture, the team conducted experiments on several benchmark datasets including ImageNet-1k, CIFAR-100, COCO object detection dataset and ADE20K semantic segmentation dataset. Their results showed that PyramidTNT outperformed other leading vision transformers such as Swin Transformer, DeiT and ViT in terms of accuracy and efficiency.

In addition to achieving state-of-the-art results, the researchers also demonstrated the versatility of PyramidTNT by applying it to various downstream tasks such as object detection and semantic segmentation. The model showed promising performance in these tasks, further highlighting its potential for practical applications.

The team plans to release the code for PyramidTNT on GitHub (https://github.com/huawei-noah/CV-Backbones/tree/master/tnt_pytorch), making it accessible for other researchers to use and build upon. This will facilitate further advancements in transformer-based approaches for computer vision tasks.

In conclusion, the introduction of PyramidTNT represents a significant improvement over the original TNT model and showcases the continuous evolution of transformer-based architectures in computer vision. With its ability to capture hierarchical representations and achieve state-of-the-art results, this new baseline is expected to have a significant impact on future research and practical applications in this field.