# End-to-end Temporal Action Detection with Transformer

**Authors:** Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Song Bai, Xiang Bai  
**arXiv:** [2106.10271](https://arxiv.org/abs/2106.10271)  
**Date:** 2021-06-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.10271v1/

## Key points

- Temporal Action Detection (TAD) is a crucial task in video understanding
- Existing methods for TAD have limitations such as complex pipelines, lack of end-to-end training, and reliance on hand-designed rules or operations
- The authors propose an end-to-end framework called TadTR for TAD built upon the Transformer architecture
- TadTR simultaneously predicts all action instances as a set of labels and temporal locations in parallel
- TadTR allows for adaptive extraction of temporal context information by selectively attending to relevant snippets in a video
- Compared to previous detectors, TadTR offers faster processing times due to its simplified pipeline and achieves state-of-the-art performance on benchmark datasets like HACS Segments and THUMOS14
- An early work using recurrent neural networks (RNNs) for TAD is mentioned but suffers from slow processing speeds compared to existing methods
- The proposed TadTR framework shows promising results in terms of efficiency and accuracy while simplifying the pipeline and enabling end-to-end training
- The authors provide their code for TadTR which will be made available on GitHub.

## Summary

Temporal Action Detection (TAD) is a crucial task in video understanding, aiming to determine the semantic label and boundaries of every action instance in an untrimmed video. Significant progress has been made in TAD, but existing methods have limitations such as complex pipelines, lack of end-to-end training and reliance on hand-designed rules or operations. To address these challenges, the authors propose an end-to-end framework for TAD called TadTR built upon the Transformer architecture. TadTR simultaneously predicts all action instances as a set of labels and temporal locations in parallel. This approach allows for adaptive extraction of temporal context information by selectively attending to relevant snippets in a video. Compared to previous detectors, TadTR offers several advantages including faster processing times due to its simplified pipeline and state-of-the-art performance on benchmark datasets like HACS Segments and THUMOS14. The authors also mention an early work that proposed a method for TAD using recurrent neural networks (RNNs), however this approach suffers from slow processing speeds compared to existing methods. Overall, the proposed TadTR framework for TAD shows promising results in terms of both efficiency and accuracy while simplifying the pipeline and enabling end-to-end training. The authors provide their code for TadTR which will be made available on GitHub.