# Context Matters: Refining Object Detection in Video with Recurrent Neural Networks

**Authors:** Subarna Tripathi, Zachary C. Lipton, Serge Belongie, Truong Nguyen  
**arXiv:** [1607.04648](https://arxiv.org/abs/1607.04648)  
**Date:** 2016-07-15  
**Summary page:** https://www.summarizepaper.com/en/paper/1607.04648v1/

## Key points

- The paper explores challenges of object detection in videos and proposes a new framework to improve accuracy
- Motion blur and compression artifacts introduce variability in video frames
- Video datasets often have sparsely annotated frames, making training difficult
- The framework consists of a pseudo-labeler and a recurrent neural network (RNN)
- The pseudo-labeler is trained on labeled frames and applied to all frames for better generalization
- An RNN is trained using sequences of pseudo-labeled frames to optimize accuracy and encourage consistency across consecutive frames
- The proposed approach achieves impressive results on the Youtube-Video Objects dataset, with a mean Average Precision (mAP) of 68.73
- Neighboring frames provide valuable information for object detection even without explicit labels

## Summary

The paper titled "Context Matters: Refining Object Detection in Video with Recurrent Neural Networks" by Subarna Tripathi, Zachary C. Lipton, Serge Belongie, and Truong Nguyen explores the challenges of object detection in videos and proposes a new framework to improve its accuracy. Object detection in videos is a promising field due to the vast amount of video content available online and recent advancements in object detection with static images. However, motion blur and compression artifacts introduce significant variability at the frame level, even in videos that appear smooth to the human eye. Additionally, video datasets often have sparsely annotated frames, making it challenging to train accurate object detectors. To address these challenges, the authors present a novel framework that captures temporal context and encourages consistency of predictions for object detection in videos. The framework consists of two main components: a pseudo-labeler and a recurrent neural network (RNN). The pseudo-labeler is a domain-adapted convolutional neural network (CNN) specifically designed for object detection. It is first trained individually on a subset of labeled frames and then applied to all frames in the video. This approach allows for better generalization across frames and improves the accuracy of object detection. Next, an RNN is trained using sequences of pseudo-labeled frames as input. The objective of the RNN training is twofold: to optimize accuracy on the target frame and encourage consistency across consecutive frames. This is achieved by incorporating strong supervision on target frames, weak-supervision on context frames, and regularization through a smoothness penalty. The proposed approach achieves impressive results on the Youtube-Video Objects dataset, with a mean Average Precision (mAP) of 68.73. This represents an improvement of 7.1 over the strongest image-based baselines. The experiments conducted demonstrate that neighboring frames can provide valuable information for object detection even without explicit labels. In conclusion, this paper presents a new framework for improving object detection in videos by capturing temporal context and encouraging consistency of predictions.