# Learning from One Continuous Video Stream

**Authors:** João Carreira, Michael King, Viorica Pătrăucean, Dilara Gokay, Cătălin Ionescu, Yi Yang, Daniel Zoran, Joseph Heyward, Carl Doersch, Yusuf Aytar, Dima Damen, Andrew Zisserman  
**arXiv:** [2312.00598](https://arxiv.org/abs/2312.00598)  
**Date:** 2023-12-01  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2312.00598v1/

## Key points

- Authors propose a framework for learning from a single continuous video stream
- Challenges of learning from highly correlated consecutive video frames and lack of prior work in this area are highlighted
- Introduce a collection of streams and tasks composed from two existing video datasets to address these challenges
- Methodology presented considers both adaptation and generalization
- Pixel-to-pixel modeling employed as a practical and flexible approach to switch between streams and tasks
- Framework achieves significant gains in single-stream learning through pre-training with novel family of tasks
- Key findings include negative impact of momentum on performance, importance of pace of weight updates, and matching performance of IID learning without replay buffers
- Related work discussed in semi-supervised object detection, training ConvNets or ViTs from single images or long videos, parallelization in batch size 1 setting, online continual learning, and continual learning with temporal correlations
- Exploration of representation learning to mitigate challenges in continual learning using features pretrained in IID settings
- Proposal of new future prediction pretraining approaches that transfer well to single-stream learning
- Comprehensive framework for online learning from continuous video streams presented with valuable insights into optimizing performance

## Summary

In this paper, the authors propose a framework for  from a single continuous . They highlight the challenges of learning from highly correlated consecutive video frames and note the lack of prior work in this area. To address these challenges, they introduce a collection of streams and tasks composed from two existing video datasets. They also present a methodology for  that considers both adaptation and generalization. The authors employ pixel-to-pixel modeling as a practical and flexible approach to switch between  and single-stream evaluation, as well as between arbitrary tasks. Importantly, their framework does not require changes to models and always uses the same pixel loss. By leveraging this framework, they achieve significant gains in single-stream learning through pre-training with a novel family of  tasks. The authors make several key findings in their study. They discover that momentum negatively impacts performance and emphasize the importance of the pace of weight updates. Interestingly, they demonstrate that their approach can match the performance of IID (independent and identically distributed) learning with batch size 1, without relying on costly replay buffers. The paper also provides additional context by discussing related work in semi-supervised object detection from streaming video, training ConvNets or ViTs from single images or long videos, parallelization in a batch size 1 setting, online continual learning, and continual learning with temporal correlations. Furthermore, the authors explore how representation learning can mitigate some challenges in continual learning when using features pretrained in IID settings. They investigate this aspect in their setup with temporally-correlated data and propose new future prediction pretraining approaches that transfer well to single-stream learning. Overall, this paper presents a comprehensive framework for online learning from continuous video streams. The authors provide valuable insights into optimizing performance in single-stream learning scenarios while considering temporal correlations and demonstrate the effectiveness of their approach through experimental results.